HiiL POLICY BRIEF
How to figure out “What works”
in People-centered justice?
“What works” is a crucial yet still unanswered question in the justice field. This policy brief calls for urgent policy, research and funding action to address four aspects of the problem:
- Shift the focus towards the outcomes of justice
- Clarify the concept of justice outcomes
- Define the justice interventions rigorously
- Delve deeper into the modalities of “What works”
Introduction
At any time, more than 1.5 billion people worldwide have legal problems they cannot solve. Data from counties where HiiL worked make the problem more concrete. For instance, in the US, every year, there are 260 million significant and difficult-to-resolve legal problems. One hundred twenty million of them do not find a fair solution.1 In Ethiopia, there are 7.6 million legal problems yearly, of which 5.2 million remain unresolved.2
Legal problems occur everywhere around us. More likely than not, each of us will get into a dispute or grievance and will need some legal process to bring us to a fair resolution. But are there solutions to the many legal problems? Are these solutions equally accessible? Do we know which pathways to solutions work better than others?
To date, there is minimal data about the outcomes of legal problems. However, data is emerging – mainly as the result of large-scale national legal needs surveys. These data persistently show that not enough legal problems are resolved fairly. The reasons for the access to justice gap are complex, but they are not an excuse not to delve further into the issue. The justice needs of people often sit “in the blind spot” of justice policymakers and providers – unseen and unthought-of. In fact, the rules, procedures and stakeholders are often not concerned with the people who need fair resolutions.
To achieve UN’s SDG 16.3 “Equal access to justice for all”, we need diverse justice delivery models that can scale up massively in correspondence with the huge demand. How do we know if such solutions “work”? What does it mean that something “works” in justice? These seemingly simple questions should be coupled with additional questions such as “working where”, “working for whom”, “working for what sort of questions”, “working under what conditions”, etc.
Knowing “what works” is essential for every human and social activity area. Investing scarce and limited resources in actions with a higher chance of succeeding is prudent. A huge research and development industry informs the healthcare field about which treatments and procedures lead to results. Educational professionals are constantly searching for new methods and approaches that “work” in delivering better scholarly outputs and outcomes. It is different, however, in the field of access to justice, where services and interventions are rarely measured and evaluated to figure out “what works”.
“[..] a comparable evidence-based approach is notably absent from the many efforts to expand access to the justice system for people facing such civil legal problems as foreclosure, eviction, child custody disputes, domestic violence, or consumer fraud claims.” 3
To put it simply, “what works” are interventions that alter the outcomes of the justice journeys positively.” 4
The primary purpose of this policy brief is to make a case for and provide examples of evidence-based insights about “What works” and to stimulate decision-makers and researchers to continue further with this process.
1. The problem with knowing “what works” in the field of justice
What do we know about “What works” in justice? There is some progress, but the truth is that we do not know much about “What works” in justice. First, there is very little robust data in the field of access to justice. “Gold standard” randomised control trials are extremely rare. Fortunately, the growing empirical legal research and evidence-based policy movements aspire to bridge that gap, but it will take much time and resources to make real progress.
Second, there is very little agreement about what the “outcomes of interventions” mean in the justice field. More often than not, justice outcomes are reduced to case outcomes. Administrative and court data at national and regional levels tell us whether a case is solved or is still pending, but not a lot more.
Third, the interventions for resolving legal problems are rarely viewed as packages of activities designed and implemented to solve specific problems. This makes it challenging to design appropriate research, gather data and answer the “what works” question in sufficient detail. For instance, adjudication alone is often a combination of various treatments such as advice, mediation and reconciliation, representation, and deciding the matter – not to mention all of these treatments occurring for different clients with different needs in different contexts. With so many ingredients, it is difficult to claim that adjudication “works” or “does not work”.
Fourth, the attribution problem makes it even more difficult to establish “what works” in the justice field. The question itself is based on a solid causal relationship between an intervention and a (positive) outcome. Randomised control trials based on experimental and quasi-experimental designs are not easy to implement in the justice field.
Nevertheless, “What works” is an essential part of people-centred justice.
The OECD calls for “developing and implementing policies and services that meet [people’s legal] needs, and removing barriers to access as part of a holistic vision for a people-centred justice ecosystem”.5 To achieve that, there is a need for “ongoing and co-ordinated research and evaluation conducted to identify and maintain an evidence base about what strategies “work” most effectively and cost-effectively, for whom, and in what circumstances to address legal and justice needs, including in the planning and delivery of legal and justice services”.6
“Implicit in designing cost-effective and effective services for the particular context is the need to have sound knowledge in relation to what strategies, interventions and services are most effective and cost-effective at addressing particular legal and justice needs. This in turn implies understanding what works in any circumstance, including for those people with multiple disadvantages and experiencing multiple problems who may not act in the rational way that legal systems may expect them to. In other words, identifying “what works” would need to take into account people, circumstances and emotions; pathways to resolution and support must be informed by how people experience legal and justice problems, and how they engage available pathways to address them.” 7
2. “What works” in delivering justice to the people: an exercise with legal needs data
HiiL has conducted legal needs surveys in more than 20 countries worldwide. More than 115,000 individuals from more than 25 countries were interviewed about their legal problems and needs for justice. We queried this rich source to understand more about what works in justice.
The main purpose of the analysis is to establish an interest in “what” works by demonstrating the importance and potential of people-centred data. To keep this policy paper short, we explain the data, methodology and detailed findings in a separate background paper. Here, we restate the main findings of three multivariate models that look at the legal needs dataset to answer the “What works” question.
Model 1: Courts and lawyers have significant resolution power but take a lot of time to solve legal problems
Model 1 analyses the association between the results of justice journeys and three sets of variables: party-related variables8, problem-related variables9 and process-related variables. The dependent variable in Model 1 is the resolution of the problem measured at four levels – “Completely resolved”, “Partially resolved”, “Ongoing”, and “Not resolved”. The dispute resolution process in this model is represented by the process perceived as most helpful in resolving the legal issue. We aggregated the many types of dispute resolution mechanisms into a few major categories – “Courts and lawyers”, “Police”, “Other organised procedures”, “Personal network”, and “Self-action”.
The main finding from this analysis is that using “Courts and lawyers” increases the likelihood that a problem is “Completely resolved” compared to other mechanisms. However, “Courts and lawyers” are slow – using “Courts and lawyers” greatly increases the risk that a legal problem is “Ongoing”. The use of “Police” and “Self-action” increases the risk that a problem is “Not resolved.”
Model 2: The users do not see considerable differences in the quality of the outcome delivered by various dispute resolution mechanisms
Model 2 uses essentially the same set of independent variables 10, however, the dependent variable in this model is a composite measure of the quality of the outcome. The elements of this outcome variable are measures of distributive justice, restorative justice, enforcement and the potential of the result to resolve the underlying problem.11
The results imply no significant differences in the outcome quality of the various dispute resolution mechanisms. We compared all categories of dispute resolution mechanisms to the category of “Courts and lawyers” and found that the differences are not statistically significant. Men report higher satisfaction with the quality of the outcome than women. However, in the multivariate model, this difference is not significant.
There are statistically significant differences in the perceived quality of the outcome in some person- and problem-related variables. Compared to the youngest category, young adults (24-35) and middle age (35-55) report better outcomes. People with medium and high education report significantly better results than people without education. However, higher-income individuals report worse results than those with lower incomes. The justice journeys for employment and family-related legal problems receive lower outcome scores compared to land problems.
Model 3: Deciding a matter is an intervention that “works”, but the caveat is that deciding can take a long time
Model 3 explores the impact of dispute resolution interventions on the outcomes of legal problems. Interventions are the discrete activities that third parties apply to resolve problems. A third party can use one or more interventions in a dispute. Therefore, the variable is based on a multiple-choice selection – more than one of the following interventions (or lack of interventions) is possible.12
“Advice” is the most frequently used intervention in resolving legal problems related to land, employment and family issues. However, “Advice” seems to be the least effective of the interventions except for the “Other” and “Doing nothing” options. “Mediating/reconciling”, “Deciding the matter”, and even “Referring” increase considerably the chance that a problem is “Completely resolved” as compared to being “Not resolved”. The options “Doing nothing” or “Other” interventions increase considerably the risk that a problem remains “Not resolved”.
“Deciding” is the intervention that most considerably outperforms “Advice” 13 as a strategy to “Completely resolve” a problem. “Deciding” an issue substantively decreases the risk of the problem being “Ongoing”. “Representing” and “Doing nothing” both increase the likelihood that a problem will be “Ongoing” instead of “Completely resolved”.
“Deciding” also decreases the risk that a problem is “Not resolved” compared to the “Advising” intervention. “Doing nothing” significantly increases the risk that a problem ends as “Not resolved”. Compared to “Preparing documents”, “Mediating”, “Deciding” and even “Referring”, “Advice” significantly increases the risk that people consider a problem is “Not resolved”. “Preparing documents”, “Mediating”, “Deciding”, and “Referring” increase the likelihood that a legal problem is Ongoing.
3. A call to action to further the “What works” knowledge
The above empirical analysis does not yield a definitive conclusion about “What works”. It is not a big surprise that the findings are nuanced and do not always intuitively indicate straightforward solutions. In fact, the results point in many directions and invoke the need for more data and analysis.
As expected, the results of the „What works” analysis are ambivalent. This inconclusiveness is consistent with the diverse and rarely evidence-based theory and practice of access to justice. Most likely, there will never be silver bullet solutions that resolve legal problems regardless of the specifics of individuals, issues and contexts. Dispute resolution is highly context-specific. Legal, cultural, social and economic factors play large roles in how people encounter, process and resolve disputes. Questions such as “how”, “when”, “for whom”, and “where” are important parts of the pursuit of knowledge about the ways to resolve legal disputes. Nevertheless, the justice gap cannot be addressed without at least generally knowing “What works” in access to justice.
Regardless of the diversity and depth of the problem, there is an urgent need to shift the focus to the outcomes of justice processes. Three specific areas need the urgent attention of policymakers, service providers, researchers and access to justice donors. Our policy call is to invest attention, efforts and resources to make advances in the one general and three specific challenges listed below. The results of such investments will not immediately show “What works” in access to justice. However, small and big advancements will bring better tools and mechanisms to gather robust evidence and make justice more people-centred.
- General challenge: Shift the focus towards the outcomes of justice interventions.
- What does “to work” mean? Define clear, valid and measurable outcomes of interventions.
- What is the “what” in “What works”? Develop and share a clear idea about the justice interventions. Delineate the interventions analytically from the dispute resolution mechanisms. Elaborate and deepen the knowledge about the interventions.
- Elaborate on the “what Works” modalities? Establish which are the important aspects of these modalities.
The good news is that the People-centered justice movement does not start from scratch on these critical topics. There is already progress in all three questions.
Specific challenge 1: Clarify the concept of justice outcomes
There is little agreement on what outcomes mean, but the discussion is gaining strength. The OECD elaborates on the outcomes and stresses the need for more research. 14 More research and development investments are needed to look beyond case outcomes and conceptualise and validate outcomes in a broader people-centred meaning. “Civil justice research must step back from narrow definitions of effectiveness that are limited to case outcomes and consider the broader, systemic effects of representation on individuals and those around them”. 15 The OECD also links positive and fair outcomes to key social objectives. “The ability of the legal and justice system to effectively respond and address those needs for all people and generate fair outcomes is critical to ensure well-being, equal opportunity and access to public services.” 16
Isabella Banks and Manon Huchet-Bodet define outcomes as “a positive result or change in well-being that a person with a legal problem achieves through the resolution process.” 17 Starting from desk research, Banks and Huchet-Bodet propose eight general justice outcomes and make a case for problem-specific outcomes. Using the case of intimate partner violence, they operationalise specific outcomes such as increased safety from intimate partner violence, improved physical health, increased confidence etc.
Laura Abel recognises that “no generally accepted metric for evaluating access to justice tools exists.” and proposes broader use of randomised control trial studies with outcome-based measures of justice interventions. 18
The TaskForce Justice report insists on investing in measuring the outcomes of justice journeys:
“It is essential to measure progress towards achieving fair outcomes. Justice systems need a new sense of accountability to the people they are designed to serve. An evidence-based approach that asks participants in judicial processes about their perceptions of fairness and their experience of the justice process is needed to hold providers to account and to give them feedback on the service they provide.” 19
The Canadian Action Committee on Access to Justice in Civil and Family Matters demands that the focus of the justice reform is on the outcomes that people want and receive.
“..at the end of the day, what people want most is a safe, healthy and productive life for themselves, their children and their loved ones. In a recent survey of public views about justice, one respondent defined justice as “access to society.” 20
Specific challenge 2: Define the justice interventions rigorously
The notion of justice interventions is new and still underdeveloped. It is more established in dispute resolution research, where scholars pay considerable attention to the various modes of interventions. In the practice of justice delivery, the focus is on larger-scale delivery models such as adjudication, mediation, arbitration, neutral evaluation etc. There is a need to look beyond the services and analyse what the dispute-resolution provider is doing to resolve problems. Hence, the field needs robust taxonomies and operationalisations. 21
In the Background paper, we distinguish various forms of adjudication, mediation, reconciliation etc. Interventions are the building blocks of the processes and rules that dispute-resolution providers apply to resolve legal problems. In the “Understanding Justice Needs: the Elephant in the Courtroom” report, HiiL insists that the justice solutions should be designed “with the fair end in mind” and that common legal problems have standard solutions. 22 Elements of solutions are protection (safe space), understanding, agreeing, etc. Furthermore, HiiL developed the concept of building blocks which together form interventions. Examples of building blocks are: documenting, containing (a problem), meeting, understanding, deciding etc. 23 An effective justice intervention will consist of a combination of interventions.
Challenge 3: Delve into the modalities of “What works”
Ample empirical legal studies explore diverse perspectives of the modalities of “what works”. The research and theoretical frameworks, methods and approaches used are so different that it is difficult to see them as part of a consistent field of study. Considering the difference in the research questions, it is not surprising that the results of such studies are quite dissimilar. For instance, using country data from the World Justice Project, Maurits Barendrecht found that interventions that include agreeing and complying correlate with more access to justice. 24 The same study finds that using adjudication is less promising for achieving access to justice. A study of 449 cases administered by four major providers of alternative dispute resolution services in the US found that 78% of the cases referred to mediation settle. Less likely to settle were cases with a potential of a large recovery and those for which one party did not have the financial incentive to settle. 25
The robustness of the findings of the studies of the “what works” modalities varies considerably. Anecdotal evidence from Bangladesh found that Shalish, the traditional method of dispute resolution, resolved between 80 and 95% of the disputes. Based on this high rate, the researchers claim that “Shalish is able to establish true justice within society”. 26 Using the more robust randomised controlled trial method, Greiner et al. found that 46% of the individuals assigned to attorneys had terminated their marriages in the proper legal venue, compared to 9% of the control group. 27 Another example of the use of RCTs is the study of Seron et al., which found that represented tenants in eviction proceedings receive better outcomes than non-represented clients. 28
4. Conclusion
Sound knowledge about “what works” in access to justice is needed more than ever. The analysis above gave an example of extracting “what works” insights from survey data. The results suggest that courts and lawyers deliver results but tend to be slow. Similarly, the intervention of deciding resolved disputes, however, is slower compared to other methods. The findings are interesting and invite more research to make this data useful and actionable. The analysis above highlighted how people-centred data could indicate “what works” in justice.
This Policy brief identifies one general and three specific areas where the People-centered justice movement needs to be strengthened through more attention, investments, data, research and development:
- Shift the focus towards the outcomes of justice
- Identify what it means “to work”
- What is the “What” in “What works”
- Elaborate on the “What works” modalities?
More data and advancements in the four areas above will make People-centered justice a considerably more feasible strategy to transform the justice sector. Actionable knowledge about interventions that deliver fair resolutions will empower decision-makers and service providers to continuously improve their methods and services in the search for better outcomes. The capacity of the justice sector to work in an evidence-based manner will improve. Moreover, the systemic gathering and evaluation of evidence about “What works” will firmly establish an evidence-based culture in the justice system. Lastly, the “What works” knowledge will become the constitutive ingredient of integrated People-centered justice programmes in which the legal needs of people and businesses are met by various services and interventions with proven effectiveness, fairness and ability to deliver positive outcomes.
5. Authors
View References
[1] https://www.hiil.org/wp-content/uploads/2019/09/Justice-Needs-and-Satisfaction-in-the-US-web.pdf
[2] https://www.hiil.org/wp-content/uploads/2019/09/JNS_Ethiopa_2020-1.pdf
[3] Abel, L. (2010). Evidence-Based Access To Justice. University of Pennsylvania Journal of Law and Social Change, 13, 295–313, p. 295
[4] Ibid, p. 302
[5] OECD. (2019). Equal Access to Justice for Inclusive Growth, p. 16
[6] OECD. (2021). OECD Framework and Good Practice Principles for People-Centred Justice, p. 20
[7] Ibid, p. 31
[8] Individual variables for location (urban-rural), gender, age, education, and income.
[9] Type of legal problem and perceived impact of the problem.
[10] Only location (urban-rural) has been removed from the model.
[11] See more at: https://dashboard.hiil.org/justice-dashboard-methodology/
[12] The discrete interventions are: 1) Provided advice; 2) Prepared documents; 3) Mediated between the parties; 4) Decided; 5) Referred; 5) Represented; 6) Emotional support; 7) Other; 9) Did nothing
[13] “Advice” is usually part of a broader set of interventions. For instance, lawyers usually advice, prepare documents, negotiate, reconcile, refer, and represent as part of a service. In this policy brief, “Advice” is operationalized and analysed as a discrete activity.
[14] OECD. (2021). OECD Framework and Good Practice Principles for People-Centred Justice, p. 76
[15] OECD. (2019). Equal Access to Justice for Inclusive Growth.
[16] OECD. (2021). OECD Framework and Good Practice Principles for People-Centred Justice, p. 12
[17] See https://dashboard.hiil.org/focusing-on-outcomes-for-people/
[18] Abel, L. (2009). Evidence-Based Access To Justice. University of Pennsylvania Journal of Law and Social Change, 13(3), p. 297
[19] The Task Force on Justice. (2019). Justice for All, p. 74
[20] Action Committee on Access to Justice in Civil and Family Matters. (2013). Access to Civil and Family Justice: A Roadmap for Change, p. 9
[21] See an overview of outcomes frameworks in Buttler, K. (2022). Legal Assistance Services Outcomes Framework – A Rapid Scoping Review.
[22] Barendrecht, M. (2018). The Elephant in the Courtroom Basic justice services for everyone.
[23] See more at https://dashboard.hiil.org/building-blocks/
[24] See https://www.hiil.org/news/making-people-agree-and-comply-perhaps/
[25] Brett, J. M., Barsness, Z. I., & Goldberg, S. B. (1996). The effectiveness of mediation: An independent analysis of cases handled by four major service providers. Negotiation Journal, 12(3), 259–269.
[26] Rahman, Z. (2022). Effectiveness of alternative dispute resolution (ADR) in rural area of Bangladesh: a study on village shalish system of Madhukhali Upazilla. Sociology International Journal, 6(3), 105–108.
[27] Greiner, D. J., Degnan, E. L., Ferriss, T., & Sommers, R. (2021). Using random assignment to measure court accessibility for low-income divorce seekers. Proceedings of the National Academy of Sciences, 118(14).
[28] Seron, C., Frankel, M., Van Ryzin, G., & Kovath, J. (2001). The Impact of Legal Counsel on Outcomes for Poor Tenants in New York City’s Housing Court: Results of a Randomized Experiment. Law & Society Review, (2), 419–434.
