What works in people-centered justice

HiiL POLICY BRIEF

How to figure out “What works”
in People-centered justice?

January 24, 2023

Photo by fauxels from Pexels

“What works” is a crucial yet still unanswered question in the justice field. This policy brief calls for urgent policy, research and funding action to address four aspects of the problem:

Introduction

At any time, more than 1.5 billion people worldwide have legal problems they cannot solve. Data from counties where HiiL worked make the problem more concrete. For instance, in the US, every year, there are 260 million significant and difficult-to-resolve legal problems. One hundred twenty million of them do not find a fair solution.1 In Ethiopia, there are 7.6 million legal problems yearly, of which 5.2 million remain unresolved.2

Legal problems occur everywhere around us. More likely than not, each of us will get into a dispute or grievance and will need some legal process to bring us to a fair resolution. But are there solutions to the many legal problems? Are these solutions equally accessible? Do we know which pathways to solutions work better than others?

To date, there is minimal data about the outcomes of legal problems. However, data is      emerging – mainly as the result of large-scale national legal needs surveys. These data persistently show that not enough legal problems are resolved fairly. The reasons for the      access to justice gap are complex, but they are not an excuse not to delve further into the issue. The justice needs of people often sit “in the blind spot” of justice policymakers and providers – unseen and unthought-of. In fact, the rules, procedures and stakeholders are often not concerned with the people who need fair resolutions.

To achieve UN’s SDG 16.3 “Equal access to justice for all”, we need diverse justice delivery models that can scale up massively in correspondence with the huge demand. How do we know if such solutions “work”? What does it mean that something “works” in justice? These seemingly simple questions should be coupled with additional questions such as “working where”, “working for whom”, “working for what sort of questions”, “working under what conditions”, etc.

Knowing “what works” is essential for every human and social activity area. Investing scarce and limited resources in actions with a higher chance of succeeding is prudent. A huge research and development industry informs the healthcare field about which treatments and procedures lead to results. Educational professionals are constantly searching for new methods and approaches that “work” in delivering better scholarly outputs and outcomes. It is different, however, in the field of access to justice, where services and interventions are rarely measured and evaluated to figure out “what works”.

“[..] a comparable evidence-based approach is notably absent from the many efforts to expand access to the justice system for people facing such civil legal problems as foreclosure, eviction, child custody disputes, domestic violence, or consumer fraud claims.” 3

To put it simply, “what works” are interventions that alter the outcomes of the justice journeys positively.” 4

The primary purpose of this policy brief is to make a case for and provide examples of evidence-based insights about “What works” and to stimulate decision-makers and researchers to continue further with this process.

1. The problem with knowing “what works” in the field of justice

What do we know about “What works” in justice? There is some progress, but the truth is that we do not know much about “What works” in justice. First, there is very little robust data in the field of access to justice. “Gold standard” randomised control trials are extremely rare. Fortunately, the growing empirical legal research and evidence-based policy movements aspire to bridge that gap, but it will take much time and resources to make real progress.

Second, there is very little agreement about what the  “outcomes of interventions” mean in the justice field. More often than not, justice outcomes are reduced to case outcomes. Administrative and court data at national and regional levels tell us whether a case is solved or is still pending, but not a lot more.

Third, the interventions for resolving legal problems are rarely viewed as packages of activities designed and implemented to solve specific problems. This makes it challenging to design appropriate research, gather data and answer the “what works” question in sufficient detail. For instance, adjudication alone is often a combination of various treatments such as advice, mediation and reconciliation, representation, and deciding the matter – not to mention all of these treatments occurring for different clients with different needs in different contexts. With so many ingredients, it is difficult to claim that adjudication “works” or “does not work”.

Fourth, the attribution problem makes it even more difficult to establish “what works” in the justice field. The question itself is based on a solid causal relationship between an intervention and a (positive) outcome. Randomised control trials based on experimental and quasi-experimental designs are not easy to implement in the justice field.

Nevertheless, “What works” is an essential part of people-centred justice.

The OECD calls for “developing and implementing policies and services that meet [people’s legal] needs, and removing barriers to access as part of a holistic vision for a people-centred justice ecosystem”.5  To achieve that, there is a need for “ongoing and co-ordinated research and evaluation conducted to identify and maintain an evidence base about what strategies “work” most effectively and cost-effectively, for whom, and in what circumstances to address legal and justice needs, including in the planning and delivery of legal and justice services”.6

“Implicit in designing cost-effective and effective services for the particular context is the need to have sound knowledge in relation to what strategies, interventions and services are most effective and cost-effective at addressing particular legal and justice needs. This in turn implies understanding what works in any circumstance, including for those people with multiple disadvantages and experiencing multiple problems who may not act in the rational way that legal systems may expect them to. In other words, identifying “what works” would need to take into account people, circumstances and emotions; pathways to resolution and support must be informed by how people experience legal and justice problems, and how they engage available pathways to address them.” 7

2. “What works” in delivering justice to the people: an exercise with legal needs data

HiiL has conducted legal needs surveys in more than 20 countries worldwide. More than 115,000 individuals from more than 25 countries were interviewed about their legal problems and needs for justice. We queried this rich source to understand more about what works in justice.

The main purpose of the analysis is to establish an interest in “what” works by demonstrating the importance and potential of people-centred data. To keep this policy paper short, we explain the data, methodology and detailed findings in a separate background paper. Here, we restate the main findings of three multivariate models that look at the legal needs dataset to answer the “What works” question.

Model 1: Courts and lawyers have significant resolution power but take a lot of time to solve legal problems

Model 1 analyses the association between the results of justice journeys and three sets of variables: party-related variables8, problem-related variables9 and process-related variables. The dependent variable in Model 1 is the resolution of the problem measured at four levels – “Completely resolved”, “Partially resolved”, “Ongoing”, and “Not resolved”. The dispute resolution process in this model is represented by the process perceived as most helpful in resolving the legal issue. We aggregated the many types of dispute resolution mechanisms into a few major categories – “Courts and lawyers”, “Police”, “Other organised procedures”, “Personal network”, and “Self-action”.

The main finding from this analysis is that using “Courts and lawyers” increases the likelihood that a problem is “Completely resolved” compared to other mechanisms. However, “Courts and lawyers” are slow – using “Courts and lawyers” greatly increases the risk that a legal problem is “Ongoing”. The use of “Police” and “Self-action” increases the risk that a problem is “Not resolved.”

Model 2: The users do not see considerable differences in the quality of the outcome delivered by various dispute resolution mechanisms  

Model 2 uses essentially the same set of independent variables 10, however, the dependent variable in this model is a composite measure of the quality of the outcome. The elements of this outcome variable are measures of distributive justice, restorative justice, enforcement and the potential of the result to resolve the underlying problem.11

The results imply no significant differences in the outcome quality of the various dispute resolution mechanisms. We compared all categories of dispute resolution mechanisms to the category of “Courts and lawyers” and found that the differences are not statistically significant. Men report higher satisfaction with the quality of the outcome than women. However, in the multivariate model, this difference is not significant.

There are statistically significant differences in the perceived quality of the outcome in some person- and problem-related variables. Compared to the youngest category, young adults (24-35) and middle age (35-55) report better outcomes. People with medium and high education report significantly better results than people without education. However, higher-income individuals report worse results than those with lower incomes. The justice journeys for employment and family-related legal problems receive lower outcome scores compared to land problems.

Model 3: Deciding a matter is an intervention that “works”, but the caveat is that deciding can take a long time

Model 3 explores the impact of dispute resolution interventions on the outcomes of legal problems. Interventions are the discrete activities that third parties apply to resolve problems. A third party can use one or more interventions in a dispute. Therefore, the variable is based on a multiple-choice selection – more than one of the following interventions (or lack of interventions) is possible.12

“Advice” is the most frequently used intervention in resolving legal problems related to land, employment and family issues. However, “Advice” seems to be the least effective of the interventions except for the “Other” and “Doing nothing” options. “Mediating/reconciling”, “Deciding the matter”, and even “Referring” increase considerably the chance that a problem is “Completely resolved” as compared to being “Not resolved”. The options “Doing nothing” or “Other” interventions increase considerably the risk that a problem remains “Not resolved”.

“Deciding” is the intervention that most considerably outperforms “Advice” 13 as a strategy to “Completely resolve” a problem. “Deciding” an issue substantively decreases the risk of the problem being “Ongoing”. “Representing” and “Doing nothing” both increase the likelihood that a problem will be “Ongoing” instead of “Completely resolved”.

“Deciding” also decreases the risk that a problem is “Not resolved” compared to the “Advising” intervention. “Doing nothing” significantly increases the risk that a problem ends as “Not resolved”. Compared to “Preparing documents”, “Mediating”, “Deciding” and even “Referring”, “Advice” significantly increases the risk that people consider a problem is “Not resolved”. “Preparing documents”, “Mediating”, “Deciding”, and “Referring” increase the likelihood that a legal problem is Ongoing.

3. A call to action to further the “What works” knowledge

The above empirical analysis does not yield a definitive conclusion about “What works”. It is not a big surprise that the findings are nuanced and do not always intuitively indicate straightforward solutions. In fact, the results point in many directions and invoke the need for more data and analysis.

As expected, the results of the „What works” analysis are ambivalent. This inconclusiveness is consistent with the diverse and rarely evidence-based theory and practice of access to justice. Most likely, there will never be silver bullet solutions that resolve legal problems regardless of the specifics of individuals, issues and contexts. Dispute resolution is highly context-specific. Legal, cultural, social and economic factors play large roles in how people encounter, process and resolve disputes. Questions such as “how”, “when”, “for whom”, and “where” are important parts of the pursuit of knowledge about the ways to resolve legal disputes. Nevertheless, the justice gap cannot be addressed without at least generally knowing “What works” in access to justice.

Regardless of the diversity and depth of the problem, there is an urgent need to shift the focus to the outcomes of justice processes. Three specific areas need the urgent attention of policymakers, service providers, researchers and access to justice donors. Our policy call is to invest attention, efforts and resources to make advances in the one general and three specific challenges listed below. The results of such investments will not immediately show “What works” in access to justice. However, small and big advancements will bring better tools and mechanisms to gather robust evidence and make justice more people-centred.

The good news is that the People-centered justice movement does not start from scratch on these critical topics. There is already progress in all three questions. 

Specific challenge 1: Clarify the concept of justice outcomes

There is little agreement on what outcomes mean, but the discussion is gaining strength. The OECD elaborates on the outcomes and stresses the need for more research. 14 More research and development investments are needed to look beyond case outcomes and conceptualise and validate outcomes in a broader people-centred meaning. “Civil justice research must step back from narrow definitions of effectiveness that are limited to case outcomes and consider the broader, systemic effects of representation on individuals and those around them”. 15 The OECD also links positive and fair outcomes to key social objectives. “The ability of the legal and justice system to effectively respond and address those needs for all people and generate fair outcomes is critical to ensure well-being, equal opportunity and access to public services.” 16

Isabella Banks and Manon Huchet-Bodet define outcomes as “a positive result or change in well-being that a person with a legal problem achieves through the resolution process.” 17 Starting from desk research, Banks and Huchet-Bodet propose eight general justice outcomes and make a case for problem-specific outcomes. Using the case of intimate partner violence, they operationalise specific outcomes such as increased safety from intimate partner violence, improved physical health, increased confidence etc.

Laura Abel recognises that “no generally accepted metric for evaluating access to justice tools exists.” and proposes broader use of randomised control trial studies with outcome-based measures of justice interventions. 18

The TaskForce Justice report insists on investing in measuring the outcomes of justice journeys:

“It is essential to measure progress towards achieving fair outcomes. Justice systems need a new sense of accountability to the people they are designed to serve. An evidence-based approach that asks participants in judicial processes about their perceptions of fairness and their experience of the justice process is needed to hold providers to account and to give them feedback on the service they provide.” 19

The Canadian Action Committee on Access to Justice in Civil and Family Matters demands that the focus of the justice reform is on the outcomes that people want and receive.

“..at the end of the day, what people want most is a safe, healthy and productive life for themselves, their children and their loved ones. In a recent survey of public views about justice, one respondent defined justice as “access to society.” 20

Specific challenge 2: Define the justice interventions rigorously

The notion of justice interventions is new and still underdeveloped. It is more established in dispute resolution research, where scholars pay considerable attention to the various modes of interventions. In the practice of justice delivery, the focus is on larger-scale delivery models such as adjudication, mediation, arbitration, neutral evaluation etc. There is a need to look beyond the services and analyse what the dispute-resolution provider is doing to resolve problems. Hence, the field needs robust taxonomies and operationalisations. 21

In the Background paper, we distinguish various forms of adjudication, mediation, reconciliation etc. Interventions are the building blocks of the processes and rules that dispute-resolution providers apply to resolve legal problems. In the “Understanding Justice Needs: the Elephant in the Courtroom” report, HiiL insists that the justice solutions should be designed “with the fair end in mind” and that common legal problems have standard solutions. 22 Elements of solutions are protection (safe space), understanding, agreeing, etc.  Furthermore, HiiL developed the concept of building blocks which together form interventions. Examples of building blocks are: documenting, containing (a problem), meeting, understanding, deciding etc. 23 An effective justice intervention will consist of a combination of interventions.

Challenge 3: Delve into the modalities of “What works”

Ample empirical legal studies explore diverse perspectives of the modalities of “what works”. The research and theoretical frameworks, methods and approaches used are so different that it is difficult to see them as part of a consistent field of study. Considering the difference in the research questions, it is not surprising that the results of such studies are quite dissimilar. For instance, using country data from the World Justice Project, Maurits Barendrecht found that interventions that include agreeing and complying correlate with more access to justice. 24 The same study finds that using adjudication is less promising for achieving access to justice. A study of 449 cases administered by four major providers of alternative dispute resolution services in the US found that 78% of the cases referred to mediation settle. Less likely to settle were cases with a potential of a large recovery and those for which one party did not have the financial incentive to settle. 25

The robustness of the findings of the studies of the “what works” modalities varies considerably. Anecdotal evidence from Bangladesh found that Shalish, the traditional method of dispute resolution, resolved between 80 and 95% of the disputes. Based on this high rate, the researchers claim that “Shalish is able to establish true justice within society”. 26 Using the more robust randomised controlled trial method, Greiner et al. found that 46% of the individuals assigned to attorneys had terminated their marriages in the proper legal venue, compared to 9% of the control group. 27 Another example of the use of RCTs is the study of Seron et al., which found that represented tenants in eviction proceedings receive better outcomes than non-represented clients. 28

4. Conclusion

Sound knowledge about “what works” in access to justice is needed more than ever. The analysis above gave an example of extracting “what works” insights from survey data. The results suggest that courts and lawyers deliver results but tend to be slow. Similarly, the intervention of deciding resolved disputes, however, is slower compared to other methods. The findings are interesting and invite more research to make this data useful and actionable. The analysis above highlighted how people-centred data could indicate “what works” in justice.

This Policy brief identifies one general and three specific areas where the People-centered justice movement needs to be strengthened through more attention, investments, data, research and development:

More data and advancements in the four areas above will make People-centered justice a considerably more feasible strategy to transform the justice sector. Actionable knowledge about interventions that deliver fair resolutions will empower decision-makers and service providers to continuously improve their methods and services in the search for better outcomes. The capacity of the justice sector to work in an evidence-based manner will improve. Moreover, the systemic gathering and evaluation of evidence about “What works” will firmly establish an evidence-based culture in the justice system. Lastly, the “What works” knowledge will become the constitutive ingredient of integrated People-centered justice programmes in which the legal needs of people and businesses are met by various services and interventions with proven effectiveness, fairness and ability to deliver positive outcomes.

5. Authors

This Policy Brief was written by Dr. Martin Gramatikov Programme Director Kenya, Ukraine, Colombia, South Sudan.

[1] https://www.hiil.org/wp-content/uploads/2019/09/Justice-Needs-and-Satisfaction-in-the-US-web.pdf

[2] https://www.hiil.org/wp-content/uploads/2019/09/JNS_Ethiopa_2020-1.pdf

[3] Abel, L. (2010). Evidence-Based Access To Justice. University of Pennsylvania Journal of Law and Social Change, 13, 295–313, p. 295

[4] Ibid, p. 302

[5] OECD. (2019). Equal Access to Justice for Inclusive Growth, p. 16

[6] OECD. (2021). OECD Framework and Good Practice Principles for People-Centred Justice, p. 20

[7] Ibid, p. 31

[8] Individual variables for location (urban-rural), gender, age, education, and income.

[9] Type of legal problem and perceived impact of the problem.

[10] Only location (urban-rural) has been removed from the model.

[11] See more at: https://dashboard.hiil.org/justice-dashboard-methodology/

[12] The discrete interventions are: 1) Provided advice; 2) Prepared documents; 3) Mediated between the parties; 4) Decided; 5) Referred; 5) Represented; 6) Emotional support; 7) Other; 9) Did nothing

[13] “Advice” is usually part of a broader set of interventions. For instance, lawyers usually advice, prepare documents, negotiate, reconcile, refer, and represent as part of a service. In this policy brief, “Advice” is operationalized and analysed as a discrete activity.

[14] OECD. (2021). OECD Framework and Good Practice Principles for People-Centred Justice, p. 76

[15] OECD. (2019). Equal Access to Justice for Inclusive Growth.

[16] OECD. (2021). OECD Framework and Good Practice Principles for People-Centred Justice, p. 12

[17] See https://dashboard.hiil.org/focusing-on-outcomes-for-people/

[18] Abel, L. (2009). Evidence-Based Access To Justice. University of Pennsylvania Journal of Law and Social Change, 13(3), p. 297

[19] The Task Force on Justice. (2019). Justice for All, p. 74

[20] Action Committee on Access to Justice in Civil and Family Matters. (2013). Access to Civil and Family Justice: A Roadmap for Change, p. 9

[21] See an overview of outcomes frameworks in Buttler, K. (2022). Legal Assistance Services Outcomes Framework – A Rapid Scoping Review.

[22] Barendrecht, M. (2018). The Elephant in the Courtroom Basic justice services for everyone.

[23] See more at https://dashboard.hiil.org/building-blocks/

[24] See https://www.hiil.org/news/making-people-agree-and-comply-perhaps/

[25] Brett, J. M., Barsness, Z. I., & Goldberg, S. B. (1996). The effectiveness of mediation: An independent analysis of cases handled by four major service providers. Negotiation Journal, 12(3), 259–269.

[26] Rahman, Z. (2022). Effectiveness of alternative dispute resolution (ADR) in rural area of Bangladesh: a study on village shalish system of Madhukhali Upazilla. Sociology International Journal, 6(3), 105–108.

[27] Greiner, D. J., Degnan, E. L., Ferriss, T., & Sommers, R. (2021). Using random assignment to measure court accessibility for low-income divorce seekers. Proceedings of the National Academy of Sciences, 118(14).

[28] Seron, C., Frankel, M., Van Ryzin, G., & Kovath, J. (2001). The Impact of Legal Counsel on Outcomes for Poor Tenants in New York City’s Housing Court: Results of a Randomized Experiment. Law & Society Review, (2), 419–434.

Table of Contents

Referrences:

Background paper

BACKGROUND PAPER

How to figure out “What works”
in People-centered justice?

January 24, 2023

Photo by fauxels from Pexels

This paper provides inputs for a policy brief about “what works in justice”. The primary purpose of the background paper is to serve as an example for analysing data to understand “what works” in resolving disputes. Together with the policy brief, its long-term objective is to stimulate policymakers and service providers to gather data systematically to identify effective and scalable approaches for delivering access to justice.

The paper starts with theoretical deliberations about access to justice. We look at the linkages between inputs, processes and results of justice processes. Then three data models are tested with people-centred justice data obtained through survey research in diverse jurisdictions. The results of the models are discussed in each section, and the overall implications are elaborated in the policy brief.

Problem background

Success in justice delivery is not random, but we do not know much about what affects the chances of success and the risks of failure. Formal and informal justice systems and mechanisms resolve certain legal problems better than others. For instance, HiiL’s research (refs.) consistently finds that problems with lower impact are more frequently resolved than problems with higher impact. Similar findings are reported by Ter Voert and Hoekstra. 1 The category of problems also matters – land and crimes are less frequently resolved than other issues. Certain people are less likely to report positive results and outcomes of their legal problems – i.e. the legal problems of urban residents usually are more likely to be resolved compared to the problems of rural residents.

However, the available evidence about “what works” and “what doesn’t work” is minimal. Access to justice is very often designed and delivered on the basis of intuition. Very often, positive interventions are entirely based on normative criteria. The OECD observes, “Given the relative paucity of empirical evidence about which interventions result in effective or meaningful access to justice, decision-makers rely on a range of guiding principles, indicators and other criteria to guide the policy development process. These criteria integrate theories about access to justice and be seen as predictors of, or proxies for, “what works”. Criteria for what works can also be derived from emerging promising practices in promoting accessibility and people-centricity of legal and justice services.” 2 The question is how to “capture”, understand and scale such promising practices. In this paper, we will look at data reflecting the experiences and perceptions of users of justice to understand “what works”.

Our fundamental hypothesis is that three sets of factors influence the results of dispute resolution. The first set of factors is related to the parties involved. The second set concerns the problem’s type, gravity and impact. The third set of factors is related to the quality of the dispute resolution process. Below we extend the 3P (party, problem, process) model.

These third parties employ diverse approaches to resolving disputes. Some of the most often used approaches are advice, adjudication, mediation, reconciliation, and referral to other options for dispute resolution. In practice, relatively rarely do third parties use highly structured dispute resolution mechanisms such as adjudication or formalised mediation. 

An essential tenet of this background paper and the related policy brief is that some interventions are more effective in resolving legal problems. Such interventions “work” – they are more accessible, fair, effective, efficient and fair. The meaning of interventions, however, is not widely accepted in justice research and practice. Under interventions, we understand discrete parts of the overall dispute resolution process, such as

In a way, the interventions are the main building blocks of dispute resolution. There are many more “Lego parts” that can be added to this list. 6 Many of these building blocks occur together with others and form new blocks. For instance, advice is often part of a broader service which includes document preparation, filing suits, motions and requests, representation, appeal etc.

To sum up, the theoretical foundation of “what works” is based on three broad sets of factors that are believed to explain a significant portion of the variation of the outcomes. The sets of factors are: 1)  parameters of the problems, 2) characteristics of the parties, and 3) dispute resolution interventions. Many other factors affect the outcomes of justice, and these factors are not unimportant. We invite others to continue the research efforts to understand the ultimate question of “what works in justice”.

Data and methods

Cross-sectional survey research data collected with HiiL’s Justice Needs and Satisfaction instrument are used for this analysis.7 Three countries ( Ethiopia, Mali and Uganda) participated with two data sets with unrelated samples. Model 1 and Model 2 are based on a dataset with data from:

Model 3 uses data from Ethiopia, Burkina Faso and Niger. In these 3 surveys, the concept of interventions was added to the research instrument and hence made additional research available.

Limitations of the data and this analysis

The results of this analysis must be considered carefully and should be interpreted in light of the dataset and analysis limitations, which include the following:

Model 1: Resolution of the legal problems

Dependent variable in Model one is the resolution of the problem. Resolution is measured at four levels – “Completely resolved”, “Partially resolved”, “Ongoing” and “Not resolved”. A multinomial logit model is used to explore the relationships between the levels of resolution and the independent variables. All coefficients from the table below are expressed as the likelihood of achieving the particular outcome versus the outcome level “Completely resolved”. The relative risk ratio represents the regression coefficient because of its more intuitive value.

Socio-demographic variables in the model are gender, age, education,  and location (urban-rural). The problem category and its impact reflect the properties of the issue. The key independent variable of interest in Models 1 and 2 is the dispute resolution process which was considered the most useful mechanism for resolving the particular problem. The many types of mechanisms are aggregated into several major categories – “Courts and lawyers”; “Police”, “Other organised procedures”, “Personal network”, and “Self-action”. Other organised procedures include formal and informal mechanisms for resolving legal problems which do not fall in the category of “Courts of lawyers”. Most often, these are various community-level dispute-resolution mechanisms. The “Personal network” category combines family members, friends, and neighbours.

Relative to the Completely resolved category

Model 1 is statistically significant, which means that the independent variables are related to the change in problem resolution. In summary, the model tells us that younger people are less likely to resolve their problems than older people. An increase in education increases the chance of resolving legal issues. On the other hand, urban people (who are more educated) are less likely to resolve their problems. The more impactful problems are less likely to be “Completely resolved” or “Partially resolved”. Surprisingly, land problems, which are more serious on average, are more often “Completely” or “Partially resolved”. “Self-action” reduces the likelihood of a problem being “Completely resolved”. The engagement of “Courts and lawyers” increases the likelihood of a problem being “Completely resolved”.

Looking at the socio-demographic, we see that women are less likely than men to have their problems “Completely resolved” instead of being “Not resolved”. People in middle adulthood (40-64) are more likely to have their problems solved compared to young people. Individuals with a high degree of education are more likely to have their problems solved compared to individuals with no education. Low-income people are much less likely to resolve their problems “Completely”. Individuals with medium and high education are significantly more likely to report their legal problems as “Partially resolved” than those without education.

Urban people are more likely to have their problems “Ongoing” than rural people. Young adulthood (25-39) and middle adulthood (40-64) individuals are more likely to report problems as “Ongoing” than young (18-24) individuals. People with a high level of education are more likely to have their problems “Ongoing” than individuals without education. Lower-middle income increases the likelihood of a problem being “Ongoing” instead of “Not resolved” compared to the low-income category.

In the characteristics of the problem, we see that employment and family problems are less likely to be “Completely resolved” or “Ongoing” instead of “Not resolved” than land problems. The higher impact is associated with fewer “Completely” and “Partially resolved” problems. “Self-action” reduces the likelihood of a problem being “Partially resolved” than “Not resolved”.. Problems with higher impact are more likely to be “Partially resolved”, “Ongoing”, or “Not resolved” than problems with lower impact.

In the process part, we see that using “Courts and lawyers” increases the likelihood that a problem is “Completely resolved” compared to the other mechanisms. 8

Estimating the marginal effects of the multinomial regression allows us to analyse the probabilities, which are easier to interpret. The marginal effects show some interesting trends:

Completely resolved problems

“Other organised procedures” are associated with a 5% increased likelihood that the problem is “Completely resolved” than “Courts and lawyers”.

Partially resolved

“Other organised procedures” are associated with a 2% increased likelihood that the problem is “Partially resolved” compared to “Courts and lawyers”. When the most useful process is “Personal network” there is a 2% increased likelihood that the problem is “Partially resolved” compared to “Courts and lawyers”.

Ongoing

All options other than “Courts and lawyers” have a lower risk that the problem is still “Ongoing”: “Police” (12% decrease), “Other organised procedures” (11% decrease), “Personal network” (13% decrease), “Self-action” (10% decrease).

Not resolved

Compared to “Courts and lawyers”, “Police” is associated with an 8% increased likelihood that a problem is “Not resolved”. Compared to “Courts and lawyers”, the use of “Other organised procedure”  is associated with a 6% increased likelihood that a problem is “Not resolved”. Compared to “Courts and lawyers”, the use of “Personal network” is associated with a 12% increased likelihood that a problem is “Not resolved”. Compared to “Courts and lawyers”, the use of “Self-action” is associated with a 13% increased likelihood that a problem is “Not resolved”.

The key take-away from Model 1 is that “Courts and lawyers” resolve problems but they are also slow. Using “Courts and lawyers” is related to a significant decrease in the risk of “Not resolved”. All other sources increase that risk. But using “Courts and lawyers” has a significant drawback – the option increases the risk that a problem is “Ongoing”.

Model 2: Composite measure of fair process, fair result and costs of resolving a problem

In Model 2, the dependent variable is a composite measure of the quality of dispute resolution. The elements of this variable are measures of procedural justice, distributive justice, restorative justice, enforcement, the ability of the result to resolve the problem, and the costs of the paths to justice. All elements of the quality of the outcome are measured with a 5-point Likert scale. In the next step, all variables are aggregated using a simple mean function. Linear regression is used to explore the effect of the independent variables on the quality of dispute resolution. 

The key independent variable of interest is the dispute resolution process which was perceived as the most useful mechanism for resolving the particular problem. Similar to Model 1, the other independent variables reflect characteristics of the party (gender, age, education and income) as well as the type of the problem and its perceived impact on the respondent.

The results demonstrate that there are no significant differences in the outcome quality across the various dispute resolution mechanisms. All categories of dispute resolution mechanisms are compared to “Courts and lawyers”, and the differences are not statistically significant. Men report higher satisfaction with the quality of the outcome than women (3.24 v 3.11), however, in the multivariate model, this difference is not significant.

There are statistically significant differences in the perceived quality of the outcome in some of the person and problem-related variables. Young people (24-35) and middle age (35-55) report better outcomes than the very young and the senior respondents. People with medium and high education report significantly better results than people without education. However, individuals with higher incomes report worse results than people with lower education. The justice journeys for employment and family-related legal problems receive lower outcome scores compared to land problems. The impact of the problem is not related to the outcome in this model

Model 3: Focus on the interventions

Model 3 analyses the interventions that the third parties apply to resolve legal problems. In 3 of the countries (Ethiopia, Niger and Burkina Faso, the datasets contain information about the discrete interventions applied by the third parties). Similar to the previous models, for simplicity, we focus on land, employment and family problems. 

The dataset is structured a little differently than Models 1 and 2, where each record corresponds to one respondent. In Model 3, one respondent (or problem) could appear on more than one row because more than one intervention can be applied to resolve a problem.

Dispute resolution mechanisms and interventions

In Model 3, we explore two sets of variables – the dispute resolution mechanisms and the interventions. Dispute resolution mechanisms are the types of justice journeys as defined by their main mode of resolution. In Model 3, we distinguish between the following dispute resolution mechanisms:

The interventions are the activities that the neutrals perform as part of their strategy to resolve a legal problem. A third party can apply one or more interventions. Therefore the variable is multiple choice – more than one of the following interventions (or lack of interventions) are possible:

The results of a multinomial logit regression model are provided below. Similar to Model 1 the coefficients are expressed in relative risk ratios (RRR). 9

From the regression results, we see that the selected explanatory variables explain mostly the difference between the “Ongoing” and “Unresolved” resolution levels compared to “Completely resolved”. The use of “Formal adjudication mechanisms” significantly increases the likelihood that a legal problem is “Completely resolved” instead of “Not resolved” compared to other dispute resolution mechanisms such as “Community justice mechanisms”, “Social environment”, “Police and other public authorities”, and “Negotiation”. The difference between “Formal adjudication” and the involvement of “Legal professionals” is not statistically significant. “Negotiation” or “Other DRM” increase the risk that a problem remains “Unresolved”.

“Formal adjudication”, however, is significantly more likely to lead to “Ongoing” problems. The use of “Community justice mechanisms”, “Social environment”, and “Legal professionals” decreases the risk that a problem is “Ongoing” instead of “Unresolved”. 

“Advice” is the most frequently used intervention for resolving legal problems related to land, employment and family issues. However, “Advice” seems to be the least effective of the interventions, excluding the options “Other” and “Doing nothing”. “Mediating/reconciling”, “Deciding the matter”, and even “Referring” increase considerably the chance that a problem will be “Completely resolved” as compared to “Not resolved”. “Doing nothing” or “Other” increase massively the risk that a problem remains unresolved.

“Deciding” is the intervention which most considerably outperforms “Advice” as a means to “Completely resolve” a problem (coefficient 0.30) instead of the problem being “Unresolved”. “Deciding” decreases substantively the risk that the problem is “Pending” compared to “Advising”. “Representing” and “Doing nothing” both increase the likelihood that a problem will be “Pending” instead of “Completely resolved”.

“Deciding” also decreases the risk that a problem is “Not resolved” compared to “Advising”. “Doing nothing” increases the risk that a problem is “Not resolved”. Compared to “Preparing documents”, “Mediating”, “Deciding”, and even “Referring”, “Advice” significantly increases the risk that a problem will be considered “Not resolved”, instead of “Ongoing”. To put it differently, “Advice” is more likely to ‘place’ a problem into the category of “Not resolved”.  “Preparing documents”, “Mediating”, “Deciding” and even “Referring” increase the likelihood that a legal problem is “Ongoing”.

Besides the interventions and dispute resolution processes, some other characteristics of the legal problems impact the resolution status. Problems in Ethiopia are more likely to be “Completely resolved” than “Not resolved”. Employment problems are more likely to be “Not resolved” than to land problems. Land problems are more likely to be “Ongoing” instead of “Completely resolved” compared with family problems. The less impactful problems are more often “Completely resolved”. Impactful problems are less often “Completely resolved”. The most impactful problems are most likely to be “Ongoing”. Women are less likely to resolve their problems than men.

Reference to the policy brief

The broader implications of the results continue in the policy brief.

Authors

This Policy Brief was written, Dr. Martin Gramatikov, Programme Director Kenya, Ukraine, Colombia, South Sudan.

[1] Voert, M. J. ter, & Hoekstra, M. S. (2020). Geschilbeslechtingsdelta 2019.

[2] OECD. (2019). Equal Access to Justice for Inclusive Growth. p. 113

[3] Pleasence, P., & Coumarelos, C. (2014). Reshaping legal assistance services: building on the evidence base. Retrieved from http://www.lawfoundation.net.au/ljf/site/articleIDs/D76E53BB842CB7B1CA257D7B000D5173/$file/Reshaping_legal_assistance_services_web.pdf

[4] Ibid.

[5] A neutral third party decides the outcome of the dispute based on the relevant rules and evidence.

[6] For more elaborated version of the building blocks see https://dashboard.hiil.org/building_blocks

[7] More details about the surveys are available at: https://dashboard.hiil.org/justice-dashboard-methodology/

[8] The use of dispute resolution mechanism in Model 1 and Model 2 is relative because the actual variable is based on the question – “Which was the most useful source of dispute resolution?”

[9] RRR indicates how the risk of the outcome falling in the comparison group compares to the risk of the outcome falling in the referent group changes with the variable in question. For instance, a relative risk ratio of 2.49 for “Community justice mechanism” in the level “No, the problem is not resolved and I am no longer taking actions to solve it”, indicates that using a “Community justice mechanism” increases the likelihood that a problem is at this level instead of “Yes, completely resolved” (the reference level), compared with the option “Adjudication”.

Justice Dashboard

Justice Dashboard