This paper provides inputs for a policy brief about “what works in justice”. The primary purpose of the background paper is to serve as an example for analysing data to understand “what works” in resolving disputes. Together with the policy brief, its long-term objective is to stimulate policymakers and service providers to gather data systematically to identify effective and scalable approaches for delivering access to justice.
The paper starts with theoretical deliberations about access to justice. We look at the linkages between inputs, processes and results of justice processes. Then three data models are tested with people-centred justice data obtained through survey research in diverse jurisdictions. The results of the models are discussed in each section, and the overall implications are elaborated in the policy brief.
Success in justice delivery is not random, but we do not know much about what affects the chances of success and the risks of failure. Formal and informal justice systems and mechanisms resolve certain legal problems better than others. For instance, HiiL’s research (refs.) consistently finds that problems with lower impact are more frequently resolved than problems with higher impact. Similar findings are reported by Ter Voert and Hoekstra. 1 The category of problems also matters – land and crimes are less frequently resolved than other issues. Certain people are less likely to report positive results and outcomes of their legal problems – i.e. the legal problems of urban residents usually are more likely to be resolved compared to the problems of rural residents.
However, the available evidence about “what works” and “what doesn’t work” is minimal. Access to justice is very often designed and delivered on the basis of intuition. Very often, positive interventions are entirely based on normative criteria. The OECD observes, “Given the relative paucity of empirical evidence about which interventions result in effective or meaningful access to justice, decision-makers rely on a range of guiding principles, indicators and other criteria to guide the policy development process. These criteria integrate theories about access to justice and be seen as predictors of, or proxies for, “what works”. Criteria for what works can also be derived from emerging promising practices in promoting accessibility and people-centricity of legal and justice services.” 2 The question is how to “capture”, understand and scale such promising practices. In this paper, we will look at data reflecting the experiences and perceptions of users of justice to understand “what works”.
Our fundamental hypothesis is that three sets of factors influence the results of dispute resolution. The first set of factors is related to the parties involved. The second set concerns the problem’s type, gravity and impact. The third set of factors is related to the quality of the dispute resolution process. Below we extend the 3P (party, problem, process) model.
These third parties employ diverse approaches to resolving disputes. Some of the most often used approaches are advice, adjudication, mediation, reconciliation, and referral to other options for dispute resolution. In practice, relatively rarely do third parties use highly structured dispute resolution mechanisms such as adjudication or formalised mediation.
An essential tenet of this background paper and the related policy brief is that some interventions are more effective in resolving legal problems. Such interventions “work” – they are more accessible, fair, effective, efficient and fair. The meaning of interventions, however, is not widely accepted in justice research and practice. Under interventions, we understand discrete parts of the overall dispute resolution process, such as
In a way, the interventions are the main building blocks of dispute resolution. There are many more “Lego parts” that can be added to this list. 6 Many of these building blocks occur together with others and form new blocks. For instance, advice is often part of a broader service which includes document preparation, filing suits, motions and requests, representation, appeal etc.
To sum up, the theoretical foundation of “what works” is based on three broad sets of factors that are believed to explain a significant portion of the variation of the outcomes. The sets of factors are: 1) parameters of the problems, 2) characteristics of the parties, and 3) dispute resolution interventions. Many other factors affect the outcomes of justice, and these factors are not unimportant. We invite others to continue the research efforts to understand the ultimate question of “what works in justice”.
Cross-sectional survey research data collected with HiiL’s Justice Needs and Satisfaction instrument are used for this analysis.7 Three countries ( Ethiopia, Mali and Uganda) participated with two data sets with unrelated samples. Model 1 and Model 2 are based on a dataset with data from:
Model 3 uses data from Ethiopia, Burkina Faso and Niger. In these 3 surveys, the concept of interventions was added to the research instrument and hence made additional research available.
The results of this analysis must be considered carefully and should be interpreted in light of the dataset and analysis limitations, which include the following:
Dependent variable in Model one is the resolution of the problem. Resolution is measured at four levels – “Completely resolved”, “Partially resolved”, “Ongoing” and “Not resolved”. A multinomial logit model is used to explore the relationships between the levels of resolution and the independent variables. All coefficients from the table below are expressed as the likelihood of achieving the particular outcome versus the outcome level “Completely resolved”. The relative risk ratio represents the regression coefficient because of its more intuitive value.
Socio-demographic variables in the model are gender, age, education, and location (urban-rural). The problem category and its impact reflect the properties of the issue. The key independent variable of interest in Models 1 and 2 is the dispute resolution process which was considered the most useful mechanism for resolving the particular problem. The many types of mechanisms are aggregated into several major categories – “Courts and lawyers”; “Police”, “Other organised procedures”, “Personal network”, and “Self-action”. Other organised procedures include formal and informal mechanisms for resolving legal problems which do not fall in the category of “Courts of lawyers”. Most often, these are various community-level dispute-resolution mechanisms. The “Personal network” category combines family members, friends, and neighbours.
Model 1 is statistically significant, which means that the independent variables are related to the change in problem resolution. In summary, the model tells us that younger people are less likely to resolve their problems than older people. An increase in education increases the chance of resolving legal issues. On the other hand, urban people (who are more educated) are less likely to resolve their problems. The more impactful problems are less likely to be “Completely resolved” or “Partially resolved”. Surprisingly, land problems, which are more serious on average, are more often “Completely” or “Partially resolved”. “Self-action” reduces the likelihood of a problem being “Completely resolved”. The engagement of “Courts and lawyers” increases the likelihood of a problem being “Completely resolved”.
Looking at the socio-demographic, we see that women are less likely than men to have their problems “Completely resolved” instead of being “Not resolved”. People in middle adulthood (40-64) are more likely to have their problems solved compared to young people. Individuals with a high degree of education are more likely to have their problems solved compared to individuals with no education. Low-income people are much less likely to resolve their problems “Completely”. Individuals with medium and high education are significantly more likely to report their legal problems as “Partially resolved” than those without education.
Urban people are more likely to have their problems “Ongoing” than rural people. Young adulthood (25-39) and middle adulthood (40-64) individuals are more likely to report problems as “Ongoing” than young (18-24) individuals. People with a high level of education are more likely to have their problems “Ongoing” than individuals without education. Lower-middle income increases the likelihood of a problem being “Ongoing” instead of “Not resolved” compared to the low-income category.
In the characteristics of the problem, we see that employment and family problems are less likely to be “Completely resolved” or “Ongoing” instead of “Not resolved” than land problems. The higher impact is associated with fewer “Completely” and “Partially resolved” problems. “Self-action” reduces the likelihood of a problem being “Partially resolved” than “Not resolved”.. Problems with higher impact are more likely to be “Partially resolved”, “Ongoing”, or “Not resolved” than problems with lower impact.
In the process part, we see that using “Courts and lawyers” increases the likelihood that a problem is “Completely resolved” compared to the other mechanisms. 8
Estimating the marginal effects of the multinomial regression allows us to analyse the probabilities, which are easier to interpret. The marginal effects show some interesting trends:
“Other organised procedures” are associated with a 5% increased likelihood that the problem is “Completely resolved” than “Courts and lawyers”.
“Other organised procedures” are associated with a 2% increased likelihood that the problem is “Partially resolved” compared to “Courts and lawyers”. When the most useful process is “Personal network” there is a 2% increased likelihood that the problem is “Partially resolved” compared to “Courts and lawyers”.
All options other than “Courts and lawyers” have a lower risk that the problem is still “Ongoing”: “Police” (12% decrease), “Other organised procedures” (11% decrease), “Personal network” (13% decrease), “Self-action” (10% decrease).
Compared to “Courts and lawyers”, “Police” is associated with an 8% increased likelihood that a problem is “Not resolved”. Compared to “Courts and lawyers”, the use of “Other organised procedure” is associated with a 6% increased likelihood that a problem is “Not resolved”. Compared to “Courts and lawyers”, the use of “Personal network” is associated with a 12% increased likelihood that a problem is “Not resolved”. Compared to “Courts and lawyers”, the use of “Self-action” is associated with a 13% increased likelihood that a problem is “Not resolved”.
The key take-away from Model 1 is that “Courts and lawyers” resolve problems but they are also slow. Using “Courts and lawyers” is related to a significant decrease in the risk of “Not resolved”. All other sources increase that risk. But using “Courts and lawyers” has a significant drawback – the option increases the risk that a problem is “Ongoing”.
In Model 2, the dependent variable is a composite measure of the quality of dispute resolution. The elements of this variable are measures of procedural justice, distributive justice, restorative justice, enforcement, the ability of the result to resolve the problem, and the costs of the paths to justice. All elements of the quality of the outcome are measured with a 5-point Likert scale. In the next step, all variables are aggregated using a simple mean function. Linear regression is used to explore the effect of the independent variables on the quality of dispute resolution.
The key independent variable of interest is the dispute resolution process which was perceived as the most useful mechanism for resolving the particular problem. Similar to Model 1, the other independent variables reflect characteristics of the party (gender, age, education and income) as well as the type of the problem and its perceived impact on the respondent.
The results demonstrate that there are no significant differences in the outcome quality across the various dispute resolution mechanisms. All categories of dispute resolution mechanisms are compared to “Courts and lawyers”, and the differences are not statistically significant. Men report higher satisfaction with the quality of the outcome than women (3.24 v 3.11), however, in the multivariate model, this difference is not significant.
There are statistically significant differences in the perceived quality of the outcome in some of the person and problem-related variables. Young people (24-35) and middle age (35-55) report better outcomes than the very young and the senior respondents. People with medium and high education report significantly better results than people without education. However, individuals with higher incomes report worse results than people with lower education. The justice journeys for employment and family-related legal problems receive lower outcome scores compared to land problems. The impact of the problem is not related to the outcome in this model
Model 3 analyses the interventions that the third parties apply to resolve legal problems. In 3 of the countries (Ethiopia, Niger and Burkina Faso, the datasets contain information about the discrete interventions applied by the third parties). Similar to the previous models, for simplicity, we focus on land, employment and family problems.
The dataset is structured a little differently than Models 1 and 2, where each record corresponds to one respondent. In Model 3, one respondent (or problem) could appear on more than one row because more than one intervention can be applied to resolve a problem.
In Model 3, we explore two sets of variables – the dispute resolution mechanisms and the interventions. Dispute resolution mechanisms are the types of justice journeys as defined by their main mode of resolution. In Model 3, we distinguish between the following dispute resolution mechanisms:
The interventions are the activities that the neutrals perform as part of their strategy to resolve a legal problem. A third party can apply one or more interventions. Therefore the variable is multiple choice – more than one of the following interventions (or lack of interventions) are possible:
The results of a multinomial logit regression model are provided below. Similar to Model 1 the coefficients are expressed in relative risk ratios (RRR). 9
From the regression results, we see that the selected explanatory variables explain mostly the difference between the “Ongoing” and “Unresolved” resolution levels compared to “Completely resolved”. The use of “Formal adjudication mechanisms” significantly increases the likelihood that a legal problem is “Completely resolved” instead of “Not resolved” compared to other dispute resolution mechanisms such as “Community justice mechanisms”, “Social environment”, “Police and other public authorities”, and “Negotiation”. The difference between “Formal adjudication” and the involvement of “Legal professionals” is not statistically significant. “Negotiation” or “Other DRM” increase the risk that a problem remains “Unresolved”.
“Formal adjudication”, however, is significantly more likely to lead to “Ongoing” problems. The use of “Community justice mechanisms”, “Social environment”, and “Legal professionals” decreases the risk that a problem is “Ongoing” instead of “Unresolved”.
“Advice” is the most frequently used intervention for resolving legal problems related to land, employment and family issues. However, “Advice” seems to be the least effective of the interventions, excluding the options “Other” and “Doing nothing”. “Mediating/reconciling”, “Deciding the matter”, and even “Referring” increase considerably the chance that a problem will be “Completely resolved” as compared to “Not resolved”. “Doing nothing” or “Other” increase massively the risk that a problem remains unresolved.
“Deciding” is the intervention which most considerably outperforms “Advice” as a means to “Completely resolve” a problem (coefficient 0.30) instead of the problem being “Unresolved”. “Deciding” decreases substantively the risk that the problem is “Pending” compared to “Advising”. “Representing” and “Doing nothing” both increase the likelihood that a problem will be “Pending” instead of “Completely resolved”.
“Deciding” also decreases the risk that a problem is “Not resolved” compared to “Advising”. “Doing nothing” increases the risk that a problem is “Not resolved”. Compared to “Preparing documents”, “Mediating”, “Deciding”, and even “Referring”, “Advice” significantly increases the risk that a problem will be considered “Not resolved”, instead of “Ongoing”. To put it differently, “Advice” is more likely to ‘place’ a problem into the category of “Not resolved”. “Preparing documents”, “Mediating”, “Deciding” and even “Referring” increase the likelihood that a legal problem is “Ongoing”.
Besides the interventions and dispute resolution processes, some other characteristics of the legal problems impact the resolution status. Problems in Ethiopia are more likely to be “Completely resolved” than “Not resolved”. Employment problems are more likely to be “Not resolved” than to land problems. Land problems are more likely to be “Ongoing” instead of “Completely resolved” compared with family problems. The less impactful problems are more often “Completely resolved”. Impactful problems are less often “Completely resolved”. The most impactful problems are most likely to be “Ongoing”. Women are less likely to resolve their problems than men.
 Voert, M. J. ter, & Hoekstra, M. S. (2020). Geschilbeslechtingsdelta 2019.
 OECD. (2019). Equal Access to Justice for Inclusive Growth. p. 113
 Pleasence, P., & Coumarelos, C. (2014). Reshaping legal assistance services: building on the evidence base. Retrieved from http://www.lawfoundation.net.au/ljf/site/articleIDs/D76E53BB842CB7B1CA257D7B000D5173/$file/Reshaping_legal_assistance_services_web.pdf
 A neutral third party decides the outcome of the dispute based on the relevant rules and evidence.
 For more elaborated version of the building blocks see https://dashboard.hiil.org/building_blocks
 More details about the surveys are available at: https://dashboard.hiil.org/justice-dashboard-methodology/
 The use of dispute resolution mechanism in Model 1 and Model 2 is relative because the actual variable is based on the question – “Which was the most useful source of dispute resolution?”
 RRR indicates how the risk of the outcome falling in the comparison group compares to the risk of the outcome falling in the referent group changes with the variable in question. For instance, a relative risk ratio of 2.49 for “Community justice mechanism” in the level “No, the problem is not resolved and I am no longer taking actions to solve it”, indicates that using a “Community justice mechanism” increases the likelihood that a problem is at this level instead of “Yes, completely resolved” (the reference level), compared with the option “Adjudication”.
Table of Contents