BACKGROUND PAPER

How to figure out “What works”
in People-centered justice?

January 24, 2023

This paper provides inputs for a policy brief about “what works in justice”. The primary purpose of the background paper is to serve as an example for analysing data to understand “what works” in resolving disputes. Together with the policy brief, its long-term objective is to stimulate policymakers and service providers to gather data systematically to identify effective and scalable approaches for delivering access to justice.

The paper starts with theoretical deliberations about access to justice. We look at the linkages between inputs, processes and results of justice processes. Then three data models are tested with people-centred justice data obtained through survey research in diverse jurisdictions. The results of the models are discussed in each section, and the overall implications are elaborated in the policy brief.

Problem background

Success in justice delivery is not random, but we do not know much about what affects the chances of success and the risks of failure. Formal and informal justice systems and mechanisms resolve certain legal problems better than others. For instance, HiiL’s research (refs.) consistently finds that problems with lower impact are more frequently resolved than problems with higher impact. Similar findings are reported by Ter Voert and Hoekstra. ¹ The category of problems also matters – land and crimes are less frequently resolved than other issues. Certain people are less likely to report positive results and outcomes of their legal problems – i.e. the legal problems of urban residents usually are more likely to be resolved compared to the problems of rural residents.

However, the available evidence about “what works” and “what doesn’t work” is minimal. Access to justice is very often designed and delivered on the basis of intuition. Very often, positive interventions are entirely based on normative criteria. The OECD observes, “Given the relative paucity of empirical evidence about which interventions result in effective or meaningful access to justice, decision-makers rely on a range of guiding principles, indicators and other criteria to guide the policy development process. These criteria integrate theories about access to justice and be seen as predictors of, or proxies for, “what works”. Criteria for what works can also be derived from emerging promising practices in promoting accessibility and people-centricity of legal and justice services.” ² The question is how to “capture”, understand and scale such promising practices. In this paper, we will look at data reflecting the experiences and perceptions of users of justice to understand “what works”.

Our fundamental hypothesis is that three sets of factors influence the results of dispute resolution. The first set of factors is related to the parties involved. The second set concerns the problem’s type, gravity and impact. The third set of factors is related to the quality of the dispute resolution process. Below we extend the 3P (party, problem, process) model.

Party-related variables: Socio-demographic characteristics such as gender, age, education, income, and rurality play a role in legal disputes. This is just the basic interaction between people’s characteristics and dispute resolution. At a much deeper level, one can look for the impact of power disbalances, legal awareness and legal capabilities, cultural structures and beliefs. In this research, we limit ourselves to the most essential factors but encourage others to dig deeper. For instance, Pleasence et al. assert that the legal capability of the party “has the potential to confound socio-legal studies and vex evaluation of legal service provision.” ³ In that vein, when failing to account for the legal capability dimension the “...evaluation of legal services may not provide an accurate picture of whether or not certain forms and modes of legal service provision ‘work’.” ⁴
Problem-related variables: Legal problems vary enormously, and this variance is hypothesized to affect their resolution significantly. Some problems are normatively regulated and can only be resolved by a specific process, or specific source of help - i.e. divorce, insolvency or change of name in most countries can only be done through some sort of adjudication. The density and complexity of the legal regulation also affect the knowledge required to solve the issue. The resolution of problems is highly correlated with their complexity, duration, actual and perceived impact. Most land or family problems, for instance, are more impactful than most consumer problems.
Process quality: Third parties are involved in dispute resolution to resolve the problem fairly. The role of the third parties is to achieve a quick, objective, unbiased, legitimate and stable resolution. National and international courts, public authorities, and arbitration tribunals have their dispute resolution mandate by law. Community institutions such as village elders, cultural or religious leaders, or customary courts draw their mandate primarily from tradition and cultural norms.

These third parties employ diverse approaches to resolving disputes. Some of the most often used approaches are advice, adjudication, mediation, reconciliation, and referral to other options for dispute resolution. In practice, relatively rarely do third parties use highly structured dispute resolution mechanisms such as adjudication or formalised mediation.

An essential tenet of this background paper and the related policy brief is that some interventions are more effective in resolving legal problems. Such interventions “work” – they are more accessible, fair, effective, efficient and fair. The meaning of interventions, however, is not widely accepted in justice research and practice. Under interventions, we understand discrete parts of the overall dispute resolution process, such as

Advising
Deciding/settling the matter is the process of adjudicating a dispute ⁵
Mediating/reconciling
Referring the matter
Preparing documents
Representing
Giving emotional/moral support
Providing advice
Doing nothing.

In a way, the interventions are the main building blocks of dispute resolution. There are many more “Lego parts” that can be added to this list. ⁶ Many of these building blocks occur together with others and form new blocks. For instance, advice is often part of a broader service which includes document preparation, filing suits, motions and requests, representation, appeal etc.

To sum up, the theoretical foundation of “what works” is based on three broad sets of factors that are believed to explain a significant portion of the variation of the outcomes. The sets of factors are: 1) parameters of the problems, 2) characteristics of the parties, and 3) dispute resolution interventions. Many other factors affect the outcomes of justice, and these factors are not unimportant. We invite others to continue the research efforts to understand the ultimate question of “what works in justice”.

Data and methods

Cross-sectional survey research data collected with HiiL’s Justice Needs and Satisfaction instrument are used for this analysis.⁷ Three countries ( Ethiopia, Mali and Uganda) participated with two data sets with unrelated samples. Model 1 and Model 2 are based on a dataset with data from:

Bangladesh
Ethiopia
Fiji
Indonesia
Jordan
Kenya
Lebanon
Mali
Morocco
Netherlands
Nigeria
Tunisia
UAE
Uganda
Ukraine
United States
Yemen

Model 3 uses data from Ethiopia, Burkina Faso and Niger. In these 3 surveys, the concept of interventions was added to the research instrument and hence made additional research available.

Limitations of the data and this analysis

The results of this analysis must be considered carefully and should be interpreted in light of the dataset and analysis limitations, which include the following:

Survey research, such as legal need survey research, is not the optimal design to investigate intrinsically causal questions such as “what works”. Moreover, HiiL’s justice needs and satisfaction research has not been designed with “works” questions in mind.
This background paper and the related policy brief, therefore, do not provide general answers to the “what works” in justice questions. Overly generalised questions cannot yield robust, valid and reliable results;
For the purposes of this background paper and the related policy brief, to limit the variation due to the different types of legal problems, we only focus on three categories of problems: land, neighbours and family.
Many significant personal, problem or environment-level factors were not measured in the original research (i.e. legal capabilities, power misbalances, legal institutions) and/or are not accounted for in the models.

Model 1: Resolution of the legal problems

Dependent variable in Model one is the resolution of the problem. Resolution is measured at four levels – “Completely resolved”, “Partially resolved”, “Ongoing” and “Not resolved”. A multinomial logit model is used to explore the relationships between the levels of resolution and the independent variables. All coefficients from the table below are expressed as the likelihood of achieving the particular outcome versus the outcome level “Completely resolved”. The relative risk ratio represents the regression coefficient because of its more intuitive value.

Socio-demographic variables in the model are gender, age, education, and location (urban-rural). The problem category and its impact reflect the properties of the issue. The key independent variable of interest in Models 1 and 2 is the dispute resolution process which was considered the most useful mechanism for resolving the particular problem. The many types of mechanisms are aggregated into several major categories – “Courts and lawyers”; “Police”, “Other organised procedures”, “Personal network”, and “Self-action”. Other organised procedures include formal and informal mechanisms for resolving legal problems which do not fall in the category of “Courts of lawyers”. Most often, these are various community-level dispute-resolution mechanisms. The “Personal network” category combines family members, friends, and neighbours.

Relative to the Completely resolved category

Model 1 is statistically significant, which means that the independent variables are related to the change in problem resolution. In summary, the model tells us that younger people are less likely to resolve their problems than older people. An increase in education increases the chance of resolving legal issues. On the other hand, urban people (who are more educated) are less likely to resolve their problems. The more impactful problems are less likely to be “Completely resolved” or “Partially resolved”. Surprisingly, land problems, which are more serious on average, are more often “Completely” or “Partially resolved”. “Self-action” reduces the likelihood of a problem being “Completely resolved”. The engagement of “Courts and lawyers” increases the likelihood of a problem being “Completely resolved”.

Looking at the socio-demographic, we see that women are less likely than men to have their problems “Completely resolved” instead of being “Not resolved”. People in middle adulthood (40-64) are more likely to have their problems solved compared to young people. Individuals with a high degree of education are more likely to have their problems solved compared to individuals with no education. Low-income people are much less likely to resolve their problems “Completely”. Individuals with medium and high education are significantly more likely to report their legal problems as “Partially resolved” than those without education.

Urban people are more likely to have their problems “Ongoing” than rural people. Young adulthood (25-39) and middle adulthood (40-64) individuals are more likely to report problems as “Ongoing” than young (18-24) individuals. People with a high level of education are more likely to have their problems “Ongoing” than individuals without education. Lower-middle income increases the likelihood of a problem being “Ongoing” instead of “Not resolved” compared to the low-income category.

In the characteristics of the problem, we see that employment and family problems are less likely to be “Completely resolved” or “Ongoing” instead of “Not resolved” than land problems. The higher impact is associated with fewer “Completely” and “Partially resolved” problems. “Self-action” reduces the likelihood of a problem being “Partially resolved” than “Not resolved”.. Problems with higher impact are more likely to be “Partially resolved”, “Ongoing”, or “Not resolved” than problems with lower impact.

In the process part, we see that using “Courts and lawyers” increases the likelihood that a problem is “Completely resolved” compared to the other mechanisms. ⁸

Estimating the marginal effects of the multinomial regression allows us to analyse the probabilities, which are easier to interpret. The marginal effects show some interesting trends:

Completely resolved problems

“Other organised procedures” are associated with a 5% increased likelihood that the problem is “Completely resolved” than “Courts and lawyers”.

Partially resolved

“Other organised procedures” are associated with a 2% increased likelihood that the problem is “Partially resolved” compared to “Courts and lawyers”. When the most useful process is “Personal network” there is a 2% increased likelihood that the problem is “Partially resolved” compared to “Courts and lawyers”.

Ongoing

All options other than “Courts and lawyers” have a lower risk that the problem is still “Ongoing”: “Police” (12% decrease), “Other organised procedures” (11% decrease), “Personal network” (13% decrease), “Self-action” (10% decrease).

Not resolved

Compared to “Courts and lawyers”, “Police” is associated with an 8% increased likelihood that a problem is “Not resolved”. Compared to “Courts and lawyers”, the use of “Other organised procedure” is associated with a 6% increased likelihood that a problem is “Not resolved”. Compared to “Courts and lawyers”, the use of “Personal network” is associated with a 12% increased likelihood that a problem is “Not resolved”. Compared to “Courts and lawyers”, the use of “Self-action” is associated with a 13% increased likelihood that a problem is “Not resolved”.

The key take-away from Model 1 is that “Courts and lawyers” resolve problems but they are also slow. Using “Courts and lawyers” is related to a significant decrease in the risk of “Not resolved”. All other sources increase that risk. But using “Courts and lawyers” has a significant drawback – the option increases the risk that a problem is “Ongoing”.

Model 2: Composite measure of fair process, fair result and costs of resolving a problem

In Model 2, the dependent variable is a composite measure of the quality of dispute resolution. The elements of this variable are measures of procedural justice, distributive justice, restorative justice, enforcement, the ability of the result to resolve the problem, and the costs of the paths to justice. All elements of the quality of the outcome are measured with a 5-point Likert scale. In the next step, all variables are aggregated using a simple mean function. Linear regression is used to explore the effect of the independent variables on the quality of dispute resolution.

The key independent variable of interest is the dispute resolution process which was perceived as the most useful mechanism for resolving the particular problem. Similar to Model 1, the other independent variables reflect characteristics of the party (gender, age, education and income) as well as the type of the problem and its perceived impact on the respondent.

The results demonstrate that there are no significant differences in the outcome quality across the various dispute resolution mechanisms. All categories of dispute resolution mechanisms are compared to “Courts and lawyers”, and the differences are not statistically significant. Men report higher satisfaction with the quality of the outcome than women (3.24 v 3.11), however, in the multivariate model, this difference is not significant.

There are statistically significant differences in the perceived quality of the outcome in some of the person and problem-related variables. Young people (24-35) and middle age (35-55) report better outcomes than the very young and the senior respondents. People with medium and high education report significantly better results than people without education. However, individuals with higher incomes report worse results than people with lower education. The justice journeys for employment and family-related legal problems receive lower outcome scores compared to land problems. The impact of the problem is not related to the outcome in this model

Model 3: Focus on the interventions

Model 3 analyses the interventions that the third parties apply to resolve legal problems. In 3 of the countries (Ethiopia, Niger and Burkina Faso, the datasets contain information about the discrete interventions applied by the third parties). Similar to the previous models, for simplicity, we focus on land, employment and family problems.

The dataset is structured a little differently than Models 1 and 2, where each record corresponds to one respondent. In Model 3, one respondent (or problem) could appear on more than one row because more than one intervention can be applied to resolve a problem.

Dispute resolution mechanisms and interventions

In Model 3, we explore two sets of variables – the dispute resolution mechanisms and the interventions. Dispute resolution mechanisms are the types of justice journeys as defined by their main mode of resolution. In Model 3, we distinguish between the following dispute resolution mechanisms:

Formal adjudication
Community justice
Social environment
Police & other public authorities
Legal professional
Negotiation
Other

The interventions are the activities that the neutrals perform as part of their strategy to resolve a legal problem. A third party can apply one or more interventions. Therefore the variable is multiple choice – more than one of the following interventions (or lack of interventions) are possible:

Provided advice
Prepared documents
Mediated between the parties
Decided
Referred
Represented
Emotional support
Other
Did nothing

The results of a multinomial logit regression model are provided below. Similar to Model 1 the coefficients are expressed in relative risk ratios (RRR). ⁹

From the regression results, we see that the selected explanatory variables explain mostly the difference between the “Ongoing” and “Unresolved” resolution levels compared to “Completely resolved”. The use of “Formal adjudication mechanisms” significantly increases the likelihood that a legal problem is “Completely resolved” instead of “Not resolved” compared to other dispute resolution mechanisms such as “Community justice mechanisms”, “Social environment”, “Police and other public authorities”, and “Negotiation”. The difference between “Formal adjudication” and the involvement of “Legal professionals” is not statistically significant. “Negotiation” or “Other DRM” increase the risk that a problem remains “Unresolved”.

“Formal adjudication”, however, is significantly more likely to lead to “Ongoing” problems. The use of “Community justice mechanisms”, “Social environment”, and “Legal professionals” decreases the risk that a problem is “Ongoing” instead of “Unresolved”.

“Advice” is the most frequently used intervention for resolving legal problems related to land, employment and family issues. However, “Advice” seems to be the least effective of the interventions, excluding the options “Other” and “Doing nothing”. “Mediating/reconciling”, “Deciding the matter”, and even “Referring” increase considerably the chance that a problem will be “Completely resolved” as compared to “Not resolved”. “Doing nothing” or “Other” increase massively the risk that a problem remains unresolved.

“Deciding” is the intervention which most considerably outperforms “Advice” as a means to “Completely resolve” a problem (coefficient 0.30) instead of the problem being “Unresolved”. “Deciding” decreases substantively the risk that the problem is “Pending” compared to “Advising”. “Representing” and “Doing nothing” both increase the likelihood that a problem will be “Pending” instead of “Completely resolved”.

“Deciding” also decreases the risk that a problem is “Not resolved” compared to “Advising”. “Doing nothing” increases the risk that a problem is “Not resolved”. Compared to “Preparing documents”, “Mediating”, “Deciding”, and even “Referring”, “Advice” significantly increases the risk that a problem will be considered “Not resolved”, instead of “Ongoing”. To put it differently, “Advice” is more likely to ‘place’ a problem into the category of “Not resolved”. “Preparing documents”, “Mediating”, “Deciding” and even “Referring” increase the likelihood that a legal problem is “Ongoing”.

Besides the interventions and dispute resolution processes, some other characteristics of the legal problems impact the resolution status. Problems in Ethiopia are more likely to be “Completely resolved” than “Not resolved”. Employment problems are more likely to be “Not resolved” than to land problems. Land problems are more likely to be “Ongoing” instead of “Completely resolved” compared with family problems. The less impactful problems are more often “Completely resolved”. Impactful problems are less often “Completely resolved”. The most impactful problems are most likely to be “Ongoing”. Women are less likely to resolve their problems than men.

Reference to the policy brief

The broader implications of the results continue in the policy brief.

Authors

This Policy Brief was written, Dr. Martin Gramatikov, Programme Director Kenya, Ukraine, Colombia, South Sudan.

View References

^[1] Voert, M. J. ter, & Hoekstra, M. S. (2020). Geschilbeslechtingsdelta 2019.

^[2] OECD. (2019). Equal Access to Justice for Inclusive Growth. p. 113

^[3] Pleasence, P., & Coumarelos, C. (2014). Reshaping legal assistance services: building on the evidence base. Retrieved from http://www.lawfoundation.net.au/ljf/site/articleIDs/D76E53BB842CB7B1CA257D7B000D5173/$file/Reshaping_legal_assistance_services_web.pdf

^[4] Ibid.

^[5] A neutral third party decides the outcome of the dispute based on the relevant rules and evidence.

^[6] For more elaborated version of the building blocks see https://dashboard.hiil.org/building_blocks

^[7] More details about the surveys are available at: https://dashboard.hiil.org/justice-dashboard-methodology/

^[8] The use of dispute resolution mechanism in Model 1 and Model 2 is relative because the actual variable is based on the question – “Which was the most useful source of dispute resolution?”

^[9] RRR indicates how the risk of the outcome falling in the comparison group compares to the risk of the outcome falling in the referent group changes with the variable in question. For instance, a relative risk ratio of 2.49 for “Community justice mechanism” in the level “No, the problem is not resolved and I am no longer taking actions to solve it”, indicates that using a “Community justice mechanism” increases the likelihood that a problem is at this level instead of “Yes, completely resolved” (the reference level), compared with the option “Adjudication”.