4. Strategy 1:
data on problems,
impact and outcomes

Trend Report 2021 – Delivering Justice / 4. Strategy 1: data on problems, impact and outcomes

Effective people-centred justice critically depends on the availability of data on the effects on people of injustices and justice interventions (Chapman et al. 2021). What is the impact of an unsolved land problem, for example? Which interventions are implemented to allocate rights to use the land and do these result in a solution with outcomes that are acceptable to the parties? Data collected at the level of service delivery provide information on the quality of a particular service. Data collected and published at a national level make it possible to monitor the extent to which justice problems are prevented and resolved within a broader population.

Measuring justice delivery: the benefits of further standardisation

A standardised approach to monitoring the quality of processes and outcomes is crucial for increasing the quality of justice interventions that together make up a resolution process. A standardised set of outcomes allows a series of interventions that add up to a treatment to be compared and evaluated systematically. Currently, evaluation studies for justice interventions each make use of their own methods. Ideally, practitioners and researchers would use similar methods to monitor the quality of the process and outcomes of, for example, personal injury cases.

When HiiL (n.d.-n) developed its measuring justice methodology, standard indicators of procedural justice existed: voice, neutrality, respect, and trust. Further standardisation is needed to measure the quality of justice outcomes across other dimensions, such as distributive justice, restorative justice, effectiveness and transparency.

Measuring the time, money and emotional costs needed to obtain resolution has proven to be difficult. People go through complicated processes to achieve justice and generally find it difficult to disentangle the costs of resolution from the impact of the problem. A better methodology to measure the burden of seeking justice is needed. A clear and consistent finding is that the emotional costs of this process should not be ignored. The existence of secondary victimisation as a concept – being victimised by the procedure after being victimised by a crime or accident – is a case in point. International standards for monitoring problems, impact, outcomes and justice journeys are being developed.

Regular national surveys: needed and difficult to fund

Countrywide data is needed. Ideally, data enables the task force and the providers of justice services to monitor progress towards people-centred justice. Widening justice gaps, or increases in the burden of injustice, can signal a need to redirect resources or to develop new treatments.

Data on the economy are published on a quarterly or monthly basis. Crime statistics are typically published on a yearly basis. Data on justice problems, impact, vulnerable groups and outcomes achieved can be collected through standardised annual – or more frequent – surveys. For most sustainable development goals, time series data exist that show trends in performance for different countries. Our World in Data has become a core hub for this data (University of Oxford and Global Change Data Lab n.d.). Few time series related to justice are available (ibid). Data comparisons occur mostly between countries, while survey repetitions are few and far between (HiiL n.d.-o).

When publishing data about justice problems in the population, the task force should reflect on actionability. What information should be shared with which audience? Who can take action on which elements of the data? How do they learn about the data? By conducting Justice Needs and Satisfaction Surveys in a wide range of countries over the years, HiiL has learned how data can be made more actionable (HiiL n.d.-p).

Survey data are often presented as percentages of populations (OECD and Open Society Foundation 2019). However, for a team looking to scale up a gamechanger and do capacity planning, for example, the number of potential users is more meaningful than a percentage. This can easily be estimated from survey data. Disaggregation is key. Data users often request that data be grouped by type of justice problem. Breakdowns of specific issues (i.e. divorce or child support) and specific complications (i.e. violence, loss of job, personal injury, relational problems) are also useful. In some cases, however, sample size may become a problem, because a survey will not always capture many people with one particular type of justice problem.

Data about impact and outcomes achieved must always be interpreted. User stories can be used to represent and illustrate the average justice journey. Do people need more information to resolve their justice problem? Is contacting the other party for meaningful negotiation their main bottleneck? Do they need more interpersonal respect? Was the amount of money they received through the resolution process unfair? Justice journey maps can answer these questions in a memorable and engaging way.

Survey results are much easier to interpret if they include benchmarks. International rankings such as those provided by the World Justice Project and comparisons to neighbouring countries can be helpful (World Justice Project n.d.). However, few countries consistently perform at a level high enough to be visible in national surveys. Most high-performing services operate at a small scale. Few countries have scaled a particular service to the entire target group. When selecting benchmarks, this needs to be taken into account.

Securing resources for annual surveys is a challenge. Victimisation surveys, once done in a standardised way across Europe, have been discontinued (University of Lausanne n.d). Legal needs surveys are administered irregularly. National statistics offices are now asked to include questions about justice in their large population surveys (Statistics South Africa 2019).

Surveys can be carried out in person, in people’s homes, through panels organised by data collection companies, or through social media. Each method has pros and cons in relation to representativeness. Collecting social media data creates an opportunity to monitor trends in justice needs in real time. HiiL (n.d.-q; n.d.-r) experimented with this during the COVID-19 crisis by comparing social media trends with the observations of experts. Conversations on social media can be searched using keywords that are associated with particular justice problems. Trends in problems can thus be monitored during times of crisis, whereas in normal times the number of family problems or land problems is likely to be rather stable.

Triangulation with other data

Survey data need to be compared with other justice data. In most of the countries where HiiL has carried out a Justice and Needs Survey (JNS), the World Justice Project has collected basic access to justice data in the three largest cities (World Justice Project 2019). Courts sometimes collect user experience data. These forms of justice data can be used to enrich existing survey data.

Access to justice reform programmes are executed by private, public or civil society organisations. Sharing performance and output data with the task force (and the public) in open formats should be part of implementation activities planned by the task force. Indicators for which there is no valid, reliable and regular data have little value.

A flaw of current survey methods is that they miss people who are in prison, are homeless, or who are hiding from the authorities. Some people may have problems that they are ashamed to talk about and deny. Depending on how questions are asked, surveys may miss the gravest injustices: people dying or disappearing. Survey companies contact people at home, through phone calls, through email or through social media. Reaching the populations excluded by these methods requires a more creative approach.

Data on the number of people in these hard-to-reach categories can be used as a proxy. Estimates of these numbers are available in many countries. Each individual who is incarcerated, homeless or unregistered is very likely to experience their situation as a justice problem. More reliable data on the problems these people have can be obtained by surveying segments of these populations.

View References

Chapman, Peter et al., (2021). Grasping the Justice Gap: Opportunities and Challenges for People-Centered Justice Data. Washington, D.C.: World Justice Project; New York: Pathfinders for Peaceful, Just and Inclusive Societies; Paris: OECD.

HiiL, (n.d.-n). Justice Dashboard – Methodological Note. URL: https://dashboard.hiil.org/justice-dashboard-methodology/. Accessed on August 1, 2022.

HiiL, (n.d.-o). Country comparison page. URL: https://dashboard.hiil.org/country-comparison-page/. Accessed on August 1, 2022.

HiiL, (n.d.-p). Projects. URL: https://www.hiil.org/projects/?_sft_service=service-1. Accessed on August 1, 2022.

HiiL, (n.d.-q). Delivering justice in the Covid-19 crisis (web report).

HiiL, (n.d.-r). Justice in the Covid-19 crisis: What people are saying on social media (web report).

OECD and Open Society Foundations (2019). Legal Needs Surveys and Access to Justice. Paris: OECD Publishing.

Statistics South Africa, (2019). Governance, public safety and justice delivery. Republic of South Africa.

University of Oxford and Global Change Data Lab, (n.d.). Our World in Data.

University of Oxford and Global Change Data Lab, (n.d.). Promote just, peaceful and inclusive societies. Our World in Data. URL: https://sdg-tracker.org/peace-justice. Accessed on August 1, 2022.

World Justice Project, (2019). Global insights on access to justice: Findings from the World Justice Project general population poll in 101 countries.

World Justice Project, (n.d.). Research and data. URL: https://worldjusticeproject.org/our-work/wjp-rule-law-index. Accessed on August 1, 2022.