8. Strategy 4: data on problems, impact and outcomes

8. Strategy 4:
data on problems,
impact and outcomes

Trend Report 2021 – Delivering Justice / 8. Strategy 4: data on problems, impact and outcomes

Effective people-centred justice critically depends on availability of data. Data collected at the level of service delivery inform the quality of a particular service. Data collected and published at a national level make it possible to monitor progress on the extent to which justice problems are prevented and resolved within the broader population.

Measuring justice delivery: the benefits of further standardisation

A standardised approach to monitoring the quality of processes and outcomes is crucial for increasing the quality of justice interventions that combine into a process that resolves a justice problem. A standard allows treatments to be compared and evaluated systematically. Currently, evaluation studies for justice interventions each make use of their own methods. Ideally, practitioners and researchers use similar monitoring methods for the process and outcomes of, for example, personal injury cases.

When HiiL developed its measuring justice methodology, standard indicators of procedural justice existed: voice, respect and information people receive when they follow a certain procedure. Further standardisation is needed to measure the quality of justice outcomes, such as distributive justice, restorative justice, effectiveness and transparency.

Measuring the time, money and emotional costs of getting access to justice has also proven to be a challenge. People go through complicated processes to achieve justice and generally find it difficult to disentangle the costs of resolution from the impact of the problem. A better methodology for measuring the burden of seeking justice is needed. One clear and consistent finding is that the emotional costs of this process should not be ignored. The existence of secondary victimisation as a concept is a case in point.

A task force working on a specific category of justice problems in a country can hardly be expected to contribute to developing these standards. The task force can, however, align with standards that are being considered by international experts. International standards for monitoring problems, impact, outcomes and justice journeys are developing. The sooner this work leads to first versions of worldwide standards, the better.

View References

Measures for procedural justice are described in J.A. Colquitt, J.B. Rodell, Measuring justice and fairness, in The Oxford handbook of justice in the workplace, 2015. See also Clara Sabbagh and Manfred Schmitt, Handbook of Social Justice Theory and Research, 2016.
Literature review on the psychological costs of litigation: Michaela Keet and Heather Hevin in Farrow and Jacobs, The Justice Crisis, 2020, Chapter 14. A more extensive version can be found in Michaela Keet, Heather Heavin and John Lande, Litigation Interest and Risk Assessment: Help Your Clients Make Good Litigation Decisions, 2020.

Regular national surveys: needed and difficult to fund

Countrywide data is needed as well. Ideally, data enables the task force and the providers of justice services to monitor progress towards people-centred justice. Widening justice gaps, or increases in the burden of injustice, can signal a need to redirect resources or develop new treatments.

Data on the economy are published on a quarterly or monthly basis. Crime statistics are typically published on a yearly basis. Data on justice problems, impact, vulnerable groups and outcomes achieved can be collected through standardised surveys that are repeated every year or more frequently. For most sustainable development goals, time series exist that show trends in the performance of different countries. Our World in Data has become a core hub for these data. Few time series related to justice are available. Comparing data occurs mostly between countries while survey repetitions are few and far between.

When publishing data about justice problems in the population, the task force should reflect on actionability. What information should be shared with which audience? Who can take action on which elements of the data? How do they learn about the data? By conducting Justice Needs and Satisfaction Surveys in a wide range of countries over the years, HiiL has learned how data can be made more actionable.

Survey data are often presented as percentages of populations. For a team looking to scale up a game-changer and do capacity planning, for example, the number of potential users is more meaningful than a percentage. This can easily be estimated from survey data. Specialisation is key. Data users often request that data be grouped by type of justice problem. Breakdowns of specific issues (i.e. divorce or child support) and specific complications (i.e. violence, loss of job, personal injury, relational problems) are also useful. Sample size may become a problem, though, because a survey will not capture that many people with one particular type of justice problem.

Data about impact and outcomes achieved must always be interpreted. Stories, representing the average justice journey, are suggested as an illustration. Justice journey maps – which visualise people’s experiences seeking justice – are another frequent request. Do people need more information to resolve their justice problem? Or is contacting the other party for meaningful negotiation their main bottleneck? Do they need more interpersonal respect? Or was the amount of money they received through the resolution process unfair?

Survey results are much easier to interpret if they include benchmarks. International rankings such as the ones provided by the World Justice Project and comparisons to neighbouring countries can be helpful. However, few countries consistently perform at a level high enough to be visible in national surveys. Most high-performing services are operating at a small scale. Few countries have scaled a particular service to the entire target group. When selecting benchmarks, this needs to be taken into account.

Securing resources for annual surveys has proven to be a challenge. Victimisation surveys, which were once done in a standardised way across Europe, have been discontinued. Legal needs surveys are administered irregularly. National statistics offices are now asked to include questions about justice in their large population surveys. This happened in South Africa.

Surveys can be carried out in person, in people’s homes, through panels organised by data collection companies or through social media. Each method has pros and cons in relation to representativeness. Collecting social media data creates an opportunity to monitor trends in justice needs in real time. HiiL experimented with this during the COVID-19 crisis by comparing social media trends to the observations of experts.

Triangulation with other data

Survey data need to be compared with other data. In most of the countries where HiiL has carried out a Justice and Needs Survey (JNS), the World Justice Project has collected more basic access to justice data in the three largest cities. Courts may have user experience data. These forms of data can both be used to enrich existing survey data.

Access to justice reform programmes are executed by private, public or civil society organisations. Sharing performance and output data with the task force (and the public) in open formats should be part of implementation activities planned by the task force. Indicators for which there is no valid, reliable and constant data have little value.

A flaw of current survey methods is that they miss people who are in prison, homeless, hiding from the authorities. Some people also may have problems that are shameful. They may avoid talking about them or are in denial. Depending on how questions are asked, surveys also may miss the gravest injustices: people dying or disappearing. Survey companies contact people at home, through phone calls, through email or through social media. Reaching the population excluded by these methods requires other methods.

Data on the number of people in these hard-to-reach categories can be used as a proxy. Estimates of these numbers are available in many countries. Each person who is in custody, is homeless or is unregistered is very likely to experience this situation as a justice problem. More reliable data about the problems these people have can be obtained by surveying segments of these populations.