Close this search box.
Close this search box.

Justice Needs and Satisfaction Survey - Methodological note

This document describes the methods for collecting, analysing and displaying data in the Justice Dashboard. Its purpose is to place the data published on the Justice Dashboard in context so it can be properly understood, interpreted and replicated.

About the methodology of Justice Needs and Satisfaction surveys

Our methodology for Justice Needs and Satisfaction (JNS) surveys is based on the legal needs and justiciable events research tradition. See for an overview of the methodologies and the challenges the following papers:

  • Pleasence, Balmer and Sandefur, Paths to justice, A past, present and future roadmap, 2013 (download here)
  • Law and Justice Foundation of New South Wales, Methodology of legal needs surveys, 2014 (download here)

Our main new contributions to this line of data collection about user needs are the the following:

  • We developed a standard methodology that now has been applied in 15 countries. See the countries listed below, plus Fiji and Morocco that are being surveyed during the final months of 2018.
  • We added much more detailed assessment of access to justice (procedural justice, outcome justice and costs of access to justice), following a methodology that was developed at Tilburg University, see Gramatikov, A Handbook for Measuring the Costs and Quality of Access to Justice, 2010.

How we measure justice

Below we outline the main concepts and approaches underlying the JNS methodology.

Justice Needs and Satisfaction (JNS) Survey is the tool that we use to measure justice from the bottom-up. Randomly selected adult (18+) respondents are interviewed about their legal problems, resolution strategies and perceptions about the costs of justice and quality of processes and outcomes.

(Gramatikov, Martin et al. 2010. A Handbook for Measuring the Costs and Quality of Access to Justice. Apeldoorn, Antwerpen, Portland: Maklu)</span

Justice is accessible when people can reach fair solutions whenever they encounter disputes, disagreements or rights violations.

(Barendrecht, Maurits et al. 2012. Towards Basic Justice Care for Everyone. Challenges and Promising Approaches. The Hague: Hague Institute for the Innovation of Law. link)

Bottom-up justice is an approach to measuring justice from the perspective of the people. In that way bottom-up justice is focused on the needs for justice as the people encounter these needs in their lives. The alternative – top-down approach looks at justice from the perspective of the formal institutions (i.e. how many court cases, complaints to police, administrative requests, how are cases processed, resolved etc.).

(Commission on Legal Empowerment of the Poor. 2008. II Making the law work for everyone. New York: United Nations Development Programme. link
Piest, Johanna et al. 2016. Justice Needs in Uganda: Legal Problems in Daily Life. The Hague. link)

A legal problem refers to a problem that takes place in daily life – a dispute, disagreement or grievance for which there is a resolution in the (formal or informal) law. In the legal needs research the term justiciable events is also used. The resolution of the problem could be through an intervention of a third party – i.e. adjudication, administrative process, arbitration or mediation or through negotiation or reconciliation between the parties. It is not necessary that the respondent knows or recognizes its legal aspects. It is also possible that nothing has been done to resolve the problem.

(Gramatikov, Martin et al. 2010. A Handbook for Measuring the Costs and Quality of Access to Justice. Apeldoorn, Antwerpen, Portland: Maklu

Genn, Hazel G, and Sarah Beinart. 1999. Paths to Justice : What People Do and Think about Going to Law. Oxford, England ; Portland, Or.: Hart Pub.

Currie, Ab. 2010. The Legal Problems of Everyday Life The Nature, Extent and Consequences of Justiciable Problems Experienced by Canadians. Department of Justice, Canada. Link

OECD Guideline Document. Upcoming link)

The JNS questionnaire uses plain language to ask people about the legal problems that they encountered in daily life. People in the streets do not use legal language. They talk about divorce, accidents, losing a job or being robbed. In the questionnaire we ask:

Have you experienced serious disputes, disagreements and grievances that have resolution through a legal process as shown in Show-card 1 in the past 4 years? This means since [date adjusted to specific study]. Think of either personal or business problems in which you were involved.  It does not matter whether you did something or what you did about the situation or who was the other side – we want to hear about your experiences. Look carefully at the list (Show card 1) and report all problems which occurred to you. It is possible to list multiple problems from the same type and category.

(Gramatikov, Martin et al. 2010. A Handbook for Measuring the Costs and Quality of Access to Justice. Apeldoorn, Antwerpen, Portland: Maklu

See the methodological chapters in HiiL’s JNS reports. link)

In order to avoid marginal legal problems the instrument asks about serious problems. This means that trivial issues are excluded from the scope of the JNS. Enumerators are provided with the following instructions:

We are interested in problems such as disputes, disagreements and grievances. These problems should potentially have resolution through a legal process. However, it does not matter whether the respondent sees the legal aspect of the problem or not. It also does not matter whether the individual respondent formulated legal or non-legal strategies for responding to the problem. 

What is important is whether such problem or problems took place in the respondent’s life.

  • We are interested to record as many problems as possible – help the respondent to think back and report problems.
  • Display Show card 1 and go through it with the respondent.
  • Explain that we are interested to hear about serious problems but also less severe problems. Encourage the respondent to report problems.
  • Explain that it does not matter what the respondent did (or did not do) about the problem.
  • A person might experience more than one type of problem or multiple problems from the same category.

(Pleasence, Pascoe et al. 2004. Causes of Action: Civil Law and Social Justice London: Stationery Office.

Gramatikov, Martin et al. 2010. A Handbook for Measuring the Costs and Quality of Access to Justice. Apeldoorn, Antwerpen, Portland: Maklu)

Users of justice are individuals who experience a legal problem and therefore need justice. From their perspective justice is the fair resolution of the specific problem.

The method only accounts for problems experienced directly by the respondent; i.e. the respondent must have been personally involved and affected by the specific problem.

Gramatikov, Martin et al. 2010. A Handbook for Measuring the Costs and Quality of Access to Justice. Apeldoorn, Antwerpen, Portland: Maklu

Genn, Hazel G, and Sarah Beinart. 1999. Paths to Justice : What People Do and Think about Going to Law. Oxford, England ; Portland, Or.: Hart Pub.)

A neutral third party is a person/s or institution/s that has vested powers to solve the problem in an unbiased and fair manner. Neutral third parties are expected to be objective and neutral in their positions and approaches to resolving legal problems.

(Barendrecht, Maurits et al. 2012. Towards Basic Justice Care for Everyone. Challenges and Promising Approaches. The Hague: Hague Institute for the Innovation of Law. link)

The JNS is neutral towards formal and informal justice. A justice journey can include both formal and informal dispute resolution mechanisms. The instrument asks the users of justice about their perceptions about the quality of the dispute resolution process, the quality of the outcomes and the costs of accessing justice regardless of whether the dispute resolution mechanism is formal or informal.

In each country, the JNS is adapted to the national and local specifics of dispute resolution. Hence, there are many distinct neutral third parties, particularly in the informal justice. In order to facilitate comparisons we bring the various categories of neutrals into fewer aggregated categories:

  • Courts and lawyers — formal courts of law and legal professionals
  • Police
  • Personal network – family members, friends, neighbours, colleagues
  • Self-action – contacting the other party without a neutral, other actions aiming to resolve the problem – i.e. removing a barrier, paying money to sort out the issue etc.
  • Other organised procedure – informal justice mechanisms, public authorities (except for courts and police)

Our aim is to map out bottom-up justice as it happens. Therefore we ask about all types of legal problems – regardless of they belong to civil, family, criminal or administrative law.

Individual legal problems are grouped into categories that combine problems based on their similarity. The problems are organized according to people’s relationships and sometimes might differ from scholastic legal classification. For instance, a consumer problem might be also seen as a contractual issue or domestic violence can be also categorized as a crime. What is important, however, for the people is that they regularly engage into interactions about purchasing goods and services and sometimes these interactions end up in a conflict, or that victims of specific forms of domestic violence can identify it in the list, and it does not get buried under other crimes. From people’s perspective what is important is that they need protection as customers or domestic violence survivors. The list of problem categories can be found here.

Each category consists of specific instances of legal problems. For example, Family problems consist of problems like divorce, separation, custody, alimony etc. We always discuss these categories with experts from the particular country where the JNS study is being conducted. These local experts tell us if a problem has to be removed or added. They also help us to formulate and translate properly and validate the legal problems.

It is possible that people encounter more than one legal problem during the 4-years time frame. Therefore we ask them to report all legal problems they can recall. Thus we map out the overall environment of legal problems as they occur in people’s lives.

The JNS follows in great depth one legal problem per respondent. If the particular respondent has experienced more than one problem we ask for the one that was the most serious problem. The enumerators are trained to explain to the respondent that the most serious problem is that one that has the largest positive or negative impact in the life of the respondent.

We ask about the negative impact that a problem has on people’s lives. In particular, we focus on five dimensions that cover a wide array of the key elements of livelihood: mental health; physical health; important personal relations; personal income; ability to work. We use the values people assign to the level of impact in each dimension to create an impact score that goes from 0 (no impact at all in all dimensions) to 1 (great extent of impact in all dimensions. It is formally calculated as the average score in the five dimensions, divided by five and normalized to obtain the distribution above mentioned.

We ask about legal problems that emerged in the 4 years prior to the interview.

When there is a legal problem a well functioning legal system provides one or more dispute resolution mechanisms. We call the chains of these dispute resolution mechanisms justice journeys. The concept of a justice journey recognizes that there are many different steps to resolve a problem. Most often these steps are not linear. The entirety of steps that people take to resolve a problem is a justice journey.

(Barendrecht, Maurits et al. 2012. Towards Basic Justice Care for Everyone. Challenges and Promising Approaches. The Hague: Hague Institute for the Internationalisation of Law. link
Gramatikov, Martin et al. 2010. A Handbook for Measuring the Costs and Quality of Access to Justice. Apeldoorn, Antwerpen, Portland: Maklu)

The beginning of the justice journey is the moment the user first takes action. This can be the moment when the user seeks information about her possibilities, seeks legal information or advice (from relative, friend, internet, legal adviser, police, mediator, etc.), or files a petition.
(Gramatikov, Martin et al. 2010. A Handbook for Measuring the Costs and Quality of Access to Justice. Apeldoorn, Antwerpen, Portland: Maklu)

Before, during or after travelling a justice journey the users of justice might seek for information or advice about their legal problem. Such information or advice might come from various sources. We make a difference between legal information and legal advice: Legal information is obtained through public sources such as the internet, catalogues, books, radio, etc. Legal advice is obtained from people who offer personalized information, and they can be professionals (lawyers, paralegals, social workers, etc) or non-professionals, such as family members or friends.

A dispute resolution process is defined as a commonly applied process that people address in order to cope with their legal problems. A court procedure is an obvious example of a dispute resolution process. However, the definition includes both formal and informal procedures. This means that a mediation procedure, or a procedure before an informal Commission or other neutral, also qualifies as a path to justice.

(Barendrecht, Maurits, Peter Kamminga, and Jin Ho Verdonschot. 2008. “Priorities for the Justice System: Responding to the Most Urgent Legal Problems of Individuals.” link)

We ask the respondents to reflect whether their legal problem has been resolved at the moment of the interview. People can choose from four options:
The problem has been completely resolved
The problem has been partially resolved – a part of the problem has been resolved but part has not been solved
The problem is still ongoing – there is still a pending (formal or informal) process and there is an expectation that the process can be resolved
The problem has been abandoned and no further actions are envisaged.

Some of these mechanisms are formal, meaning that they are regulated by law. Others are informal, based on custom, traditions, beliefs or power relationships.Justice journeys can also differ by the extent of formality. Some processes, such as adjudication or arbitration, are usually tightly structured. Others, such as self-help, involvement of a family member or reconciliation, have much less of an uniform structure.

The methodology recognizes the following broad categories of dispute resolution processes:

  • Self-action
  • Personal network
  • Professional neutral party

In the Dashboard the categories Professional neutral party is further split into two separate categories (Courts and lawyers and Police):

  • Courts and lawyers
  • Police
  • Personal network
  • Self-action
  • Other organised process

(Gramatikov, Martin, Maurits Barendrecht, and Jin Ho Verdonschot. 2008. “Measuring the Costs and Quality of Paths to Justice: Contours of a Methodology.”

Reaching Further: New Approaches to the Delivery of Legal Services: 26.)

There might be a difference between the dispute resolution mechanisms people use and the dispute resolution mechanisms they would like to use. To document these differences we ask the respondents who have encountered a serious problem “In an ideal situation how would you have resolved the problem?”. In principle, we avoid hypothetical questions but in this case the question is linked to the context of a specific problem.

The respondent can select multiple options. To further dive into the expectations and desires we ask “Which are the three most important characteristics of a process for resolving the [specific] legal problem?”. The available answer categories are:

  • Procedure is affordable
  • Solution is fast
  • Solution is fair
  • I can express my emotions
  • Having direct contact with the other party
  • The other party and/or third party respects me
  • The other party understands my position
  • Advice is available when I need it
  • Other

(Azevedo, C. D., Herriges, J. A., & Kling, C. L. (2000). Ask a Hypothetical Question, Get a Valuable Answer? Retrieved from

When multiple dispute resolution processes are used the JNS asks the respondent to report the costs, quality of procedure and quality of outcome of the process which helped the most to resolve the problem. These are aggregated further into three categories of justice journeys

  • Bilateral – the parties communicate and try to resolve the problem without the involvement of a neutral third party
  • Multilateral – one or both parties involve a neutral third party to help the parties reach a fair solution
  • Mixed – both of the above mentioned approaches are used.

The quality of the procedure is conceptualized as the amount of procedural (7 items), interpersonal (3 items), and informational (4 items) justice that the users of justice receive. The questions ask about the quality of a specific dispute resolution mechanism (recognized as the one that helped the most to resolve the problem).

(Klaming, Laura, and Ivo Giesen. 2008. Access to Justice: The Quality of the Procedure. link)

In the dashboard the value of the quality of the process is derived following the following steps: 1) the mean value of each sub-dimension is calculated; 2) the mean of the 3 sub-dimensions is calculated.

The result from the last part of the justice journey is the outcome. It can be an award for
damages, an agreement about future conduct, or an apology, or a combination of these
We believe the outcome has four dimensions:

  • Fair distribution (distributive justice)
  • Damage restoration (restorative justice)
  • Explanation of the outcome (motivation of outcomes)
  • Problem resolution (enforcement of the result)

(Verdonschot, Jin Ho, Maurits Barendrecht, Laura Klaming, and Peter Kamminga. 2008. Measuring Access to Justice: The Quality of Outcomes. link)

The private costs of justice borne by the user in her pursuit to solve the legal problem. The instrument measures the following costs spent on the path to justice (not only the specific dispute resolution process):

  • Out of pocket (monetary) expenses
  • Personal time
  • Stress incurred on the path to justice
  • Negative emotions incurred on the path to justice. Three types of commonly occurring negative emotions are measured: frustration, anger and humiliation

(Currie, Ab. 2008. The Costs of Experiencing Justiciable Problem.
Gramatikov, Martin. 2009. “A Framework for Measuring the Costs of Access to Justice.” link)

Justice outcomes are positive changes in the well-being of people that result from measures or specific interventions in the process of resolving the problem. For instance, justice outcomes include receiving an apology (family problems), fair land ownership (land dispute), financial compensation (employment dispute), safety of children (children related problems) or no resolution of problem.

The moment when a neutral makes a decision, the parties come to an agreement, or one of the parties leaves the procedure, thus ending the procedure

(Gramatikov, Martin et al. 2010. A Handbook for Measuring the Costs and Quality of Access to Justice. Apeldoorn, Antwerpen, Portland: Maklu)

We ask about possible legal problems that people expect to face in the near future. People can select as many problem categories as they expect to face. Depending on the sample, it may represent awareness of the people who did not report any problem in the past four years, about how common problems are in their society. Or it may provide information about what vulnerable groups anticipate to face should they move from their current place of residence, in particular, for refugees and IDPs.

How do we collect data

The JNS data is collected through surveys of randomly selected individuals. We select a local data collection company with experienced enumerators and supervisors to conduct fieldwork. The size of the samples of the JNS range from 2400 to more than 6000.

When implementing JNS surveys we use a full multi-stage randomized sampling procedure, which gives each adult an equal chance to be part of the sample. Clustering approach is used in order to ensure randomization at regional level. When security of respondents and enumerators is a concern we consult the selection of regions of local experts. To ensure representativeness to the population parameters we include quotas, such as gender, age and rural/urban status. These quotas ensure that the sample reflects the distribution in the population. At the level of primary sampling units (PSU), households are selected on the basis of a random step. The paths of the interviewers as well as the procedure for moving across the PSU are designed to ensure randomization. In the households, the interviewers consider the quotas and select randomly one adult individual to answer the questions. Only one individual is interviewed per household. The questionnaire asks about the legal problems of the selected respondent.

(Northrop, A., 1999. Sampling and Data Collection. In G. Miller & M. L. Whicker, eds. Handbook of research methods in public administration. New York: Marcel Decker, pp. 99–124.)

For our surveys we interview randomly selected adults (18 years or older). Until January 2019, we have collected data in the following countries:

(Northrop, A., 1999. Sampling and Data Collection. In G. Miller & M. L. Whicker, eds. Handbook of research methods in public administration. New York: Marcel Decker, pp. 99–124.)

CountryNumber of respondentsYear
Indonesia (Java, West Kalimantan and South Sulawesi)2,4002014
The Netherlands5,0212013
Yemen (Sanaa, Taiz and Adden governorates)3,0002014
We select a local data collection company with experienced enumerators and supervisors to conduct fieldwork. In some countries where data collection companies are not present or are prohibitively expensive we collaborate with NGOs with national presence. Usually these NGOs are engaged in the provision of legal aid.

The interviews are conducted face-to-face. The data is collected directly on tablets unless the circumstances require the use of pen-and-pencil questionnaires. In The Netherlands, the data was collected online using a panel of respondents provided by the data collection company.

Interviewers are carefully selected based on a number of criteria (age, literacy, experience, language fluency etc). Interviewers and supervisors undergo a comprehensive training on the subject matter, sampling procedure, interviewing and technology.

Despite the efforts to have a strong research design that includes quantitative survey data, qualitative interview data with users of justice, and interviews and rounds of feedback with local experts, inevitably, there are limitations to the data, just as in every study. For detail about people’s experiences with specific justice journeys, different and larger samples are needed, for example, with problem-specific surveys.

People tend to underreport specific legal problems (see also below). This situation is often confirmed by local experts, who indicate that this might be the case for certain problems like land disputes or family problems such as domestic violence and other gender-based violence problems. Family disputes, for example, are considered to be a sensitive topic, making people less likely to seek justice outside of the family.

  • Some people might not report problems due to shame and fear. For example, when people have spent time in prison or have had problems with figures of authority.
  • Cultural norms may cause people to under or over-report problems.
  • A high level of insecurity in a country can have a small effect on the survey fieldwork. In some areas the presence of our field team can make residents generally suspicious, apprehensive as well as scared about their safety. However, the presence of trained interviewers, who are also natives of the locations, can suppress the suspicion and facilitate drawback-free fieldwork.
  • We do not include sub-samples of some of the most vulnerable groups, such as the disabled.

Additional data and triangulation

After the surveys are executed and the data is collected, we conduct a triangulation workshop in joint effort with our local partner organisation and with the participation of stakeholders. During these workshops we present the preliminary results of the report to the attendees, which provides the opportunity for dialogue around the data. This dialogue with stakeholders and local experts adds more depth to our understanding of the occurrence of and response to legal problems. The local data collection company also conducts qualitative interviews with users of justice, particularly the most vulnerable groups, with the objective of getting a more in-depth account of their experiences. Their stories, in their own words, illustrate and provide in-depth clarifications of the JNS data.

Aggregation of indicators and generalizations

Some measures on the Dashboard are displayed as aggregated scores (see Process and Resolution tabs). The scheme below explains the aggregations:
    • Quality of procedure
      • Mean of procedural justice (7 items), interpersonal justice (3 items) and informational justice (4 items)
    • Quality of outcome
      • Mean of distributive justice (5 items), restorative justice (3 items), informational justice (3 items) and functionality of the outcome (4 items)
    • Costs of justice
      • Out of pocket expenses (normalized distribution, rescalled to 1 to 5 scale), number of days (normalized distribution, rescalled to 1 to 5 scale) and stress and emotions (mean of stress (1 item) and negative emotions (3 items).

In order to assess the number of new problems per year, we have to take into account that people tend to forget problems that are less recent. See Pleasence, Balmer and Sandefur, Paths to justice, A past, present and future roadmap, 2013 (download here) for the following graph.

As a rule of thumb, which is on the safe side, we assume that people forget one third of problems occurring in 4 years. So the generalizations in the Justice Dashboard assume that the reported problems in the 4 years time period actually occur in three years.

To extrapolate the numbers from survey data, we use the World Bank data about the population above 15 years of age from the year in which data was collected for each country. We first calculate the weight using the following formula:
Weight= Population of the country in JNS year/(sample size*3)
Now to extrapolate the population, we multiply the weight with the sample size. Similarly, to extrapolate the number of problems, we multiply the weight with the number of problems in the sample.

The percentage of problems that have not been resolved is calculated using the following formula:
% people who did not take action + % unresolved problems + % on-going problems = % unresolved problems