Recommendations
on neighbour
problems

5.1 Monitoring outcomes for neighbours on an individual level

Guideline for neighbour problems / AFTERCARE: 5.1 Monitoring outcomes for neighbours on an individual level

Interventions and evidence explained

Most plausible interventions explained

During the orientation process of the available literature, we identified the following interventions as most plausible for improving (monitoring outcomes and ensuring sustainability) agreements in neighbour disputes:

Measuring neighbour conflict programs or neighbour intervention outcomes
Measuring outcomes for neighbours on an individual level

Measuring neighbour conflict programs or neighbour intervention outcomes

When measuring the effectiveness of an intervention applied in a neighbour dispute setting, the focus lies on the intervention or service. Three elements are measured:

An intervention is effective if it directly increases the likelihood that a desired outcome will occur, and that it does this independently of the effects of other concurrent factors, which may also potentially increase the likelihood of that outcome occurring.
An intervention is efficient to the extent that it achieves a desired outcome economically, with minimum waste of resources and effort.
An intervention is cost-effective if it achieves a desired outcome more efficiently, with a lower resource cost, than a specific alternative intervention (Forell, p. 5).

Examples of such neighbour disputes interventions could be community courts, neighbour mediation and provision of advice and information by community centers.

For the purpose of practical and applicable legal assistance service evaluation, outcomes must be specific, meaningful (relevant), measurable and proportionate to the program (Forell, p. 7).

An approach to evaluating the effectiveness of legal services is to consider how well services reflect principles which the research evidence-base suggests should improve individuals’ access to justice. This can be done through a community-wide focus (Forell, p. 8):

Measuring outcomes for neighbours on an individual level

The following outcomes in neighbor disputes can be measured on the level of an individual, through surveys or other means. Rather than measuring general outcomes of programmes and interventions, these outcomes are tailored to individuals’ needs. People involved in neighbor disputes or looking to prevent them in the future could, for example, seek to achieve the following specific outcomes:

Respectful communication
Less nuisance
Repairing relationships
Repair or compensation
Solutions to border issues

Good neighbour agreements point to the following other examples of outcomes for neighbours to achieve (Croucher, p. 36):

Being a good neighbour
Taking pride in where they live
Respect other people’s property
Look out for the neighbours [maintaining a good relationship with the neighbour]
Take care of the environment
Be part of the community [maintaining a good relationship with the neighbourhood)

According to the study by Forrest and Kearns, social cohesion and social capital within a neighbourhood enhances the well-being of neighbors and prevents conflict between them. Outcomes related to social cohesion are (Forrest and Kearns, p. 2140):

- Empowerment. People have a voice and feel listened to. They are involved in the dispute process and can take action to initiate change.
- Participation. People take part in social and community activities.
- Associational activity and common purpose. People co-operate with one another through the formation of formal and informal groups to further their interests.
- Supporting networks and common purpose. People co-operate with one another through the formation of formal and informal groups to further their interests.
- Collective norms and values. People share common values and norms of behaviour.

Trust. People feel they can trust their co-residents and local organisations responsible for governing or serving their area.

Safety. People feel safe in their neighbourhood and are not restricted in their use of public space by fear.
Belonging. People feel connected to their co-residents, their home area, have a sense of belonging to the place and its people.

Selected interventions for comparison (defined as a PICO question)

For parties looking to monitor outcomes of a neighbour dispute resolution process and ensure sustainability (improving), is measuring program or intervention outcomes or measuring client outcomes more effective for well-being?

Search strategy

The databases used are: HeinOnline, Westlaw, Wiley Online Library, JSTOR, Taylor & Francis, and ResearchGate.

For this PICO question, keywords used in the search strategy are: solution, outcomes, monitoring, decision-making, aftercare, sustainability, enforcement, neighbour, disputes.

Assessment and grading of evidence

The main sources of evidence used for this particular subject are:

Suzie Forell and Hugh M. McDonald, Evaluation of legal service delivery: challenges, opportunities and work towards a framework (2017)
Karen Croucher, Anwen Jones and Alison Wallace, Good Neighbour Agreements and the Promotion of Positive Behaviour in Communities (2007)
Ray Forrest and Ade Kearns, Social Cohesion, Social Capital and the Neighbourhood (2001)
OECD, Equal Access to Justice for Inclusive Growth; Putting people at the centre (2019)
Alice M. Black and Garee W. Earnest, Measuring the Outcomes of Leadership Development Programs (2009)
J. Biggs, and C. Tang, Teaching for quality learning at university: What the student does (2011)

Quality of evidence and research gap

The paper by Forell and McDonald reports ideas and progress towards establishing a framework that engages with the evaluative and service environment challenges to identify clear, modest but rigorous opportunities for evaluation. Noting the ‘tool kit’ of legal assistance strategies deployed by Australian Legal Aid Commissions, it serves as a framework to identify what service ‘works’, for whom, when, to what end and why. The study relies on a series of surveys conducted by a large study group. Therefore, the strength of evidence of this study is regarded as moderate.
In November 2005 the then Office of the Deputy Prime Minister (Communities and Local Government) and the Home Office commissioned the Centre for Housing Policy to undertake a study of the use, extent and effectiveness of Good Neighbour Agreements in promoting positive behaviour in England. The research concocted by Croucher, Jones and Wallace involved a survey of social landlords including: local authorities, Arms Length Management Organisations (ALMOs) and Registered Social Landlords (RSL); focus groups involving representatives of national organisations, practitioners and tenant representatives; and twenty in-depth interviews with landlords across England who were operating or had considered introducing Good Neighbour Agreements. The research found that almost half of the 221 landlord organisations that responded to the survey operated some sort of scheme to promote positive behaviour and nearly a fifth of respondents were considering developing a scheme. Only nine per cent of respondents had considered and rejected the development of such a scheme. This source is based on a large empirical study and the strength of evidence can therefore be regarded as moderate.
The paper by Forrest and Kearns is a literature review that explores initially and briefly the idea that societies face a new crisis of social cohesion and outlines the key dimensions of societal cohesion. The core of the paper is then devoted to an examination of where the contemporary residential neighbourhood fits into these wider debates, particularly in relation to the interaction between social cohesion and social capital. In this context, some of the key debates around the concept of social capital are outlined. In moving beyond abstraction, the paper also shows how social capital can be broken down into relevant domains for policy action at the neighbourhood level and how concepts such as social cohesion and social capital can be operationalised for research purposes. The strength of this source can be classified as low.
The OECD report (an expert opinion) looks at how governments can ensure that everyone has access to justice, and that justice processes and services are responsive to people’s needs. Based on lessons derived from people-centred service delivery, the report identifies access to justice principles and promising practices, as well as measurement tools and indicators to help countries monitor their progress. It sets out a framework for people-centred service design and delivery that can be applied to the entire legal and justice chain. Drawing on over five years of research and collaboration with OECD member countries and partner economies, the report contributes to our collective understanding of effective access to justice and the crucial role it plays in inclusive and sustainable growth and development. The strength of this source can be classified as low. We have bumped up the strength from very low to low, considering the high level of agreement from other literature sources on conclusions and implications in the report.
The study by Black and Earnest represents the first attempt at providing a comprehensive method to evaluate and measure leadership development programs on a post-program level. This study is not focused on neighbour dispute programmes, however it provides insights in advantages and disadvantages of measuring programme outcomes on a general level. This is a small empirical study and the quality of evidence can therefore be classified as moderate.
The article by Biggs is an expert opinion, focused on ways of measuring education programs. Similar to study by Black and Earnest, this article provides insights in advantages and disadvantages of measuring programme outcomes. The quality of evidence of this source can be classified as very low.

According to our research method, we grade the evidence comparing programme-outcome measuring and individual-outcome measuring as very low.

A considerable research gap can be identified when comparing two ways of measuring outcomes. More research can be conducted on how to specifically measure outcomes of individuals dealing with neighbour conflicts.

Comparing the two interventions

Desirable outcomes of the interventions

Measuring neighbour conflict programs or neighbour intervention outcomes	Measuring outcomes for neighbours on an individual level
Studies have been conducted to investigate the effect of services on the justice system (such as neighbour dispute services) as a whole, which results in valuable information that can improve the system for people in the long run. While there are clear potential benefits to clients, there are also system benefits. Such an approach thus potentially yields important information about the value and benefit of relatively low-cost ‘upstream’ legal assistance services in terms of higher-cost ‘downstream’ justice system costs. (Forell, p. 10)	If different sectors work together to address common outcomes for shared clients, the added value of legal services can be identified automatically. “Different sectors may work together to address common outcomes for shared clients [for example people in a neighbour conflict]. Herein lies a particular opportunity – to work with health agencies (who often have far more developed data on outcomes) to consider the difference adding legal services makes to [neighbours]. Clearly articulated outcomes [for individual neighbours] provide the opportunity to compare the impact of potentially disparate services on those outcomes – recognising that a range of services and systems may affect these outcomes, particularly at a high level.(technological or otherwise).” (Forell, p. 16).
There is a need for a shared understanding across and within organisations and jurisdictions about what neighbour dispute ‘interventions’ or services are, in order to identify what kind of services achieve good outcomes. “… common units of ‘work’ or ‘measure’ are vital for legal assistance service evaluation. Thus, another valuable stream of work may be to identify a set of ‘standard’ services to provide the basis for comparison across jurisdictions. Overall, if we know what the strategies are (or are made up of), what they are trying to achieve and for whom, where they are based, etc., we have a basis for comparing what types or combinations of services achieve outcomes and building a picture of what works.” (Forell, p. 16-17).	When outcomes for individuals are measured and improved as a result of a service/policy, this will reduce government expenditure. A number of studies show that legal assistance programmes are found to contribute to improved outcomes for individuals, preventing clients from requiring emergency shelter and related healthcare and social services. This helps reduce demand for government expenditure and other social costs, such as unemployment, disrupted education for children, reduced productivity and family instability.
	“Insights from legal need surveys provide insight into the self-reinforcing nature of disadvantage and the ways in which unmet legal needs can contribute to a cycle of decline which inhibits economic productivity and contributes to further social exclusion.” (OECD, p. 45).
	“People’s needs and experiences are key to identifying innovation potential and provide the rationale for reflecting on the delivery of legal and justice services. The perspective of individuals, families and communities provides important input from the “outside in” about how this potential can best be realised, by suggesting how services can be more responsive to people’s needs and through the co-development of reforms guided both by service providers and users.” (OECD, p. 56)

Undesirable outcomes of the intervention

Measuring neighbour conflict programs or neighbour intervention outcomes	Measuring outcomes for neighbours on an individual level
“In program evaluation, one must be careful of response-shift bias when using quantitative methods. Response-shift bias occurs when individuals have rated themselves at one time, from one perspective, and then change their responses later because their perspectives have changed” (Black and Earnest, p. 188).	Modest, measurable and meaningful individual outcomes are required, not individual broad goals. Legal needs research draws attention to the experience of those with unresolved legal problems and limited capability to address those problems. It articulates a problem to be addressed by legal assistance services – and suggests a set of evidence-based desired outcomes. However, ‘meeting legal need’ or ‘improving access to justice are broad goals, which while important – do not lend themselves to effective evaluation. (Forell, p. 14-15)
When collecting data from programme participants in measuring programme results, it is difficult to establish a baseline. “Program participants may have limited knowledge at the beginning of a program, which prevents them from determining their baseline behaviors. By a program’s end, the content may have affected their responses” (Black and Earnest, p. 188).
“An exclusive focus on [programme] measurement is results or product-oriented – program outcomes are focused on the end goal of an experience, and therefore neglect what [people] take away from the process of achieving that end goal.” (Briggs and Tang).

Balance of Outcomes

Taken together, the available research suggests that measuring outcomes for neighbours on an individual level is preferred over measuring the outcomes of neighbour programmes or interventions. The main reason for this is that individual outcomes is the most effective way of understanding where improvement in service delivery and innovation is needed. Furthermore, measuring outcomes on an individual level will automatically provide insights into outcomes of interventions/programmes. When measuring outcomes on an individual level, it should be noted that the outcomes should not be broad goals.

On the other hand, when measuring outcomes on programme level, there is often the issue of response-shift bias. Additionally, it is difficult to set the baseline when measuring programme outcomes. Therefore, measuring outcomes for neighbours on an individual level is preferred.

Recommendation

Taking into account the balance of outcomes, measuring outcomes for neighbours on an individual level and the quality and consistency of the evidence, we make the following recommendation: For parties looking to monitor outcomes of a neighbour dispute resolution process and ensure sustainability, measuring outcomes for neighbours on an individual level is more conducive to well-being than measuring neighbour conflict programs or neighbour intervention outcomes.