Collective action for improving health services. Why the evidence will continue to be mixed and what to do about it.

2016-2019 has seen a proliferation of large studies, donor reviews and evaluations published on social accountability (especially in health), with seemingly contradictory findings. Yet, if we go behind the findings, there are key common lessons for practice, including those studies with null results. These lessons include, but are not limited to, the role of high-quality community-based facilitators, the value of collaborative approaches, and contributions to system strengthening and relationship building. We make a start in exploring these lessons across different studies. We aim to encourage new syntheses[1], going beyond randomized field trials. These are, badly needed to help guide practitioners and challenge researchers to explore questions, which build further knowledge, not on whether it works, but how, why, and in what circumstances.

Introduction

Social accountability[2] as a practice has different definitions, different frames of reference, different audiences and purposes. Different families of human development practice may lay claim to it, from human rights and social justice advocacy; democracy, governance, and participation; organizational science and systems dynamics; to more programmatic concerns for systems strengthening, quality of service delivery, and equity. Different framing will place social accountability as a goal, or as a process. Both operate on different timescales and levels of complexity. More importantly, both framings - goal and process - are legitimate.

Governments may react differently to these different agendas and framing, and, additionally, different layers of government may look at social accountability differently based on where “systemic pressure” is applied. We could also underline that a participatory process bringing clients and providers together to collaborate and problem solve, may generate genuine expression of social accountability (one party feeling accountable to the other within a dialogical relationship) without “social accountability” having ever been articulated as an approach, much less as a project.

By itself, this complexity can already explain some variance in demonstrated results. We have seen increasing research on the impact of social accountability interventions on access to and quality[3] of services, as well as experimental research on population level outcomes (mortality, malnutrition, etc.). After multiple single intervention randomised control trials, however, debate still rages within research, policy and donor circles, about their definitive impact for development outcomes per se.

Indeed, there was much collective hand wringing last year when two high powered randomised control studies in health landed with null results.[4] Yet another large RCT in India found the opposite - dramatic effects on stunting and vaccination rates.[5] 

Many social accountability practitioners (implementers, donors, and governments) have strong positions on its value from their experience, and practice is moving forward.[6] The risk for the community of interested parties is to see each positive RCT as a final validation, and every negative RCT as a disqualification. Evidence challenges are important fodder for scientific and intellectual debate. Research needs to continue, but we suspect that the final finding of “trial” approaches to evaluating social accountability will be “it depends” for a long time, and for a number of reasons.

Why a traditional reference to evidence will continue to be mixed.

Firstly, it is important to consider which outcomes are examined in these recent studies. In particular, negative findings came from RCTs testing for changes in population health outcomes. A number of donor evidence reviews, on the other hand – notably a DFID macro-evaluation, USAID’s Marshalling the Evidence review, and this year’s 3IE review suggest consensus for positive impact on intermediate service outcomes.

It’s when we draw the long causal chain to try and measure the development outcome that it becomes challenging. A blog is not going to satisfice the question of why that is, but there are inherent limitations to chasing the statistical significance of broad and somewhat distal systems interventions on health outcomes.

Secondly, the devil - or in this case, learning from RCTs - is in the detail. Social accountability is messy; we need to learn from all opportunities, while understanding the limitation of each study. RCTs are powerful tools, but relying on their finding without discussion of theory, context and model limitations, is probably one of the worst things we can do in applying science to development issues.[7],[8],[9] The complexity involved in genuine social accountability mechanisms include many necessary, but not sufficient context and process conditions, which may demand more dynamic and contextually grounded forms of investigation, from realist evaluation, to developmental evaluation. (All of which also have limitations, no doubt about it.)

Third, theory is important, and we are challenged to imagine management, political science, sociological, or systems science theories, supported by evidence, which would posit that complex human endeavours can succeed and endure without accountability. This accountability can take different forms (administrative, legal, financial, social), which can complement, or even substitute to another over time on a particular issue.[10] Some leading thinkers about social accountability have suggested that the argument should simply be framed around accountability itself.[11]

Furthermore, if accountability is a first principle for human systems, then the questions might have to move from “Does it work?” to “How do we balance different types of accountability to optimize results?”[12] “What conditions allow for social accountability to be stepped up when administrative accountability starts to lag?” and “How much can we get from institutionalized social accountability mechanisms?” “What are central mechanisms from data to negotiation, which optimize the interaction between state and citizen in the delivery of public health services?”

Finally, trial methodologies framed around testing survival of patients at a time T, may need to evolve seriously to consider the dynamical processes (at times T1, T2…Tn) through which an intervention (labelled “social accountability” or unlabelled but creating the space for it) can alter social structures and processes, which may—under certain preconditions—themselves contribute to: (1) improving a public service, (2) establishing a norm for sustaining a public good responsive to the expressed needs of citizens, (3) providing standing capacity to identify and respond to new challenges and conflictual situations, (4) bringing a new progressive framing—hence unanticipated by original researchers—in the conditions of delivery of a public service, and (5) what else?

We certainly have not circled the block on research and evidence on social accountability, but tried to shift a bit the direction of our collective discussion.

But now, what lessons can be taken from the range of studies mentioned above, whether they had positive or negative findings?

Here are some of our top suggestions:

1. Community collective action occurs, with even minimum levels of support and encouragement. 

On this, the T4D / Harvard qualitative study (forthcoming) which will complement the published RCT showed how even a minimalist, time-bound intervention, while not ideal, still led to an average of five ‘social actions’ per village across 200 villages in Tanzania and Indonesia. That was approximately 1,139 social actions. If we consider what it takes to enable communities to speak up, then this is quite a positive outcome for the communities involved. The thorough qualitative work that has accompanied this RCT, mirrors much of what is captured in the grey literature for this work, and this makes it particularly valuable. Without this, the summary message that “it didn’t work” [to effect long causal chain health outcomes] could be seriously misinterpreted.

2. Quality facilitation is essential.

Social accountability approaches need facilitators engaged with communities to support the civic education and empowerment processes required for communities to negotiate with their governments. Investment in facilitators is crucial, and has now been highlighted through several RCTs, notably, the India study earlier identified. The importance of facilitation has also been highlighted over many years by INGOs. A qualitative report for USAID in 2016 highlighted that progress in health service function was less where facilitation weaker. [13] 

As practitioners, we honestly hope the India study puts to bed what practitioners have advocated for a long time – and that budgets for facilitation will follow. The Indonesian Government and many state governments in India understand this and fund their own facilitators, albeit with their own challenges, indicating a critical ongoing need for non-government facilitators.[14] Indeed, a global call for community development facilitators in every village is the single compelling goal of the 5-year-old Community Led Development movement (CLD) made up of 70+ non-profits interested to progress community empowerment across agency and indigenous/international divides.[15]

3. Non-confrontational collaboration is important and is valued by communities.[16]

There’s a large space for debate and definitional issues in addressing what is and is not “confrontational” in social accountability. We are not going to dive in at this point, aside from the short digression as an endnote.

The point that we want to note here is the positive finding for the value of non-confrontational collaboration, which comes from two in-depth studies, undertaken over the same period in Indonesia by Harvard University and for the World Bank’s Global Partnership with Social Accountability. It is also highlighted from a 3IE systematic review on citizen engagement, which found that:

“Citizen-service provider engagement is more effective when implemented through phased, facilitated processes that are framed collaboratively, as opposed to one-off accountability meetings that are interpreted as confrontational.”

This argues at a minimum that there are certainly ample time and space for the added-value of conflict-sensitive, problem-solving approaches to social accountability

A digression, if you allow--When governments, unfortunately not-so-rarely, decide to be blind and deaf to the aspirations of people, or of particular groups of people, giving power and voice to people will be more political and more confrontational at some point. This reflects a difference in framing of social accountability, as ‘projects’ versus ‘movement’. We are not going to debate here whether one form of social accountability is better than the other, or the possibility of an approach to social accountability “in the middle”.[20] We believe that different degrees of ‘confrontation’ will be needed at different times and in different places. Projects working within the framework of bilateral and multilateral agreements will likely be biased toward pragmatic and technical contributions—changing systems seemingly ‘from within’. Social accountability movements trying to defy bias, prejudice, and injustice remain much needed in the world, and will continue to be a force for change putting pressure on public administration. This tension is not exclusive to social accountability,[21] but certainly social accountability is a space for living with this tension. Projects have to tread carefully on political issues, while movements naturally seize political platforms more forcefully. We do not believe in disqualifying one or the other approach, or pitting them against each other. We are seeking progress through a “yes, and” dialogic approach.

4. The first social accountability project supported by the Global Partnership for Social Accountability in Indonesia 5 years ago (completed last year) showed evidence of positive effects on systems strength, and relationship building with a network of groups and institutions, with shifts in power dynamics.

This obviously deserves more unpacking than we have space for here. Nonetheless, this is the first GPSA evaluation to capture it through a systematic concurrent evaluation, based on an initial theory[17] refined from the accrued evidence. GPSA has seen evidence for these types of changes in other projects, as have several other INGOs. An earlier Columbia University study in 2017 also highlighted positive system strengthening and development coordination effects in Zambia from the same intervention methodology.

Conclusion

These lessons will not be new to many practitioners. Indeed, they appear sometimes obvious after many years of field experience. However, the inherently political nature of social accountability interventions and their intangible processes make their research, monitoring and evaluation complex[18]. To forge ahead, we--practitioners, researchers, policy-makers, and donors--consequently need different syntheses approaches, with rigorous and realist frameworks, recognizing multiple sources of learning and knowledge, and not pseudo-experimental black box approaches, to forge ahead.[19] As more sophisticated questions are devised in context, as multiple knowledge sources are recognised and alternative methods supported, we hope to see deeper, more nuanced learning develop for social accountability. This learning will especially be needed from assessments within the different social and political contexts, to inform the adaptation and evolution of social accountability mechanisms. This is true for citizens within these contexts, and for governments committed to ambitious Sustainable Development Goals.

Sue Cant & Eric Sarriot                 Nov. 2019

Sue Cant is the Social Accountability Adviser for World Vision International. This blog post is the opinion of the authors, but the thinking evolved through many one on one discussions and, in particular, a group convened earlier this year, who we would like to acknowledge: Courtney Tolmie, Florencia Guerzovich, Marta Schaff, Vicky Bowdell and, in particular, Angela Bailey. These colleagues helped improve the text, and we maintain sole ownership of its shortcomings.

Notes:

[1] Note, excellent 3IE synthesis in 2014 (Rogers, at al 2014)

[2] Collective action or transparency and accountability or multi-stakeholder governance, or put simply, any activity that helps citizens make government accountable with a tangible result.

[3] Quality in the broadest sense possible includes dimensions of equity, appropriateness, etc.

[4] https://twitter.com/Integrilicious/status/1151574157882249216

[5] http://documents.worldbank.org/curated/en/340831537776397782/Mobilizing-Communities-for-a-Healthier-Future-Impact-Evaluation-of-Social-Accountability-Interventions-in-Uttar-Pradesh-India

[6] https://gpsaknowledge.org/blog/on-learning-about-transparency-accountability-and-how-to-move-this-debate-forward-part-1/

[7] https://www.3ieimpact.org/blogs/how-can-rethink-lessons-field-experiments-inform-future-research-transparency-participation

[8] R4D’s rich discussion of its own study findings (which were largely negative) has been an exemplar of commitment to intellectual rigor. Tolmie C et al., 2018 Presentation to American Evaluation Association ‘Speaking Truth to Power’ Conference, November 3, Cleveland Ohio

[9] See also: https://gh.bmj.com/content/3/3/e000811

[10] Who can argue that the social accountability generated by the #MeToo movement is leading to more legal and even administrative accountability? In global health, one of us witnessed first-hand a participatory community engagement process lead to the expression of community grievances for more balanced ethnic and linguistic representation of the communities served by care providers. Once public administration responded, this became part of the way to manage the district and administratively address this element of population needs. Sustainability of the outcome meant that social mobilization no longer was necessary to maintain the service outcome. Social accountability (those words were not used) had led to institutionalized administrative norm, at least on this one pain point.

[11] https://www.futurehealthsystems.org/news/2017/1/6/event-social-accountability-from-health-systems-strengthening-to-the-sdgs

[12] For quantitative researchers, maybe a Bayesian approach needs to be suggested—what do we already know about the requirements for accountability in human systems? How does that inform our next research question?

[13] USAID, 2016, The Maternal and Child Survival (MCSP) Program, MCSP Community Component/Tanzania Community Scorecard Documentation Report, December.

[14] Forthcoming, Guerzovich et al

[15] See: https://communityleddev.org

[16] This is not to say that contestation is not important. It is, but there are different approaches at different times that may work. 

[17] This was informed by a systematic realist review.

[18] Not to mention their implementation, as mentioned above

[19] Realist synthesis is one such methodology

[20] See forthcoming paper by Florencia Guerzovich and Maria Poli, which we will link to when available.

[21] http://systems4health.blogspot.com/2018/08/the-emergency-to-development-continuum.html