And finally.....Country Ownership and its Measurement - Part 4 of 4

Preamble: this is the last of a 4 piece series. 

• Part 1 dealt with why country ownership matters to us now. (last footnote of this entry not withstanding)
• Part 2 dealt with the understanding of ownership from large development aid actors, what our role is in advancing and measuring ownership.
• Part 3 discussed dimensions and metrics of ownership, then addressed the methodological choices we must make as happening on a rugged, and even dancing landscape.
• This is the 4th and final entry, where I discuss an fundamental choice we must make in our measurement approach. You probably should read at least Part 3 before proceeding -- the graph below is explained in Part 3.

Practically, what does all this this mean?

This discussion may have gotten a little ethereal for some, so let me try to bring it back to the practical and for this, we need to get back to the “why” of ownership assessment and consider our options. 

Imagine then that your options for measurement are between B5 (limited subject engagement) and B7 (limited objectivity), what option should you take? 

Let’s consider the options, starting as far to the left of the spectrum as we can:

We have far more experience measuring capacity than ownership. Given that capacity is conceptually part of ownership, and that ownership is even more of an abstract concept, I’ll stick to a capacity measurement example.

The more informative measures require a high level of locally relevant detail, which is very hard to obtain from the outside. Consider a simple capacity indicator, for example in human resources for health management: the percentage of a specific type of personnel actually available over a given year to perform a specific function. This seems like a very objective measure, somewhere toward B1. But experience shows that to make the indicator most informative (and useful for decision-making), you need to be guided through the complexities of public administration procedures and rules of the specific country. Coming to the proper definition of the type of personnel, what being “in the plan” means, and where the responsibility for ensuring that key positions are filled actually lies, is challenging from an outsider’s perspective. Without having a “guide” into the local health system under investigation, proper and meaningful measurement will be very difficult. Of course it gets more and more difficult once you start asking about things such as shared accountability, institutional ownership and political will.

So, staying on the left side of the spectrum, first you will struggle tremendously finding valid measures—it might be possible for a research exercise, but will certainly be challenging for a time-bound monitoring activity. (I am speaking of very practical constraints, for example getting the right staff in the right office of the MOH administration to help you figure out why this register (rather than the one anticipated) is the right one to get your denominator from.) Even assuming that you do get at measures, which can be considered valid, what happens if you were so far on the left end of the spectrum (B1 or B3) that local stakeholders are unsure of you, what you have measured, and what this even means? What is going to be the value given to your measure? Even for capacity measures, B1 or B3 are going to be uncomfortable spots to be in, unless you have the power to impose an audit and make your own rules.

If we go back to ownership, and from the perspective of why you wanted to measure it, you certainly won’t be sending actionable signals to those constituencies, even if you feel good about your measure. Like a tree falling in the proverbial forest, you might provide a valid indicator, but if no one is here to believe its signal, was it worth it?

You consequently are forced to move toward more participatory engagement of stakeholders to first determine what measures are meaningful and then define them operationally. You have to push through and over B6, and this takes you to B7.

You now have measures for which there is “buy-in” and cultural translation from stakeholders. Presumably, to have gotten such a buy-in you have developed with them a purpose, an action-orientation for your measures. Nobody claims that getting there is necessarily an easy road, but you are essentially working and measuring from some inside-the-system perspective. Having built rapport and a clarity of purpose along with some trust, your external “expert” voice will both carry more weight, as well as be kept in check by actors in the process you are measuring.

Your data might come to inform the stakeholders, but of course now your biggest concern is the external validity of your measure. If you present your findings you will dread questions from researchers with those letters after their name. The question is, having moved from B5 to B7 and gained buy-in and internal validity of your capacity or ownership questions, was it worth it if the external validity of your measure is now challenged?

My best answer at this point is, yes. And here’s why:

1-     The sole reason why you wanted to measure ownership in the first place was to engineer change with these same stakeholders. The first option (B5), leaving these stakeholders to wonder what it is that you measured does not help your goal. If measurement is here to guide change management, what is the value of a better measurement which means little to the change managers?

2-     Your (B7) measurement may have—certainly has—flaws but if it serves to inform and guide an authentic process of planned change, you have a foundation to build upon. Managers deal with uncertainty every day anyway. You will have lost some of the precision on the details, but probably gained validity on the big picture. Actually, actors of the local system will have an incentive to help you improve your measures over time—measures inform change, but change also informs (better) measures. Your measurement expertise will now be able to support a management process, rather than chase “data use”.

By starting from B7 you can influence a change in the landscape, and the possibility of moving toward better, more reliable measures as you promote more ownership – and wasn’t that the goal to start with?[*] I offer two equations as a summary:

I. {Ideal (SMART) Measure} minus {Internalized Meaning} =  {Sexy Research, but No Signal for Change Management}

II. {Imperfect Internalized Measure} = {Sub-Optimal Signal for Change Management} plus {Potential for Improvement over Time}

So, II might be more conducive to guiding change, even if--and that is a clear risk--"sub-optimal signals" carry the risk of being misleading. Hence the need for solid M&E professionals to help us manage this risk.

In conclusion, why you wanted to measure ownership, in an imperfect ‘rugged’ world, must lead you to choose to lean toward the right hand of the spectrum, to respect the process, and through that process improve the quality of your measures, rather than aim for an illusory perfect measurement in search of meaning and later begging for "data use".

I do not dismiss the importance of finding good measures of institutionalization and other elements afferent to ownership, or the risk inherent to "imperfect measures". And my argument is not about being satisfied with qualitative stakeholders’ perceptions. It is about the process we need to use to produce metrics and what must come first. 

Let me summarize this complexity in one sentence: You do not measure ownership without the owners.

Guess, it is simple enough after all

We -- and this "we" must be a true "we" -- have our work cut out.

Eric

[*] For complexity geeks, the reason for which it is easier to go from B7 to B5 than from B5 to B7 is because the landscape we have drawn is not only rugged, but it is also ‘dancing’ and changing. Starting from B7 and engagement of stakeholders, you may see the landscape change so that more reliable signals can be picked up (toward B5) without losing the sense-making of stakeholders’ involvement. But if you start at B5, you may see the next peak get higher.

Last note on "dancing landscape", it seems that "country ownership" is falling off the PEPFAR lexicon... so, to be continued I guess.

Acknowledgement: I owe the concepts of rugged and dancing landscapes to Scott E Page's presentations and books.