And finally.....Country Ownership and its Measurement - Part 4 of 4

Preamble: this is the last of a 4 piece series. 

• Part 1 dealt with why country ownership matters to us now. (last footnote of this entry not withstanding)
• Part 2 dealt with the understanding of ownership from large development aid actors, what our role is in advancing and measuring ownership.
• Part 3 discussed dimensions and metrics of ownership, then addressed the methodological choices we must make as happening on a rugged, and even dancing landscape.
• This is the 4th and final entry, where I discuss an fundamental choice we must make in our measurement approach. You probably should read at least Part 3 before proceeding -- the graph below is explained in Part 3.

Practically, what does all this this mean?

This discussion may have gotten a little ethereal for some, so let me try to bring it back to the practical and for this, we need to get back to the “why” of ownership assessment and consider our options. 

Imagine then that your options for measurement are between B5 (limited subject engagement) and B7 (limited objectivity), what option should you take? 

Let’s consider the options, starting as far to the left of the spectrum as we can:

We have far more experience measuring capacity than ownership. Given that capacity is conceptually part of ownership, and that ownership is even more of an abstract concept, I’ll stick to a capacity measurement example.

The more informative measures require a high level of locally relevant detail, which is very hard to obtain from the outside. Consider a simple capacity indicator, for example in human resources for health management: the percentage of a specific type of personnel actually available over a given year to perform a specific function. This seems like a very objective measure, somewhere toward B1. But experience shows that to make the indicator most informative (and useful for decision-making), you need to be guided through the complexities of public administration procedures and rules of the specific country. Coming to the proper definition of the type of personnel, what being “in the plan” means, and where the responsibility for ensuring that key positions are filled actually lies, is challenging from an outsider’s perspective. Without having a “guide” into the local health system under investigation, proper and meaningful measurement will be very difficult. Of course it gets more and more difficult once you start asking about things such as shared accountability, institutional ownership and political will.

So, staying on the left side of the spectrum, first you will struggle tremendously finding valid measures—it might be possible for a research exercise, but will certainly be challenging for a time-bound monitoring activity. (I am speaking of very practical constraints, for example getting the right staff in the right office of the MOH administration to help you figure out why this register (rather than the one anticipated) is the right one to get your denominator from.) Even assuming that you do get at measures, which can be considered valid, what happens if you were so far on the left end of the spectrum (B1 or B3) that local stakeholders are unsure of you, what you have measured, and what this even means? What is going to be the value given to your measure? Even for capacity measures, B1 or B3 are going to be uncomfortable spots to be in, unless you have the power to impose an audit and make your own rules.

If we go back to ownership, and from the perspective of why you wanted to measure it, you certainly won’t be sending actionable signals to those constituencies, even if you feel good about your measure. Like a tree falling in the proverbial forest, you might provide a valid indicator, but if no one is here to believe its signal, was it worth it?

You consequently are forced to move toward more participatory engagement of stakeholders to first determine what measures are meaningful and then define them operationally. You have to push through and over B6, and this takes you to B7.

You now have measures for which there is “buy-in” and cultural translation from stakeholders. Presumably, to have gotten such a buy-in you have developed with them a purpose, an action-orientation for your measures. Nobody claims that getting there is necessarily an easy road, but you are essentially working and measuring from some inside-the-system perspective. Having built rapport and a clarity of purpose along with some trust, your external “expert” voice will both carry more weight, as well as be kept in check by actors in the process you are measuring.

Your data might come to inform the stakeholders, but of course now your biggest concern is the external validity of your measure. If you present your findings you will dread questions from researchers with those letters after their name. The question is, having moved from B5 to B7 and gained buy-in and internal validity of your capacity or ownership questions, was it worth it if the external validity of your measure is now challenged?

My best answer at this point is, yes. And here’s why:

1-     The sole reason why you wanted to measure ownership in the first place was to engineer change with these same stakeholders. The first option (B5), leaving these stakeholders to wonder what it is that you measured does not help your goal. If measurement is here to guide change management, what is the value of a better measurement which means little to the change managers?

2-     Your (B7) measurement may have—certainly has—flaws but if it serves to inform and guide an authentic process of planned change, you have a foundation to build upon. Managers deal with uncertainty every day anyway. You will have lost some of the precision on the details, but probably gained validity on the big picture. Actually, actors of the local system will have an incentive to help you improve your measures over time—measures inform change, but change also informs (better) measures. Your measurement expertise will now be able to support a management process, rather than chase “data use”.

By starting from B7 you can influence a change in the landscape, and the possibility of moving toward better, more reliable measures as you promote more ownership – and wasn’t that the goal to start with?[*] I offer two equations as a summary:

I. {Ideal (SMART) Measure} minus {Internalized Meaning} =  {Sexy Research, but No Signal for Change Management}

II. {Imperfect Internalized Measure} = {Sub-Optimal Signal for Change Management} plus {Potential for Improvement over Time}

So, II might be more conducive to guiding change, even if--and that is a clear risk--"sub-optimal signals" carry the risk of being misleading. Hence the need for solid M&E professionals to help us manage this risk.

In conclusion, why you wanted to measure ownership, in an imperfect ‘rugged’ world, must lead you to choose to lean toward the right hand of the spectrum, to respect the process, and through that process improve the quality of your measures, rather than aim for an illusory perfect measurement in search of meaning and later begging for "data use".

I do not dismiss the importance of finding good measures of institutionalization and other elements afferent to ownership, or the risk inherent to "imperfect measures". And my argument is not about being satisfied with qualitative stakeholders’ perceptions. It is about the process we need to use to produce metrics and what must come first. 

Let me summarize this complexity in one sentence: You do not measure ownership without the owners.

Guess, it is simple enough after all

We -- and this "we" must be a true "we" -- have our work cut out.

Eric

[*] For complexity geeks, the reason for which it is easier to go from B7 to B5 than from B5 to B7 is because the landscape we have drawn is not only rugged, but it is also ‘dancing’ and changing. Starting from B7 and engagement of stakeholders, you may see the landscape change so that more reliable signals can be picked up (toward B5) without losing the sense-making of stakeholders’ involvement. But if you start at B5, you may see the next peak get higher.

Last note on "dancing landscape", it seems that "country ownership" is falling off the PEPFAR lexicon... so, to be continued I guess.

Acknowledgement: I owe the concepts of rugged and dancing landscapes to Scott E Page's presentations and books.

Country Ownership and its Measurement - Part 3 of now 4

This post is long overdue. In Part 1, we considered a bit of the history behind the current emphasis on country ownership—at least it was current when I started; in Part 2, we spent a little time on the crucial question of why we would want to measure country ownership, and we discussed some implications:

When it comes to ownership, the old saying “if you can’t measure it, you can’t manage it” begs the question: “should you be managing someone else’s ownership?” Since the obvious answer is “no,” we need to realize that developing, cultivating, or allowing ownership happens within the tension of desired transitions in roles between different entities. If we avoid the pretense that we are somehow “objective” and outside of the game of this transition, as donors, technical advisors, or evaluators, Part 2 concluded on the following question :

If ownership grows or withers from the net result of an interaction, a dialogue, a transfer of resources, an exchange in capabilities, the negotiation of roles in decision-making, then what is the point of ownership measurement if it exclusively focuses on the recipient? If I am the provider of assistance, or the donor, or the policy adviser, does it make sense for me to try and measure the recipient's ownership without questioning my role in this process?

This last blog is an attempt at the what and the how of measurement. Remember that we are dealing largely with monitoring and evaluation of country ownership, rather than research. In the former case, information (signals) has to be produced in a time frame and with a frequency aligned to management processes so that it can be used to inform management decisions. This is very different from a research exercise.

I hope to convince you that how we measure country ownership cannot be dissociated from why we measure it. And this has profound consequences.

Let's start at the very beginning (Von Trapp, Maria. 1965). Measuring ownership is going to require solving a great many questions, but first and foremost it's going to require making two choices. These choices are always made, but they are not always explicitly made with respect to alternatives and opportunity costs: 

1. The first thing we do with something complex is to break it down (reduce) it to elemental components. And we're going to have to decide which 'parts' of ownership we're going to focus on.

We saw in Part 1, how PEPFAR identifies four main components to ownership, namely political will and ownership, institutional ownership, capacity, and mutual accountability. I discussed briefly how there are circuitous relationships between these different things (ownership is part of stewardship, which is part of or the same as proper governance, which is part of capacity, which is part of ownership, for example). There’s also a nested Russian Dolls element as well (political ownership needs to translate into institutionalization of key values, processes, etc, which needs to translate into operational capacity and capability, which can be broken down into a number of capacity areas).

This is not a critique of any single assessment tool or conceptual model, but simply a recognition that, for one thing, we are trying to assess something which cannot be fully assessed, given that it involves layers of both institutional and human intentions, actions and reactions. We are trying to capture observable signals about something, which is at the same time social, psychological, institutional, and political. When trying to assess a whole by breaking it down, we gain in precision (reliability) in measuring specific pieces, but we lose in validity because of the glue and connecting pieces we are forced to discard as we proceed. It may be acceptable, even necessary, to do this, but we need to face that this is what we are doing.

There are other ways to look at ownership which are as valid as PEPFAR's approach. And much like capacity, each component of analysis needs to be broken down into more pieces. Here are some categories and sub-categories of analysis that have been proposed and used:

• The four components above (political will, institutional ownership, capacity, and mutual accountability) are examined in terms strategy, resource allocation, operational planning, implementation, and M&E;
• Other efforts have looked at 'readiness to implement country owned solutions' by considering country leadership, ownership and advocacy; conditions in the policy and planning environment; institutional dynamics: management, coordination and implementation; as well as the culture of learning and knowledge-based practices;
• Mutual accountability relates to a host of relationships: citizen, donors, internal accountability, with a number of sub-domains, notably finances;
• Ownership also translates at different system levels. In one of our efforts to assess ownership at district level, we had to consider operationalization of the national HIV plan at the district level (very close to the concept of institutionalization); institutional coordination of HIV care and treatment activities; and congruence of expectations between levels of the health system.
• We have also tried looking at stages of transition in roles (shared or divided between "external" and "internal/country" planners, implementers). Elements under assessment included health planning; service delivery; specific management functions (finances, supply chain, laboratory, human resources; supply chain management; laboratory management); training; health information; and capacity building.

As you can tell, we were trying to save some of that glue and connecting pieces that fall off the side. The point of this list is to show the breadth of elements that can be considered, and the potential depth of each element, not to mention layers (geographic and institutional) of systems and sub-systems. We break down the concept to simplify our life, and then realize that it's "bigger on the inside" (Who, Dr., 2007).

Nonetheless, this is the first choice and the first step of awareness:

• After we've mapped out the boundaries of our exercise / assessment / research, how are we going to break down the concept of ownership?
• How much of the glue and connecting pieces are we ready to lose in the process?
• How far down are we going to drill down?
• What will be the level of effort required by our measurement effort depends on our answer to these questions.

2. The second question is about our measurement process, and how 'emic' (from the inside) or 'etic' (from the outside) we will be in measuring ownership. Ok, in simple words: who's perspective are we taking in assessing ownership: that of external agents, or that of the purported owners? Are we trying to be objective or participatory?

You may have noticed that I gleefully dodged the issue of what type of metric we construct--this would require a longer treatment, and for now many approaches are on the table and none have been validated better than any other. Among the options:

• Scorings and ratings about perceptions of ownership in various components;
• Multi-item response indices based on discrete numerical scales (1-5; 1-10 usually), or anchored scales (where each numerical response corresponds to a descriptive textual "anchor" or "word-picture");
• Additive scales based on meeting a number of criteria (yes/no);
• Full qualitative analyses;
• Etc.

It is important to pick one approach, design, validate and use it appropriately, but the question I want to deal with here is far more fundamental. Let's say that you've stuck with four components of assessment of ownership used by PEPFAR at some level of a national social service system (health district, school, social welfare department, etc.), the key question will be whether you take one of two paths:

a- Attempt to measure ownership [of a program, policy, or project, in an institution or system] as objectively as possible, without bias and influence from the agents inhabiting the structures being assessed; or

b- Measure ownership with the actors and stakeholders-subjects of the process of taking ownership.

Now the first temptation—I use the word temptation because it usually leads to sins of program design –the temptation is to go for, "we'll blend the two approaches, and get a middle of the road tool."

I want to use a metaphor to illustrate how the temptation of a ‘middle of the road’ approach can be self-defeating.

Imagine that the methodological space we must travel from one measurement approach to another (objective versus participatory) is a landscape that we are going to walk across. In Figure A below, this landscape is a flat plain, where we can easily move from one point to the next, and stop wherever appropriate. There is an objective end of the spectrum (some sort of reliable external audit) and a subjective one (maybe asking five key informants, "how much ownership do you think there is in the province on X? Thank you."). The path between the two ends is flat, so there is no cost, or very little, to moving and stopping along. If a compromise is needed, all we need to do is move the design cursor to the ideal fair and balanced position to enjoy the best of both worlds.

This would be very nice, indeed, and it’s a pleasant illusion when your livelihood depends on maintaining the illusion. But I suspect that most choices about measurement are made on a different type of landscape. Imagine that we are not faced with  a nice clean plain, but instead have to walk through a mountainous landscape, one that looks more like Figure B. The path between the two ends is rugged, with peaks (B4, B6, B8) and valleys (B3, B5, B7, B9). It’s hard to push over a peak, and once you do, you roll down very fast into the next valley. Consequently a small inflection in design one way or the other will lead to quickly losing a lot of the features of the previous design [side note for the geeks, the peaks are “tipping point”].

What is important--essential--to understand here is that if we most likely live in a B-type world, rather than an A-type world,  we don’t get to make small adjustments to our hearts’ content, but rather we observe (or ignore) that small adjustments to our methods have big consequences. As soon as we move across points B4, B6, or B8, what results is not a small change in the balance of our methods, but a jump which can be far more consequential than we think (i.e. from B5 to B7).

(Note that our discussion has focused only on the ownership question, leaving aside that of the change process, called transition. We’ve had interesting efforts working with PEPFAR/CDC trying to move forward on this, but I don’t think it’s quite cooked and ready to serve yet. Only to say that providing information about the two sides of the equation makes a lot of sense and only strengthens the argument for a learning approach.)

Stay tuned for the next and final (!) entry on this blog series focusing on country ownership which will explore the practical implications of this discussion.