The Changing Trends of Development Funding... in Song

I was supposed to post Part 3 of our discussion of Country Ownership and its measurement. Sorry - getting delayed by a proposal.

In the meantime, microbanker.com has produced this smart marketing video. We could dissect the pros and cons of the approach, but the biggest pro is how much fun it is to watch this.

Enjoy...

Country Ownership and its measurement - Part 2

We social scientists would do well to hold back our eagerness to control that world which we so imperfectly understand. The fact of our imperfect understanding should not be allowed to feed our anxiety and so increase the need to control. (Bateson, 1972 cited by Morin).

A- Where we left off
In Part 1, I set the stage for the great interest we currently have in country ownership: a combination of missed opportunities of the past, renewed intentions (global public private partnerships, Paris declaration, Abuja, PEPFAR, PMI, and the GHI for us in the U.S.), and the unattractiveness of being caught admitting to the neo-colonial violation of the principle of country ownership.

I hinted in Part 1, that Country Ownership may have become a "sacred cow" in some global organizations, whereby we don't know why the cow is in the middle of the road, we're not sure what to do with it, we're not entirely certain that it's really sacred (I mean, we have deadlines), and even though we might end up just driving around it and moving on as planned, we want to be sure to pay due respect. Finally, I referenced a currently widespread framework at least in discussions, if not in use, which is the one developed by PEPFAR through McKinsey & Company. (The 2010 GAO Report to Congress provides a measure of how seriously the concept of Country Ownership pervaded USG thinking at all levels in preparation for PEPFAR II.) This framework gives us a starting place, with four dimensions and multiple sub-dimensions of  country ownership:

1- Political will and ownership. [UNAIDS correspondingly identifies robust national strategic plans with smart investment decisions, strong political engagement and inclusive leadership, and given the HIV/AIDS focus, integration of HIV into broader health and development strategies.]

2- Institutional ownership (an earlier version included community ownership as well). [UNAIDS identifies here full engagement of civil society, communities and people living with HIV.]

3-Capacity [UNAIDS refers to: capacity development, high-quality strategic information, and effective coordination.]

4-Mutual accountability [UNAIDS considers strong partnerships with a shared responsibility and mutual accountability.]

B- On to measurement
So now we get to the sticky wicket of 'how do we measure these things?' Based on more than a few years and few projects' worth of experience with this, I'd like to point out some decision points you will have to face if you're serious about this. Let's deal with why, what, how and who of ownership measurement. In the process, we're going to discover that measuring ownership forces us to deal with Edgar Morin's paradigm of complexity. At the risk of distortion, I will borrow a summary of this paradigm as the recognition that "knowledge is a dynamic, unfinished process, and develops by way of self-criticism." When we deal with complex realities and processes, the "objective" perspective in observation and measurement might not be tenable, and ignoring this can lead us to misconstrue reality. The other part of the equation is the inherently incomplete nature of our knowledge about ownership. I don't think that a list of four dimensions of analysis, or eight items, or even 75 variables, is going to provide us with an absolute and comprehensive understanding of anyone's "ownership." This must be part of the "known unknowns" (hum... sorry about that reference).

Why do we want to measure ownership? An old mantra of scientific management is "if you can't measure it, you can't manage it." Since we want to build country ownership, see it thrive, and since ownership is part of our sustainability plan, how can we not want to measure it then? The GAO report congratulated PEPFAR for at least trying to have a baseline. Can't measure it, can't manage it. Makes plenty of sense.

However, it is already questionable whether anyone can actually build someone else's capacity. I mean, we have a pretty good sense that capacity building is an endogenous process, which makes use of opportunities provided through peers in a network, access to expertise, to knowledge, and more importantly applied knowledge in a supportive social system (which may be a project, a team, an office, a health center...). But still, the trainer, the coach, the consultant, the technical adviser might feel with some legitimacy that they built someone's capacity. Arguably.

But ownership? Can I build your ownership? Sure I can help you institutionalize procedures, if you want... I can help translate standing capacity into more supportive routines to support a goal, and you will internalize this, if you care to... I can help you build information systems which feed into accountability processes, which will sustain the pursuit of a shared goal, if you intend to give life to the processes... We see that, even more than capacity, ownership develops endogenously, internally within people and the social system in which they operate and construct their own role.

We consequently need to modify the mantra:

"You can't manage it if you don't measure it. But should you be managing it?"

Asking why we measure ownership sends us back to the dialectic nature of development assistance, or development cooperation:

• Most likely there are things we can and should measure in how the recipient of assistance is institutionalizing some processes, policies, and building capacity, but...
• One of the main reasons for measuring ownership should be about what the provider of assistance does. What are our ways of interacting with a national system in all its diversity? How are intentions versus old and bad habits turned into actual behaviors in development practice? How are our actions construed by the recipients of our support, in an emic way, i.e. based on their perspective? 

Did I mention power and money? Or would we rather not go there. And in this case, are we just trying to get around the sacred cow?

If ownership grows or withers from the net result of an interaction, a dialogue, a transfer of resources, an exchange in capabilities, the negotiation of roles in decision-making, then what is the point of ownership measurement if it exclusively focuses on the recipient? If I am the provider of assistance, or the donor, or the policy adviser, does it make sense for me to try and measure the recipient's ownership without questioning my role in this process?

Along these lines, if we measure ownership because we care about it and want to see it grow, and  if the nature of the recipient's ownership is a shifting, and fluid, complex reality, can we assess and measure ownership independently of a process which builds ownership? We're going to get into more of that when we get to the how, but let's consider a counter-example: when I carry out a survey to measure immunization coverage, I do not change the immunization status of children in a population. Can the same be said about ownership? Most often, no. The measurement of ownership is part and parcel of a larger package of intervention and interaction, through which roles and meaning are created by donors, implementers, assistance providers, and recipient. If my assessment process is going to affect ownership anyway, do I want an objective measure (the feasibility of which might be in doubt already) even if it decreases ownership in some way, or do I want a measure, which accepts a more subjective and humble process, but may potentially improve ownership? This has to be part of the continuation of this discussion.

We'll see in dealing with the what, that there might be sub-points of ownership, which can try and avoid this problem, but it will always be there when we deal with the core of the ownership question. There are also differences between assessing ownership as a research endeavor versus a monitoring and evaluation one, but our premise here is management-oriented and the need for monitoring and evaluation.


I promised to keep these blogs at less than 900 words, however, so the what we measure and the how we measure it will have to wait for next week.

The take home message for today is that if we want to measure ownership, we should be clear about why we want to do so. And if we address why, we are brought to consider that we are not the objective external observer ready for an assessment, we are part of a relational and developmental dialectic. At the end of the day, either we accept to be part of the question ourselves, or we are probably asking a pointless question. 

To be continued... Feel free to disagree, agree, comment and enrich by leaving a comment.

Thanks and Cheers,



Eric

PS: We should not be surprised by the tension in objectivity-subjectivity introduced by the paradigm of complexity. Consider something much simpler and more objectively measurable than ownership. Consider billiard balls on a pool table. Our models can almost perfectly predict what a ball will do when hit by another ball at a given speed and angle. It gets however harder when you deal with a number of balls (as complexity increases). To correctly compute the 9th impact you need to take into account the gravitational pull of someone standing next to the table. And to compute the 56th impact every single elementary particle of the universe need to be present in your assumption. (Reference: Taleb in Black Swan, Grey Swan, Sustainability.) Now think about this: there are more than 56 people involved in creating country ownership; and each one tends to be just a little trickier to manage than a billiard ball...

---- Thanks to Reeti Desai for editorial assistance, respectful of my ownership. :) ----

Robert Chambers on Complexity in Development

This is not new, but interesting to hear an older wiser voice on this topic.
http://www.youtube.com/watch?v=jzvwDDQgyJE

Country Ownership and its measurement - Part 1

“To us, country ownership in health is the end state where a nation’s efforts are led, implemented, and eventually paid for by its government, communities, civil society and private sector. To get there, a country’s political leaders must set priorities and develop national plans to accomplish them in concert with their citizens, which means including women as well as men in the planning process. And these plans must be effectively carried out primarily by the country’s own institutions, and then these groups must be able to hold each other accountable.”

Secretary of State Hillary Rodham Clinton, June 1, 2012

I want to say a word about Country Ownership and its measurement, and notably how the paradigm of complexity (don't be afraid! I'll explain this with a pool table analogy; it won't hurt) is going to force us to try a little bit harder on this one. But first, why is there so much talk about 'Country Ownership' and its measurement nowadays?

There's a proper answer to that question, and then there's another one:

[1] Appropriate answer is: because we're in the era of the Paris Declaration, the Global Fund Country Coordination Mechanisms, and for us US-centric folks, USAID Forward, which I think is a great step forward (OK - you'll never know whether I mean it, but actually I do). In fact, just read UN, DfID, French Cooperation, GAVI, even IMF and Bank statements, the Abuja and other declarations. There's no getting around Country Ownership (CO). The concept and the intention are here to stay. We all agree.

[2] The less satisfying but nonetheless truthful answer is that emphasis on CO at this juncture is an implied recognition that we, sorry capital "We" (from the multilateral, to the bilateral, from the donors to the implementers, from the DAC countries to ODA recipient countries) have essentially failed to respect this principle from roughly the birth of the UN (1945), or the birth of US international development assistance (1949-1950) until not so long ago. No need to point fingers. We tried our best. Some folks said it mattered, but there was a lot to do, and you know how things go. We had to get the job done. And the tension between 'getting the job done' and 'respecting CO' is still with us. PEPFAR I was launched in 2003; the Paris Declaration was right smack in the middle of its implementation (2005), and by 2009, we were in PEPFAR II with CO front and center in intentions, strategies and priorities. On an overlapping timeline, the Global Fund started in 2002 with the stated principle that it would "work on programs that reflect national ownership and respect country-led formulation and implementation processes." Its inception saw a few cautions raised about the possibility of conflicts between the principle of CO and performance-based funding (not the principle of PBF, which as a mantra was inseparable from CO, but the reality of it), and the 5-year evaluation came out with the walking-on-eggs finding that CO was "in need of review." I don't think there was a 10-year evaluation. Hum. (I'm not pointing fingers; I was there, and don't get me started.)

So, we're all learning and trying to take CO seriously, but we're struggling. And of course, it's tempting to think that it's just not our fault... it's their fault. We're progressively getting clear on the fact that they (the countries, the partners, the governments) need to have ownership, so we can transition out. Given that you can't manage it if you can't measure it, we're now in the business of measuring ownership.

So what are we talking about?

PEPFAR reasonably looks at ownership as a multidimensional concept including:

1- Political will and ownership

2- Institutional ownership (an earlier version included community ownership as well)

3-Capacity

4-Mutual accountability

Let's start with a few observations, before moving to measurement.

First observation: ownership is part of ownership -- twice. I'm not disagreeing with the concept, just pointing out that not only is ownership multidimensional, but apparently it's also multi-level. To the Complex Adaptive Systems (CAS) geeks, yes, we're talking about nested or inter-related sub-systems.*

Secondly, capacity is part of ownership. Sharon Arscott-Mills and I once ran a game at the CORE Meeting called "name your theory". We gave participants literature references, and they had to combine the words "capacity", "ownership", "performance", and "sustainability" in as many logical statements as possible. We found out that you can pretty much randomize the order of the words and come up with a theory that is backed by literature. Joke aside, capacity is part of ownership pretty much to the same extent that ownership is part of capacity. This is interdependence or co-linearity between variables. And it makes sense: if you don't have capacity, how can you own the responsibility for something? But if you don't care and own something, since governance and leadership is part of capacity, how much capacity do you really have to sustain it? (See what I mean about naming your theory?) There's a dynamic tension here. We're not talking about building blocks, but maybe something like electrons and protons, dancing around to form an atom.

Finally, it's very appropriate -- actually brilliant -- to have included mutual accountability in this model. This demands a relational element of analysis when considering ownership. No man and no institution is an island. Even our capacity is built through and on relations. And a country is built on many, multiform, diverse relationships*. How could a country own anything, without the sharing of, if not a common goal, at least compatible aspirations?

These observations now set the stage for the particular predicament we're facing with measurement of ownership. But a 900 word blog is already too long. So, this will have to be for part 2.

Speak of a Da Vinci Code cliffhanger...

Eric

* at this point, CAS geeks are drooling about fractal patterns. Just leave them alone; they can't hurt anyone.

Evidence in Evaluation -- beyond a [faith-based?] approach to the dominant RCT-for-all approach?

As we try to build capacity, sustainability, ownership, results, the question of evaluation comes around frequently. We're naturally regularly faced with the "what's the evidence?" question - and behind that, is the frequent not-so-silent assumption that without an RCT (Randomized Controlled Trial) or some pseudo-experimental version of the same concept, you really don't have anything to show.

RCTs are great, when they're applied on questions and in contexts where they are appropriate and feasible. In my view, the "golden standard" image that they carry has more to do with our dominant professional cultural belief system than serious methodological discussion of the issues at hand. Talk long enough with people who actually conduct RCTs, and you'll find out that (1) it doesn't work all the time, (2) interpretation can be very sensitive to small tweaks in the model, and (3) most of the time on the current and burning complex development questions of the time, they require that parts of an intervention be parceled out for evaluation, or that an intervention be modified to fit the RCT design. In other terms, it's like going for your physical and having the nurse say: "please bend your knees as my measuring rod doesn't reach that high!" So, golden standard? Not always.

RCT implementers themselves are usually careful about their claims (except of course when responding to call for proposal; we're only human). The problem is with the mass of non-specialists who may have skipped the subtleties and have developed a nearly faith-based attachment to RCTs as the answer to every question. Disciples are always the most problematic...

Let me stop venting. I just wanted to point out to some useful and fairly recent publications, which shed an interesting and balanced light on the topic. Not surprisingly, the issue of the complexity of the evaluation question at hand comes regularly into play.

Let's promote a new Gold Standard: the method most appropriate to build evidence should be determined by careful consideration of the question at hand, and equally important, who wants to know. And yes, it's harder than having a one response-fits all solution.

So, let's keep at it, and stand straight, even when we're asked to bend our knees. Fit the evaluation method to the program, not the other way around*.

Enjoy the readings, and if you have other suggested references, please use the comment box to add them.

DfID now recommends Stern et al (2012), Broadening the Range of Designs and Methods for Impact Evaluations in their solicitations for proposals.

The American Evaluation Association (AEA) has a recent special issue about Mixed Methods and Credibility of Evidence in Evaluation. It's a rich publication (which I'm still going through) deserving some serious attention. 

Finally, Michael Quinn Patton, former president of the AEA, offers a course based on his book Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. It's not per se about impact evaluation, but it's certainly relevant to the questions we face at CEDARS about sustainability evaluation. The book, and Michael himself, pack a punch.

Thanks,


Eric

* Thanks to Florence Nyangara for inspiring this entry.
[picture source can be found here]

Bringing Method to 'Transitioning' - Looking across Sectors

In the context of PEPFAR, we spend a lot of time talking about and working on transition. Here’s an example of a methodical approach to transition in another context and from another ‘shop’ in ICF, this time in the US.

The ICF whitepaper Effective Capacity-Building Technical Assistance Using Transition Charting describes the Transition Charting tool and provides a template for guiding transition discussions and identifying areas that may require additional intervention and support.

In order for initiatives to be successfully implemented, technical assistance providers need to carefully monitor services and manage transitions during the change process, identify areas of impact as they occur, and design appropriate mid-course corrections. Transition Charting is a tool that technical assistance providers can use to monitor and analyze three transition areas:

• Changes in Behavior
• Implementation of Selected Change Strategies
• Status and Functioning of the Initiative

 “Intensive technical assistance is particularly reactive to the context of the organization,” notes Sharon Harsh, our colleague from ICF’s Education Division and Director of the Appalachia Regional Comprehensive Center at ICF, “even well-developed multiyear initiatives can be blown off course by unanticipated consequences or emerging challenges that arise during the implementation of services.”

Have a look at the tool. It was interesting to me to discover this methodical approach to guiding a transition process—and to note that domestic and international concerns are not so different after all.

Eric

PS: Don’t hesitate to contact Sharon for more information on this tool (Sharon.Harsh@icfi.com)

And if you're curious, you can read more about ICF's work in Education in the Appalachia Region. 

Systems Approaches to Development Planning and Evaluation: Fad, Reinvention of the Wheel, or Necessity?

Everywhere you turn these days, there are discussions of ‘systems approaches’, ‘complexity’, ‘holism’ and ‘holistic approaches’.  And that’s not just in global health, that’s in development and sustainable development, that’s in food security and livelihoods, that’s in evaluation.

What is this all about? 

Are people just bored with inputs-outputs, and looking for a different flavor? 

Here’s how I figure we got to where we are, and I’ll attempt a conclusion on the fad vs. necessity debate... [Read more in this light-hearted discussion paper, which also provides a substantial number of interesting references (2,000 words)...]