Thursday, July 19, 2018

The sustainment index - a new metric to inform ex post sustainability evaluations.

Greetings -

Reeti Hobson and I have recently published a paper in BMC Health Services Research proposing a measure of sustainment -- the sustainment index.


I'm an unlikely candidate for jumping on one quantitative indicator to solve complex questions, but this is -- as the title of the paper states -- a 'simple metric for a complex outcome'. It is free of causal inferences; it is simply a measure of how much an indicator improved from Time 0 to Time 1 continued to progress from Time 1 to Time 2 (presumably after a transition phase / end of a project). 

We call it the 'sustainment index' and the paper explains this choice of language. 

We've already made a strong case that empiricism about sustainability needs to move from 'is it sustainable?' -- binary, either reductive or existential -- to two types of questions:
  1. Looking forward: what are the chances that progress will continue under new conditions (sustainability)?
  2. Looking back: how much of initial progress has continued after transition and can we start to understand why and how (sustainment)?
The sustainment index is simply here to provide a reliable and valid measure to this "how much" question.

The story of its development links back to an unsuccessful attempt at an unsolicited proposal, where the concept of this metric was developed. It's really put out as a invitation to researchers to start testing the metric. If you've already collected baseline, endline, and post project service or outcomes' data, it's extremely easy to plot out the sustainment index. If you're designing an ex post study, consider its value as a dependent variable.

An interesting property of the index, which is not discussed in the paper, is that while the performance of a health indicator is affected by the initial conditions, baseline (i.e. improvements are harder to achieve when the baseline is higher), the sustainment index--since it tracks the evolution of the speed of progress -- does not seem affected in the same manner. The graph below (not in the paper) shows 3 trajectories from 3 different baseline points of a health indicator, with the same sustainment index.

If we've made some serious researchers out there curious... then our job is done.

All feedback - and study reports -  welcome.



Eric [esarriot AT savechildren DOT org]
Thanks to my former ICF/CEDARS home for getting this through completion, including Reeti Hobson, my co-author, and Jennifer Yourkavitch. Thanks to our reviewers.