10a. Assessing the Impact of Population Health Interventions in Real-World Settings

This week’s module focuses on the evaluation of population health interventions as well as strategies for disseminating (or sharing) evaluation results with key stakeholders.

Evaluation is a broad, complex topic that cannot be fully addressed within the confines of this course. If you wish to learn more about designing and implementing evaluations for population-level interventions, I encourage you to enroll in PHS 614: Foundations of Program Evaluation.

When the “Gold Standard” is Not Possible: Alternatives to Randomized Control Trials for the Evaluation of Population Health Interventions

Let’s start with a brief recap of some content presented in last week’s module. You may recall that randomized control trials (RCTs), where individuals are randomly assigned to groups that receive a new treatment/intervention, an existing treatment/intervention, or a placebo/control, are viewed as the “gold standard” of health-related research. But, in practice, the use of RCTs to evaluate population health interventions is seldom feasible for the following reasons: 1,2  

  1. the difficulty of conducting RCTs for complex programmatic population health interventions, many of which require more flexible, community-driven approaches; it’s seldom feasible to randomly assign neighbourhoods or communities as units of analysis;
  2. the difficulty of conducting RCTs for policy research (i.e., randomly assigning jurisdictions to be the recipients or non-recipients of population-level policies);
  3. the difficulty of interpreting results, especially negative findings; and
  4. the tendency to downgrade the contribution of non-RCTs, including observational studies, which are often the only practical designs for assessing the impact of community-based public health interventions.

So how do public health practitioners assess the impact of population health interventions in real-world settings? The following provides a brief summary of the key alternatives to RCTs. Please consult your course text and required readings for a more detailed description of these evaluation designs.

Time Series Designs

 
Time series design
 

These designs entail repeated observations of the same variable or outcome over time. The interrupted time series design is especially useful in situations where a population-level communication campaign, program, or policy is introduced at a specific time.3 The repeated measures allow for the comparison of outcomes of interest before, during, and after the intervention. For example, an interrupted time series design could be used to monitor trends in the purchasing of sugar-sweetened beverages (SSBs) following the introduction of an SSB tax, or the self-rated health of low-income individuals could be assessed before and after the introduction of a universal basic income.

Comparative Effectiveness Designs

Population health practitioners sometimes want to see if a new population health intervention is more successful than a standard program or practice that has been in place for some time. In this case, the control or comparison group is not composed of individuals/communities without the intervention; rather, it is composed of the individuals/communities receiving the existing intervention.4,5

Say, for example, a company wanted to evaluate a new worksite wellness program. A comparative evaluation design would compare key outcomes, such as employee morale, number of sick days, reported changes in health-related behaviours among employees, at a “pilot” site (a corporate office where the new program was implemented) with the other company offices where the current worksite wellness program is in place.

Mixed Methods Designs

Mixed methods refers to the joint use of quantitative and qualitative methods in an evaluation design.6,7 The key rationale for the use of mixed methods is to avoid limitations to validity and reliability arising from the use of a single method with a complementary combination of methods. Mixed methods evaluations have the advantages of both corroboration (better understanding, greater credibility) and triangulation (confirmation of the same patterns and insights through multiple sources).8

 
Qualitative plus quantitative with intervention in between
 

A simple axiom for the use of mixed method is that quantitative measures tell you what happened (or may have happened) as a result of a population health intervention, but they don’t tell you how or why. Say, for example, you were evaluating a province-wide program to increase parenting skills. Pre-post tests given to parents might reveal a significant increase in positive outcomes, such as the amount of time parents spent playing with their children. By adding qualitative interviews with parents, you could gain valuable insight into the key factors or mechanisms that may have contributed to this outcome as well as key barriers to positive parent-child interactions that could be addressed through program modifications or other interventions.

Pragmatic Evaluation Designs

This broad category of evaluation designs refers to instances where elements of classical experimental evaluations, such as RCTs, are adjusted or “tweaked” to adjust to real-world conditions. Pragmatic designs may share key features of RCTs, but key dimensions of RCT design (e.g., allocation to intervention or non-intervention (a.k.a. control) groups) are adjusted in order to generate evidence that is more relevant to practice.9

Here are some examples of ways in which evaluation designs may be adjusted in response to pragmatic concerns, such as real-world applicability:10

  • individuals and communities are selected to better resemble the population at risk rather than being filtered through the narrow selection criteria typical of RCTs;
  • the intervention is implemented without a strenuous effort to standardize it, thereby allowing for greater flexibility in response to unique community conditions/needs;
  • comparison or control groups may be receiving other interventions designed to achieve the same outcome (e.g., reduced injuries resulting from motor vehicle collisions);
  • the outcomes studied are the outcomes of greatest importance to decision makers (as opposed to outcomes that are easiest to measure through RCTs).

References

  1. Goodstadt, M.S., Hyndman, B., McQueen, D.V., Potvin, L., Rootman, I., & Springett, J. (2001). Evaluation in health promotion: Synthesis and recommendations. In Rootman, I., Goodstadt, M.S., Hyndman, B., McQueen, D. V., Potvin, L. et al. (Eds.), Evaluation in Health Promotion: Principles and Perspectives (pp. 517–534). Copenhagen: World Health Organisation.
  2. Sanson-Fisher, R.W., Bonevski, B., Green, L.W., & D’Este, C. (2007). Limitations of the randomized controlled trial in evaluating population-based health interventions. American Journal of Preventive Medicine, 33(2), 155–161.
  3. Shadish, W., Cook, T., & Campbell, D. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin.
  4. Slutsky, J. R., & Clancy, C. M. (2010). Patient-centered comparative effectiveness research: Essential for high-quality care. Archives of internal medicine, 170(5), 403–404.
  5. Piaggio, G., Elbourne, D. R., Pocock, S. J., Evans, S. J., Altman, D. G., & Consort Group. (2012). Reporting of noninferiority and equivalence randomized trials: Extension of the CONSORT 2010 statement. Journal of the American Medical Association, 308(24), 2594–2604.
  6. Creswell, JW, & Plano Clark, VL (2011). Designing and conducting mixed methods research. (2nd ed.). Los Angeles, CA: Sage.
  7. Bamberger, M. (2012). Introduction to mixed methods in impact evaluation. Impact evaluation guidance notes, No. 3. Washington DC: InterAction and the Rockefeller Foundation.
  8. Bartholomew Eldredge, R.K., Markham, C.M., Ruitter, R.A.C., Fernández, M., Kok, G., & Parcel, G. (2016). Planning Health Promotion Programs: An Intervention Mapping Approach (4th ed.). San Francisco: Jossey-Boss.
  9. Sidani, S. (2015). Health intervention research: Understanding research design and methods. Los Angeles, CA: Sage.
  10. Zwarenstein, M., Treweek, S., Gagnier, J.J., Altman, D.G., Tunis, S., Haynes, B., ... & Moher, D. (2008). Improving the reporting of pragmatic trials: an extension of the CONSORT statement. BMJ, 337, a2390.