TOR111 – Evidence Action with Karen Levy

Karen Levy

Listen Now

It is no secret that there is a very strong push from donors and policy makers that programming for development and humanitarian aid be “evidence based.” This push has, to a large degree, driven the technology revolutions we are seeing in data collection and analytics, and it is also responsible for the more difficult conversations that are had around value for money. So what if you were in a position where you were responsible for both testing and building a viable path to scale for promising evidence-based interventions? This dream job is currently held by Karen Levy, our guest here on the 111th episode of the Terms of Reference Podcast. Karen is the Director of Global Innovation at Evidence Action. She leads the organization’s innovation strategy and oversees the “Evidence Action Beta” department that looks for new ideas and tests them to see if they’ll work. You can connect with Karen here: https://twitter.com/kareninkenya https://ke.linkedin.com/in/kareninkenya

IN TOR 111 YOU’LL LEARN ABOUT:

  • How EA manages to bridge research outcomes into effective development initiatives at a large scale.
  • EA initiatives about water sanitation and deworming, how they grew from villages to multinational programs with millions of beneficiaries.
  • Karen’s skepticisms about silver bullets, overstuffed programs, and approaches that have simply no way of measurement. 
  • How evidence based development is still an open field, with no competition and many funders.
  • The risks of innovation, on saying “no” to big players, stopping promising innovation on its tracks (with good reason) and its emotional effects.
  • Why innovation in development is more boring than you think.

OUR CONVERSATION INCLUDES THE FOLLOWING:

Organizations

Topics

  • Evidence Based Development, Program Scaling
  • Silicon Valley
  • Embracing Failure
  • Programs with Millions or Tens of Millions of Beneficiaries
  • Water Sanitation, Dispensaries, Chlorination
  • Deworming, In Children
  • Poverty and Income Seasonality
  • Ongoing Innovation and Program Improvement
  • A/B Testing (Offline)
  • Evidence Venture Funding
  • Selectivity in Innovation, De-Risking

Places

  • Nairobi, Kenya
  • Malawi
  • Uganda
  • Bangladesh
  • Indonesia

EPISODE CRIB NOTES

Evidence Action (EA) A new organization that scales proven effective programs. Focus on poverty alleviation. Scope in the millions or tens of millions of people. There is enough research on things that work. Randomized evaluation, development economics. It is about turning interventions into standard ops with massive reach. 2 Flagship (“mature“) programs: Dispensaries for safe water, Deworm the World Initiative. These two will account for over 250 million beneficiaries. Karen leads Evidence Action Beta, to test the suitability of scaling programs with enough evidence. Is there an answer to poverty? “Oh gosh no.“ No silver bullets. More often than not, new programs face skepticism from the start. Even when the approaches are not innovative, the evidence collected does not support decision making. Only recently, organizations like IPA are in the business of rigorous evidence gathering. EA starts as soon as a proof of concept has evidence. The Dispensaries program uses a chlorine solution. While it does not solve all water sanitation problems, its net benefits against cholera are irrefutable, plus it stops the need for boiling and labor and environmental effects. This program received randomized testing and it is in expansion in Kenya, Uganda and Malawi. Ongoing improvement on scaling programs. And funding? “Donors are interested in getting value for money,” “but it’s not so simple.“ There is a gap between positive results from research and an actual successful roll out. “We live on that gap.” “We are happy to be the catalysts.“. Data and evidence helps improvement on ongoing programs. In the dispensaries program, data supported the launch of community leadership initiatives. An in-project mini trial was set up. Any pushback? Some of EA work builds upon that of other players. But the field is still open to think about competition, and there is no shortage of funding going into evidence-based initiatives. “It’s a healthy competition.” Lots of learning from one another. It is important to have room for experimentation and learning on the projects without lack of rigor. EA performs what is akin to “A/B Testing” on the ground, to fine-tune and focus. Small tweaks in local testing can mean additional million beneficiaries at full scale Deworming the world. Worms are such a staple of childhood in the developing world, it is considered a “symptom of childhood” in many places. But chronic infection can affect, among other things, school performance. Ongoing research revealed that deworming activities were more effective when done at school than at home. This insight was key for the following escalation in both impact and cost per child. EA β The β-Team “moves initially tested methods into the path of scale.“ Beta is borrowed from software product management. In EAβ, lots of questions about partnerships, emerging issues, business viability take place. It is the time to think about platforms, roles and responsibilities, standardization and maintaining quality. Keeping the spirit of the actions in line with the original methods for scaling is nontrivial. Regulatory considerations, governance, monitoring and ongoing evaluation systems. The work is vast. Not all proven successful approaches leave beta. Saying “No” to big players “It’s tough.” Karen has to say to a large donor, “we tried it, did not work, we are moving on.” The donor does not always takes this answer the best possible way. Educating on what is and is not a waste of money is key. “In Silicon Valley people are proud of their failures.“ As long as value for money and relationships are protected, there is a way forward for innovative partnerships. Honesty is fundamental, results must be shared especially when failure happens. But failure needs to be solved and can never become a systemic result. It’s all about constant confrontation. Programs with a large a number of components tend to be the most difficult to scale onto other geographic and political contexts. “Some projects will never be able to become large enough.“ Programs which might have good impact but simply cannot be rigorously evaluated are discarded as well. Niche, niche, niche, niche. Then add Karen feels that only after scaling the basic model is there room for additional components. A platform is the first step in scaling. EAβ undergoes a “No Lean Season” in which many initiatives around an issue are tested in sequence. This time the issue is rural poverty and income seasonality in Bangladesh. Hunger between planting and harvest is rampant, financial options are not there and additional incomes are limited and risky. EAβ tests solutions involving better solutions for enrollment in additional labor markets for the down farming season. It involves microfinancing, transport subsidies, food aid. Striving for sustainability is key, as is working with local partners. It is not about silver bullets, but about sound complements to established efforts. It has been a year, expanding into Indonesia. Getting capital when failure is OK “We work with donors interested in proven impact.” Risk is the mother of innovation. There is plenty of funding around maximizing impact, at several sizes. EA’s Deworm the World belongs to GiveWell Top Charities ranking. This has allowed support from venture funding, USAID. Coming up “In some ways the real innovations are really boring.“ Brand new things are not fine tuned for mass scale. “Business as usual can be an innovation.” Getting vaccines a worldwide staple would be an innovation. But Karen always pays close attention to new research and evidence based organizations. The new evidence they provide is a basic input for EA. Upcoming universal basic income study in Kenya, Karen is excited. Karen walls Selectivity is mandatory in this practice, it’s frustrating at times. The support she receives more than compensates. Some of the most promising innovations turn out to be the riskiest, and “de-risking” mitigation activities are not always enough for the program to move forward. “Always excited to keep trying.“ Karen’s favorite things David Evans @ The World Bank (blog on impact evaluation). Amanda Glassman @ Center for Global Development. IRD Pakistan on Incentives for Immunization. Use of RFID tags. Universal Basic Income.

Please share, participate and leave feedback below!

If you have any feedback you’d like to share for me or Karen, please leave your thoughts in the comment section below! I read all of them and will definitely take part in the conversation. If you have any questions you’d like to ask me directly, head on over to the Ask Stephen section. Don’t be shy! Every question is important and I answer every single one. And, if you truly enjoyed this episode and want to make sure others know about it, please share it now:
[feather_share show=”facebook, twitter, linkedin, google_plus” hide=”reddit, pinterest, tumblr, mail”]
Also, ratings and reviews on iTunes are very helpful. Please take a moment to leave an honest review for The TOR Podcast!

Share the Post:

You Might Also Like...