Tag: impact measurement

Should We Boil Social Impact Down to One Number?

~ Sasha ~ 3 Comments
Kenya clinic blood pressure
Image source: The Kenyan Experience in Combating Non-Communicable Diseases, 2013

No, we should not.

Let me tell you why, with a story about how we keep track of our health.

Last week I attended the Financial Times Investing for Good USA Conference in New York. If you haven’t noticed, the FT is taking seriously the shift towards using money as a force for good. Their Moral Money reporting is getting a lot of column space and resources—they even have a newsletter (part of a paid FT subscription) with lots of great content.

During Q&A at the social impact measurement panel at this conference, a woman in the audience, sounding exasperated, asked whether social impact is like health: is it something so nuanced and complex that we can never fully understand it in a simple, clear and comparable fashion?

The implication seemed to be that until we can boil social impact down to a single number, like IRR, we can never really understand it.

I love the health analogy, but I disagree wholeheartedly with the conclusion.

How Do We Measure Health?

Let’s think for a moment about how we measure health. While a person’s health is complex, there are some basic, universal measures that indicate well-being: blood pressure, resting heart rate, BMI, cholesterol levels, respiratory function, and so on.

Think about the characteristics of these measures: they are easy to gather, we collect them directly from patients, and they can easily be compared.

We gather this data annually in a physical (and gather a subset of them every time we visit the doctor), and doctors and nurses use these data to get an overall sense patient well-being. If these measures are way off, a patient might be unwell and in need of further testing.

Some Core Principles of an Effective Measurement System

Let’s think about the core principles that are in evidence here, because they give us good guidelines for how to think about social impact measurement:

  • Find measures that apply broadly
  • Determine what good and bad ranges look like
  • Regularly gather primary data to understand how individual patients are faring
  • When those indicators are off target, go deeper with specialized measures

It feels obvious that doctors have a core set of things they can measure to understand well-being. At the same time, we are not scared off by the complexity underneath. Indeed, we recognize that we must master that complexity to truly help patients: the human body is complex, so we must be comfortable with complexity to understand it fully.

And so, in patient care, simplicity and complexity happily coexist.

The Core, Comparable Metrics of Social Impact

Similarly, for social impact measurement, there are broad indicators that can be easily compared (many of which align with the Impact Management Project and for which we’ve developed questions and benchmarks at 60 Decibels):

  • WHO is being served: income levels, access levels, gender, members of excluded groups, etc.
  • WHAT is their experience of the product: Net Promoter Score, customer effort score, etc.
  • HOW MUCH does the product or service improve their lives: meaningfulness of impact, other indicators of changes in well-being (income, confidence, safety, empowerment, etc.)

All of these data can be easily gathered directly from the people experiencing (or not experiencing) social impact. And, just like blood pressure, gathering this data from the actual people being served is a prerequisite to understanding whether a specific product or service is making a difference.

From the People Being Served

“From the actual people being served” bears underlining: if my doctor wants to understand my health, she wouldn’t be satisfied knowing the BMI or blood pressure data of people like me. Instead, she would use population data to understand what the healthy range was and compare that range to what she reads on her dial when I’m standing right in front of her.

That might seem obvious, but in social impact measurement we seem too easily convinced that studying similar interventions is good enough—that we can simply extrapolate that data to our investment and be done. The fact is, as in medicine, studying other, similar interventions is the starting line, not the finish line. When I arm myself with that desk research, and then couple it with what I learn about the lived experience of the people my impact investment is serving, then (and only then) am I in a position to understand the impact performance of my investment.

Conversely, if we never listen to the customers being served by our investment, we’re saying the equivalent of (at my hypothetical doctor’s appointment), “typically, 46 year old white men have a blood pressure of 125 / 85.” That’s good to know, but it tells me nothing about whether I’m eating too much salt or at risk of heart disease.

Even more obvious, we would never expect that a blood pressure reading or BMI, alone, would tell us everything we need to know a person’s health. So why are we so obsessed with finding a single, one-number measure of social impact? These simplifying measures are, at best, directional indicators of a deeper reality that lies beneath. Importantly, that is not the same thing as saying that all that complexity must can and should boil up to that single number.

Finally, let’s not be frustrated that we can’t compare everything to everything. After all, we can only compare lungs to other lungs, not to livers or kidneys. Despite this limitation, we are not powerless to deduce whether one person is healthier than another.

Embracing Simplicity and Complexity

That’s good news.

It tells us that, in human health, we are comfortable with embracing both simplicity and complexity. We understand that the human body is itself a system with countless underlying organs and sub-systems. We recognize the need to understand these systems at a micro and a macro level. When we do so, we are in a position to successfully manage human health.

We can and should be just as comfortable with the notion that social impact happens as part of complex systems; and we should be optimistic that a core set of simple, easy-to-measure, comparable indicators can give us an enormous amount of insight about actual, on-ground social impact. Like in human health, we should also embrace the need to understand this deeper complexity if we are serious about managing social impact performance. That means deep, specific data about my specific intervention, coupled with cross-cutting, universal measures that apply to all interventions.

A Glimpse of the Future

This is all well within our grasp.

The first step is to stop telling ourselves that there’s some magical shortcut between here and there. Our work, and the people we aim to serve, are too important, and the amount of capital coming towards social impact is too big, for us to aim to skip steps.

Most important, let us never forget is that this work is about real, actual, living people. These people are the locus of change. It is a core part of our job to listen to them so that we can truly understand their perspective and their lived experience.

This is the only way we can manage social impact performance to achieve meaningful better outcomes.

And someday soon, doing all of this will be as normal and as natural as taking someone’s pulse or their blood pressure.

GIIN 2019 Impact Investor Survey

~ Sasha ~ Leave a comment

The Global Impact Investing Network (GIIN) has just published its survey of the impact investing market. Each year at this time, I head straight for a chart that’s been, until now, buried in the back.

It’s the chart that talks about impact performance for the estimated $502 billion of impact investing assets. In my view, it’s the most important chart in the report: since “impact” investing exists to create impact, we should care most about whether we’re pulling that off.

Unfortunately, data on impact performance is hard to come by in this report. The only chart that speaks to it directly, the one that I flip to immediately, doesn’t have performance data. Instead, it asks impact investors to self-report their performance relative to their own expectations. It’s a start.

Our Performance, Relative to Expectations

So, how do we think we’re doing? Pretty great, it turns out.

This year, 98% of the impact investors who responded to the survey said their impact performance was in line with or exceeded their expectations. Put another way, just 5 of the 266 impact investors surveyed were brave enough to say that they were under-performing on impact (or, maybe only five have clear enough impact goals and data to make it possible to under-perform).

How to make sense of this? Mathew Weatherly-White proposed on Twitter that perhaps the sector is exclusively doing place-based impact investing in Lake Wobegon (which would lead to the next question: would our version of Garrison Keillor’s famous closing line be, “Well, that’s the news from Impact Investing, where all the women are strong, all the men are good-looking, and all the funds’ impact performance is above average.”)

What happened in 2019 that makes us feel we’re doing so exceptionally well? Nothing much, it turns out, as this is not a new development. In fact, the numbers in this chart have been essentially unchanged over the last three years. Here’s a composite chart based on the data in the 2017 to 2019 GIIN reports:

GIIN 2019 impact performance

Perhaps this is our sector’s version of “too big to fail”–if you’re a self-styled impact investor, you cannot, by definition, fail at meeting expectations for impact. This isn’t for cynical reasons: most impact investors don’t yet have transparent, concrete targets around the impact their capital is meant to create; they don’t have benchmarks of impact performance; and they don’t feel they have a useful, repeatable way to measure that impact in a way that works for them and their investees.

Our Opportunity

While these results could be seen as discouraging, there’s an opportunity here as well. The GIIN, for one, describes their own rising expectations of impact investors in the opening of the report: “Growth [of dollars invested] without impact is pointless…[we believe] impact investors should have specific impact intentions; consider evidence and impact data in the design of their investment strategies; [and] manage their impact performance.”

I’d underline the phrase “manage their impact performance” and add to it “and set and share impact targets and performance for their funds.”

Setting targets, and managing to those targets, isn’t an end in itself. It’s a beginning.

The act of setting goals, and then taking them seriously, is a leverage point that is hiding in plain sight. It has the potential to jump-start a meaningful cycle of learning and improvement. We all know that there’s no way for performance to reach its full potential without knowing what excellence means, without having a bar to strive for. Nor can we improve without useful data—the kind of data that tells us both how we’re doing today and how much separates us from the best performers in our field.

How do we get from here to there?

With something as important as “impact”–the conditions of people’s lives, the fate of our ecosystems and the planet –we cannot miss our opportunity to become great at what we do.

Becoming great at anything feels daunting at the outset, but, as always, our only job is to start at the beginning: by taking one small step, and then taking the next one.

In this case, we starts by setting real targets, taking them seriously, and doing what we can to gather meaningful data about how we’re doing relative to our goals. If we do this with intention and follow through with integrity, then, bit by bit, we will get better. Once we choose to walk this path, we will discover that our small steps take us far: in a year, and then in five years, and then in 10, we’ll be at a different level in our capacity to invest to create positive change, just as we are experts, today, at deploying capital.

This need not be burdensome, heavy or expensive. The best way to start is by going directly to the source—for example, if you’re making investments designed to help people, then talk to those people. Better yet, do it in a way that is respectful, fast, and light touch, one that gives comparable performance results for impact, just like we have for financial results.

Let’s aim higher, not because we have to, but because we can.

 

[PS if you’re wondering what this looks like in practice, our recent 60 Decibels whitepaper might help].

Announcing the Launch of 60 Decibels

~ Sasha ~ 3 Comments

I have exciting news to share.

Today marks the start of a next chapter for me professionally: I’m launching a new social enterprise, called 60 Decibels, that I’ve co-founded with Tom Adams. Our goal is to reboot social impact measurement, to make it useful for people who are doing the work of building social businesses and NGOs. We want to help them serve customers better and, in so doing, create more social impact.

Our thesis is simple: understanding social impact should be based on listening directly to people.

60 Decibels will take forward the Lean Data approach, which was first built at Acumen to solve our own impact measurement challenge and has already been used by more than 200 non-profits and social businesses in 34 countries.

Imagine if we truly held ourselves accountable to the people that impact capital and philanthropy are meant to help, by systematically including their voices in how we assess impact.

(And, for those of you who don’t work in this sector, it’s worth articulating the counter-factual: yes, it’s true, today, when we ‘measure’ impact in impact investing, most of the time we don’t actually talk to the people whose lives we aim to improve. Crazy, huh?).

My belief is that if we can get this right, we have the potential to make a massive shift in the world.

Everywhere, the cracks in capitalism are being exposed. That’s leading to backlash against “plutocrats,” it’s creating waves of populism, and it’s generating calls, in some circles, for a new model of capitalism: one that creates wealth without being so extractive, one that balances the needs of shareholders, customers, suppliers, employees, and the planet.

But how are we going to put the needs of customers, suppliers, employees and the planet on more equal footing?

Our bet, with 60 Decibels, is that it starts with voice: that by listening better, and by amplifying voices that are currently left at the margins, we can create a system that’s more in balance.

The in’s and out’s of how I think we get from here to there is a longer conversation. (You can get a sneak peek here at the 60 Decibels website, where we’ve written a white paper that’s equal parts manifesto and social impact data). The short version is that 60 Decibels helps companies that are in the business of creating social change listen to their customers. We leverage the power of technology and mobile phones to make it easy to listen to anyone, anywhere, and hear from them about their lived experience. And we move fast, getting results in weeks (not months or years), because that’s the only way we’ll be relevant to the people doing the real work.

So, if you’re in the business of social change and have found social impact measurement to be challenging, burdensome, complex, or frustrating, let me know, maybe we can help.

And, if you’re wondering, 60 Decibels is the volume of human conversation.

So far, it’s been a lot of fun, it’s really challenging, and we’re just getting started. We have a team of 30 amazing people in the US, UK, Kenya and India and we’re working with customers all over the world.

And, in terms of this blog, I’ll still be here every week sharing what’s on my mind. I expect that, gradually, the content of the posts I write will shift slightly. That’s nothing new—it’s been happening since I started blogging in 2008, as my bullseye has moved from fundraising and sales, to generosity, to leadership and the work we all need to do to be grounded, effective agents of change.

A closing thought: in many ways, this blog is a chance for me to think out loud about the issues I find most important, most challenging and most meaningful. That exploration is an important part of my own evolution and growth. To the extent that I’m ready to take on this next challenge, that is due in no small part to what I’ve been able to figure out, week in and week out, through the dialogue that unfolds here on this blog.

None of that would be possible without you showing up and continuing to read and respond. So thank you.

Here’s to the next chapter. Thanks for continuing this journey with me.

Uncorrelated Impact Understanding

~ Sasha ~ 1 Comment

Not long ago, I was speaking to a group of sophisticated impact investors from across the spectrum: everything from fully liquid, market-beating financial return expectations to market builders focused on creating social impact who are open to a broader range of financial returns.

The focus of my talk was Acumen’s work on Lean Data, which is our industry-leading approach to gathering customer data at scale. We’re cracking the nut on using technology to give voice to tens of thousands of customers in ways that allow companies to serve them better. I believe that this will, over time, help the sector as a whole deploy more capital to more opportunities that have more social impact. It’s exciting.

But before digging in to the details of Lean Data, I started the talk with an assertion:

The seriousness with which you work to understand impact should be uncorrelated with your expectations around financial return.

I actually said this twice, because we’re so used to talking about correlations (positive or negative) between social impact and financial returns that I wanted to be very clear what I was, and was not, talking about.

My point is, if you say you are in the business of creating impact, then, irrespective of the instrument you use, the financial returns you expect, and the risk you’re willing to take, you’ve got to be serious about understanding impact.

Interestingly, I heard some resistance on this point. The resistance mostly took the form of “I know impact when I see it” or, “why would I waste time on this, it will just distract me from doing the real work?”

I believe there are some cases in which we really understand impact, but I believe those are the exception. Indeed we are so quick to say “we know enough” in a world in which we know shockingly little.

For example, take the $800 billion spent annually by the U.S. government. Peter Orszag, and Jim Nussle, who successively ran the U.S. Office of Management and Budget, write in Moneyball for Government that “Less than one dollar out of every $100 the federal government spends is backed by even the most basic evidence that money is being spent wisely.”

Less than $8 billion of the $800 billion spent annually by the U.S. government is backed “by even the most basic evidence?” Wow. Color me unpersuaded by the argument that we generally know enough.

I think what’s really going on is that we:

Overestimate how much we know

Overestimate the cost of getting great data – because approaches that came before Lean Data typically cost 100x as much

Create an artificial distinction between “creating customer value” and “creating social impact”

Assume that, no matter what anyone says, this is about marketing and dealing with funders, not about learning

Underestimate the value of what we can learn.

On top of this, I worry that we say too lightly that we’re in the business of creating social change, or we assume that this “caring about impact” stuff should be left to the folks who are on the frontiers of solving tough, challenging problems in innovative ways.

The truth is, we are quick to celebrate and advocate for more money walking through the “I (also) want to create social impact” door and then get awfully timid talking about whether that impact is getting created or, more broadly, how much we understand about the connection between the investment, the intervention and the impact it creates.

Caring about impact doesn’t mean you don’t understand how to make money. It doesn’t mean you’re not a serious investor. It doesn’t mean that you’re giving something up.

It’s simply saying: this is who I am, this is what I do. I’m in the business of creating massive positive change in the world. And I know how to do that better than anyone.

You can say all of those things and not blink for a second when someone asks you what your financial returns are going to be.

If we are in the business of change, then we have to be in the business of understanding how change happens.

Live stream today – the future of impact from SOCAP 2016

~ Sasha ~ Leave a comment

I’m excited to be speaking today at 2:30pm Pacific at SOCAP on a panel about the future of impact measurement. 

The panel is being live streamed in case you want to tune in: click here to tune in to the SOCAP live stream. 

The panel is with Jim Fruchterman (Benetech), Kelly McCarthy (GIIN), Paul DiLeo (Grassroots Capital) and Karim Harji (Purpose Capital) and it starts at 2:30pm Pacific. 

I hope you’ll join us!

Lean Data: Closing the Gap Between Entrepreneurs and their Customers

~ Sasha ~ Leave a comment

This post originally appeared on Acumen Ideas, our new channel on Medium.com.  If you’re interested in the nitty-gritty of impact measurement, be one of the first to sign up for Acumen’s new Impact Matters e-newsletter that will come out monthly.  You’ll get great content at the cutting edge of impact measurement, and we’ll also make sure let you know when our full piece on Lean Data comes out in Stanford Social Innovation Review this winter.

In 2006, Sam Goldman and Ned Tozun set out to eradicate kerosene as a source of light in the developing world. As a Peace Corps volunteer in Benin, Sam witnessed the damage kerosene could do when an overturned lantern created a fire that nearly killed his neighbor’s son. They also saw what a scourge the dirty fuel was for poor, rural families without access to energy, eating up 15 percent of their spending.

Sam and Ned decided to start d.light design, a social enterprise that would solve this problem once and for all. With funding from Acumen and others, d.light set out to create a business providing low-cost solar lanterns to poor customers. Since then, the company has sold tens of millions of solar-powered lights across more than 40 countries.

So is d.light a success? By one measure, absolutely. They are seeing demand for their product and on track to reach 100 million customers by 2020. That’s nearly 10 percent of the more than 1.3 billion people globally without access to electricity. But for entrepreneurs like Sam and Ned — and all of us at Acumen with a mission to make a real dent in poverty — just reaching a large number of people isn’t good enough.

At Acumen, we’ve spent the last 15 years investing in social enterprises that provide critical goods and services to the poor. We invest in these businesses because they are hard-wired to reach large numbers of people: when a social enterprise gets its model right, it will reach more people per dollar funded than traditional aid or philanthropy.

But while it makes us proud to say we’ve helped a million people acquire a reliable solar light or 10,000 women give birth in a high-quality, low-cost hospital, we need more than just big numbers to tell us if we are actually changing people’s lives.

How can we know if we are making a real difference?

Over the last 10 years, impact investing has attracted lots of attention and dollars. Thanks to the success of d.light and other ventures like it, today there are hundreds of impact investors putting their money behind companies that aim to deliver a social and financial return.

Despite this growth, impact investors have done a terrible job of analyzing whether or not these enterprises are creating meaningful social impact.

For example, in June, the Global Impact Investing Network and Cambridge Associates published the Impact Investing Benchmark, the first comprehensive analysis of the performance of impact investors. The report does an outstanding job of analyzing the financial results of impact investing funds, but it says virtually nothing about social performance. That’s a problem.

You’d assume impact investors must be good at measuring social impact. How else could we call ourselves “impact” investors? Not surprisingly, 95 percent of impact investors say they measure impact.  But, if you scratch the surface, you’ll discover their definition of impact is mostly limited to big, flashy numbers: number of farmers using an improved kind of seed, number of kids attending school or, as in the case of d.light, number of lights sold.

This is a start, but it’s not good enough. Typical impact investors may know how many farmers a company has reached, but they don’t have a clue if these farmers are better off. They may know how many kids attend schools, but they can’t tell you if the students are from low-income communities or just transplants from the private school down the street. They may know how many households bought a new solar lantern, but they don’t understand if the children in these homes are still dying from kerosene fires.

There’s a good reason impact investors have been falling short :  the existing tools for measuring social impact are nearly useless to a social entrepreneur.

These tools, mostly inherited from large-scale, international development organizations, are cumbersome, expensive and typically take a matter of months or even years to produce any real data. For a cash-strapped, resource-constrained social entrepreneur trying to build a fledgling business in tough, emerging markets, these tools don’t make sense.

The good news is, we have an opportunity to change this. Unlike five or 10 years ago, the majority of the 2.5 billion people living in poverty now have access to a cellphone and, in another five years, virtually everyone will be reachable by phone or SMS. At Acumen, we’ve developed a new approach to impact measurement that takes advantage of this shift. Our approach is optimized for entrepreneurs building social enterprises in the developing world, and it capitalizes on today’s information revolution to gather data directly from low-income customers. Our goal is to use this infrastructure to understand our social impact and better serve the poor. We call this approach Lean Data.

Unlike traditional impact measurement, Lean Data is designed to quickly and affordably generate quality customer insights that can immediately drive entrepreneurs’ decisions.

It reframes impact measurement as customer feedback by applying Lean Startup experimentation principles to the collection and use of social impact data. While Lean Startup aims to understand product-market fit with questions like “Do you like this product?” and “Will you buy this product?,” Lean Data goes a step further by working to understand how a purchased product is — or is not — changing a customer’s life.

By asking questions via mobile phones and other existing customer touchpoints (such as a salesperson’s visit to a customer’s home or a company’s call center), Lean Data allows enterprises to get social performance data in a matter of weeks and at a fraction of the cost of traditional measurement approaches.

In the last year, Acumen has helped 12 of our companies measure their social performance by surveying more than 5,000 customers across seven countries. Each of these projects took weeks, not months, and cost thousands, not hundreds of thousands, of dollars.

Here’s how.

Lean Data leverages technology, so enterprises can communicate directly with their customers. It is now possible to get reliable, meaningful data directly from low-income customers either through calls or SMS messages. For example, we worked with Ziqitza, a healthcare company that provides low-cost emergency services in India to understand what percentage of its customers in Orissa and Punjab live below the local poverty line. Our results showed that 75 percent of customers live on less than $2.50 a day. In another case, we worked with Juhudi Kilimo, a microfinance enterprise servicing smallholder farmers in Kenya, to measure its social performance using a 10-question SMS survey. The survey showed that the loans Juhudi Kilimo provided to purchase dairy cows are helping farmers see an increase in milk yields of 60 percent.

Lean Data puts the customer first, not the investor. As an investor in social enterprises, Acumen needs impact data to manage its own performance. But we believe social enterprises should first and foremost be accountable “downward” to their customers before worrying about “upward” accountability to their funders. Social enterprises set out to solve meaningful problems for their customers, and they should only systematically collect impact data if that information helps them understand how their products or services are making a difference in their customers’ lives. The information should also be shared “upward” with funders, but that cannot be the primary reason for collecting data.

Lean Data gets underneath not just the “what” but also the “why” of product-market fit. Lean Startup principles focus on product-market fit: is there a demand for a new product in a given customer set? How satisfied are customers with the new product? Social enterprises can take this a step further, asking not just whether there is product-market fit, but why that fit exists. This is the first step towards understanding impact. When we discover why products are purchased, how well or often they are being used, and which problems they solve or fail to solve — like improved productivity, increases in household savings or fewer sick days — we empower customers to articulate what impact means to them. This kind of insight is invaluable to entrepreneurs looking to drive lifetime value, customer loyalty and social impact.

We’ve been developing Lean Data for a little more than a year and, while it is still in its early days, we see huge promise.

If we can give more entrepreneurs like Sam and Ned the right tools to understand their social impact and hear from their customers, they will, for the first time, have actionable data that can tell them, in real time, how to improve their products and create meaningful change.

The truth of this work is that the big, glossy numbers allow us to sing our own praises and raise more money, but they do little to help us improve the lives of the people we aim to serve. It’s time to dig deeper, to use technology to talk directly to our customers, so that our work can realize its full potential.