How do you know if your programs actually result in the impact you want? Alan Mackie has over 25 years’ experience evaluating impact. He joins us today with a simple message: you don’t have to be a statistician; you just have to know what’s important to you and your clients. Listen in and get motivated to go get your data!
Listen to the Episode Here!
Links
Website: Get the Data
Get the Data’s LinkedIn
Get the Data’s Twitter
Article: Cantril Scale
Basic Empathy Scale (adolescents)
Podcast Episode 3: Measuring Outcomes with Khurram Hassan
Timestamps
(01:59) A Case Study: Gideon’s Promise
(11:12) Low stress data collection
(24:15) Validated measurement scales
Transcript
Dolph Goldenburg (0s):
Welcome to the Successful Nonprofits® Podcast. I’m your host, Dolph Goldenberg. Today, we are going to be talking about measuring your impact with Alan Mackie. Before we jump into that conversation, if you’re thinking about your board and about how you measure your board’s impact in 2021, please reach out to me. The first quarter of the year is always a good time for us to be thinking about board development and board retreats. So reach out if you’re thinking about how your board can get stronger in 2021.
Dolph Goldenburg (42s):
Today, we are talking about measuring your organization’s impact with Alan Mackie. Alan is an attorney who started off evaluating the impact of legislation on criminal justice in Britain’s Home Office, which is like our Department of Justice here in the United States. He was there about a dozen years before rolling into the private sector where he continued to help measure impact in areas of criminal justice. He then expanded to education, healthcare, and other public service, public sector, and nonprofit organizations.
Dolph Goldenburg (1m 27s):
One of the things I have really admired as I’ve been following his blog is the way that he breaks things down and makes what seems really complex, pretty simple. And that is a gift and a talent. And it’s one of the many reasons why I wanted to make sure that we had Alan on the podcast. Hey Alan, welcome to the podcast.
Alan Mackie (1m 53s):
Hi Dolph. It’s great to be here. I’m looking forward to our conversation.
Dolph Goldenburg (1m 59s):
Same here. I know there’s a case study that you’ve been talking about recently about Gideon’s Promise. Can you share about that?
Alan Mackie (2m 9s):
Certainly. I’m delighted to. Gideon’s Promise is a nonprofit based here in Atlanta, Georgia, and it was founded by Jonathan Rapping and his partner Ilham Askia. They train and support public defenders to put the client at the center of the public defender practice. They’re an awesome organization. They work across the South and Midwest of the United States. Their goal is to make attorneys better advocates on behalf of their clients. They really want to transform the criminal justice system here in the States.
Dolph Goldenburg (2m 51s):
As I read your case study on that, I think Gideon’s Promise had an unusual type of outcome they wanted to track – not how many are convicted and how many are not convicted like you might expect.
Alan Mackie (3m 7s):
Right. I remember my first meeting with them. A colleague of mine and I turned up and were trying to understand the impact they wanted to make. And I kept saying to them, “Is it a less severe sentence? Is it an increase in acquittals?” And Jonathan Rapping got frustrated and said, “If you don’t get it, you can leave the meeting now.” And I said, “Hang on. Be patient, I will get it.” What they wanted to measure was the client experience and that the client felt that they had been put at the center of their defender’s attention. Was their story being told? Was their case being investigated? The concern of Gideon’s Promise is that often public defenders take a case and agree to a plea and the client’s not really involved. They’re often the most vulnerable people in our society – without means, without a voice, and often marginalized. So this was very important. What we understood working with John and colleagues was thinking about what was important to public defenders and what they were trying to achieve. That was a very good starting point for any nonprofit – What are you trying to achieve? What’s important to you? – even if you think that’s difficult to measure.
Alan Mackie (4m 32s):
If we begin to understand what you’re trying to do, we can come up with the measures. So, we devised two measurements for Gideon’s Promise. One was called the defender values spectrum survey. It’s a questionnaire that we use to test with different public defenders who’ve been trained and supported by Gideon’s Promise. It evaluates the extent to which they feel they’re putting clients at the center of their practice. We’ll ask them questions about how well they think they advocate on behalf of the client, how well they investigate, and how well they’ve been able to take their client’s instructions and put their client at the center.
Alan Mackie (5m 13s):
We also have a client experience survey. And this isn’t a customer satisfaction survey, it’s a little more detailed. But we try to understand from the client’s point of view if they really felt that their story had been heard no matter what the outcome. So often the system’s rigged against these defendants. They come in and there’s a presumption of guilt when they walk in the court door. They know that’s the case, but even if they’re being sent to prison, even if they’re being found convicted, do they feel that someone had stood up for them in court? And I think that’s very important. Gideon’s Promise feels that’s very important.
Alan Mackie (5m 54s):
We find that the clients also feel that that’s very important. This was a really exciting project to be involved in. It’s still going on. We did the feasibility study. We tested our instruments and we’re ready to roll this out when the funding comes. For me, it was transformative because it got me away from being an attorney working in the British government. So often everyone measured success by if we got this guy off or he didn’t get a jail sentence, but this was something different and more important.
Dolph Goldenburg (6m 37s):
Part of what I love about Gideon’s Promise’s decision to focus on how the defendant felt about their defender over the outcome is what you said: the system is so stacked against people who’ve been accused of crimes. But also, public defenders have such a larger caseload than the prosecutors that are working for the DA’s office. And so consequently, I could totally see how a lot of public defenders start to think, “Okay. I’ve got 45 minutes with you. Let’s get this done. You’re in and you’re out.” That has a profound impact on the client. So I love the fact that they were looking at this non-traditional outcome to be evaluated.
Alan Mackie (7m 31s):
Yeah. Looking back at the early part of my career when I was an attorney and representing people in court, I was faced with the same temptations to get things moving and off my plate. It’s a very tempting thing. The stresses are there. You’ve got an overwhelming caseload. And so the temptation just to process it is very strong. I think the other aspect of what we looked at in our questionnaire with the public defenders was the empathy. We obviously didn’t want cold hearted people who didn’t care.
Alan Mackie (8m 12s):
We wanted to measure the public defender’s empathy was affective empathy (So if a defender would take all the worries of the client upon them and not establish those boundaries, that could lead to burning out – especially with an overwhelming caseload.) or more directive or cognitive empathy (where you were establishing appropriate boundaries and direct them to the next thing). However terrible the circumstances of your client in front of you might be, the best thing you can do is have that ability to direct them onto the next thing.
Dolph Goldenburg (8m 60s):
This may sound like an ignorant question, but how did you develop a tool to determine empathy among public defenders?
Alan Mackie (9m 11s):
There’s a basic empathy score, which has been validated and is widely used. We worked with an academic in England, Professor Darrick Jolliffe. He had helped devise this score for criminal justice personnel. It was the third part of our public defender questionnaire. The first and second parts we built from understanding what Gideon’s Promise wanted to do. We started by building a logic model. This helped us understand what we call our theory of change. If you get these public defenders, you train them in a client centered approach, and these are the outcomes, how does that all fit together as a series of logical inferences? This is something we would do with any client.
Alan Mackie (10m 1s):
Understanding the inputs – the training, the resources, the support you’re putting in – and then the intended outcomes. Once we had done that, we could identify the areas that Gideon’s Promise wanted to impact on their public defender’s practice. Then, we developed a suite of questions that really unpack that. Once we had done that, we went to the public defenders and tested it on them to make sure that made sense. We collected a sample and then we did something that we call validity and reliability testing to ensure that we were measuring what we wanted to measure and that the questionnaires could be used reliably.
Alan Mackie (10m 50s):
Rather like a thermometer that’s properly calibrated, it will always measure freezing point or boiling point. So we did that. We’re confident that the tools that we developed are valid, reliable, and will withstand peer review or scrutiny by any skeptics.
Dolph Goldenburg (11m 12s):
It’s awesome. A lot of organizations are hesitant to start outcome evaluation projects because they’re worried about the data collection piece. They say to themselves, “Our case managers or healthcare providers or whoever is collecting the data are already so busy. They’re already feeling overwhelmed. We cannot ask them to collect more data.” So how can nonprofit organizations collect that data in a way that allows people to do it without ripping their clothes, gnashing their teeth, and cursing under their breath?
Alan Mackie (11m 47s):
Well, I’ll let you into a little secret Dolph. I’m the least numerate person in the room. I always tell people, “Data is important, but you don’t need to be a statistician to handle data.” There’s two issues here. One is a fear of data. In our society, people are anxious about their numeracy skills. People are anxious about spreadsheets and they become overwhelmed. That’s why I think developing a theory of change or a logic model is very important. Theory of change sounds rather grand, but it’s just the story of how you want to affect the change you desire.
Alan Mackie (12m 29s):
And that’s simply storytelling. And I love doing that piece of the work. I love reading about an organization, interviewing its staff, its board members, its beneficiaries, and really understanding what it is that you want to do and what resources you have to achieve it. That’s the theory of change.
Dolph Goldenburg (12m 49s):
Can you tell us one of those stories that is a theory of change for something you’ve worked on?
Alan Mackie (12m 54s):
Yeah. Let’s go back to Gideon’s Promise. Their theory of change was that they want to transform the criminal justice system. It goes beyond the public defenders, but they see the public defenders as being an agent of change. When you have robust public defenders, you’re going to start to see a change in culture in the court. They’re going to start putting the district attorney on notice that they’re going to be a bit more robust and they’re not going to be a pushover. They are going to show that the people they represent are not just men and women in orange jumpsuits. They’re humans. They’re mothers and brothers, sons and daughters, and valuable members of the community.
Alan Mackie (13m 44s):
And from that, you begin to change how the public perceive criminals. Most criminals do fairly silly, but ordinary things. They’re not the superpredators that we’re taught that they are. And so we begin to change society and we begin to change what society expects. That’s what we heard from John Rapping and his colleagues. So let’s unpack this. We will probably never be able to measure the impact on society of Gideon’s Promise.
Alan Mackie (14m 29s):
But the bit that we can manage and where we can affect change is on the public defenders. So how do we do that? The resources you have are this wonderful organization Gideon’s Promise. It provides training. It provides mentoring. It provides support. So although you’ve got this really ambitious mission to change society, the part where you can affect the change is really: what are your resources? Who’s coming to your training? What training are they receiving? And what impact are you having?
Dolph Goldenburg (15m 23s):
These public defenders who already have a much higher case load are now being asked to complete these scales of empathy, et cetera. How do you convince them that it’s worth their time to do it?
Alan Mackie (15m 38s):
We do a census point. So someone who has recently gotten through the bar exam, has been employed as a public defender, and admitted into Gideon’s Promise does a baseline form before they join their three-year training course. And then every year we ask them to complete these forms. That is how we begin to trace them. Everyone else who’s done the training, or is in any position within the public defender organization, will also be an annual census.
Alan Mackie (16m 20s):
Convincing people to do that is a value thing. If they really feel committed to changing their practice and putting the client at the center, it’s the least that we can ask them. It’s 15 minutes to complete the form at most and we only do it once a year. With the clients, asking them to complete the form was difficult. And that remains a challenge. And there were practical challenges. We’ve tried it different ways. Do we do it at court? That’s a very busy time for them. There’s so much happening.
Alan Mackie (17m 0s):
Do we do it after the event? So many of them can be transient and move on. They might have a burn phone. Trying to locate them was challenging, but we were really committed. We felt that it was integral to make every attempt to reach out to the people who were benefiting from the client-centered approach. So we aim for a large sample. We probably get less than we had hoped, but it’s a suitable sample in which we can begin to look at the effect of the client-centered approach on these people. I often think that people can put out obstacles. They can say, “Is it unethical? We don’t want to bother people.” But I think you’ve got to look at what’s important to you. And we found that when we were testing instruments with individuals, they wanted to tell their story again. They wanted to say, “This public defender really put me at the center of the practice or they didn’t.”. I think often when we worry about the time or the ethics of at all, we’re often excluding people again and we’re deciding something’s not important to them.
Dolph Goldenburg (18m 15s):
One of the things I think I hear you saying is you helped develop an evaluation tool that feeds into Gideon Promise’s theory of change because it helps the client feel valued again.
Alan Mackie (18m 32s):
Yes. That is it. The client is central to everything that Gideon’s Promise thinks is important. How could you be concerned about how the client felt in court, but never ask them? Is what we think good for you actually good for you? Are these things that we think important about client centered practice really important to you?
Dolph Goldenburg (18m 56s):
It’s like those strategic planning projects where you never talk to a client and find out what they think about it.
Alan Mackie (19m 1s):
Exactly. And I think that this goes beyond Gideon’s Promise and criminal justice systems. So you might be in a housing nonprofit, a healthcare nonprofit, or an education nonprofit. You think the people you’re trying to help are so important. So how can we not only give a voice to those people, but measure what is important to them? Not particularly what’s important to your board or to your funder. Measure what’s important to the people you’re trying to help.
Dolph Goldenburg (19m 44s):
So that is a great segue into how you make the complex simple and how you communicate it – especially if what you have is data that your board or your funders aren’t really that interested in.
Alan Mackie (19m 58s):
In so much of my practice for the government, we would have great data collection exercises. We would do wonderful analysis. We would evaluate a government program from top to bottom and inside out. But we’d hear nothing back. It was like crickets. Looking back at my 25 years, I wonder what difference I made doing government research. What Gideon’s Promise really taught me and what working increasingly with nonprofits is teaching me is that nonprofits know what’s important.
Alan Mackie (20m 44s):
Going back to what we’re talking about, you articulate your theory of change, you tell me the impact you want to make, and you tell me the resources that you have at your disposal and how you’re going to bring about your desired change. I’ll scratch my head and I’ll think, “We’ve got to do some monitoring. We’ve got to make sure that your resources are being applied in the right way. If you’re training people or if you’re running a clinic, we need to make sure that that’s well-resourced and that the people who provide the resources are arranged in the right way.”
Alan Mackie (21m 28s):
And that’s a good bit of housekeeping. And generally the data is there because most nonprofits will have some management data. We’ll use that. Then we’ll scratch our heads again and say, “What is it that’s important to you now?” It might be that you have a number of things that are important. It might be that you want to understand how your clients think about the service, but it also might be about the difference that you’re making or how many people see improvements in their health or wellbeing?
Alan Mackie (22m 10s):
And often there’ll be data that we can use. We call those secondary data. If you’re looking at wellbeing, there are some well-established wellbeing tools. There’s lots of ways we can do it, but it’s really understanding what’s important to you. I have another client, the International Rescue Committee, who was completing a four year evaluation. I remember asking them, “What’s your outcome?” And they said, “It’s getting people into employment in the chicken factory in Gainesville.” And I said, “That doesn’t sound like the best employment, but it’s a job. You’re a humanitarian organization. Do you measure how your clients feel about themselves? They’ve come from a place of potential harm and they’re here in Metro Atlanta in a place of relative safety, but they had hopes and expectations in Africa or Asia or wherever they came from. You’ve got them a job, but that’s not the end of the story.”
Alan Mackie (23m 34s):
In our evaluation of one of the resettlement programs, we implemented the Cantril scale. And that’s a simple well-being tool, which measures whether people are thriving, struggling or suffering. And it’s a very easy tool to use. The IRC said, “Yeah, that’s really important. It’s a job, an income, but we need to understand if someone feels there’s something beyond the chicken factory. Is there a way in which they feel they could begin to thrive in their new home or that they could maybe pick up what they had hoped to do when they were back in Africa?”
Dolph Goldenburg (24m 15s):
You had mentioned at this point, a few validated scales that organizations use. How does an organization find those scales if they want to measure empathy or thriving or anything else?
Alan Mackie (24m 30s):
One of the very simple ones, the Cantril scale, is a very simple ladder scale, 10 steps of the ladder. You ask someone to pinpoint on the ladder how they’re feeling. Where they’ve identified themselves determines whether they’re suffering, struggling or thriving. So it’s very simple. The empathy scale is widely available. You can use that. There are other ones that we would search out. I’m not saying that you would need professional support to find them. But again, it’s all about understanding what a wellbeing tool is trying to measure and making sure there’s a good fit.
Alan Mackie (25m 15s):
Do not start applying them without understanding how they were created, what they’re intending to measure and how they can be used properly. But I think the Cantril scale for an organization is appropriate. It’s great for people. There’s very little language barriers to overcome. It’s very straightforward.
Dolph Goldenburg (25m 46s):
That’s awesome. We will make sure these are available in the show notes. Alan, we are rapidly approaching 30 minutes and I got to ask you an off-the-map question. A lot of our listeners were in their high school band or marching band. I understand that you played an instrument that probably most of our listeners have never played.
Alan Mackie (26m 21s):
Yes – the great highland bagpipe. I went to a school in Edinburgh many, many years ago. The principal wanted to have a pipe band. And now 40 years later, the pipe band that they have is a renowned pipe band. But I was one of four boys who were selected to learn the bagpipes. It’s a great party piece to have. If I have my bagpipes, I get invited to weddings to play. Stateside, I do have my little practice instrument and I practice my scales and my tunes and it’s great.
Dolph Goldenburg (27m 28s):
That’s really interesting. And you mentioned that you were selected. Does that mean that you competed and were selected or does that mean someone said, “You! You’re going to play the bagpipes!”
Alan Mackie (27m 39s):
It was the latter. Yes. Thinking back, I played in the equivalent to the Scouts in Scotland and they had a little band. Someone heard that I was learning. The occasion was the queen was coming to visit our school. The principal wanted her to receive a great Scottish welcome. So we stood at the gate and we played.
Dolph Goldenburg (28m 9s):
So wait a minute. This means you played for Queen Elizabeth?
Alan Mackie (28m 13s):
I did. Yes. And we were told she is quite a discerning listener of the bagpipes. I believe every morning she’s woken up by a piper. So she knows her stuff. We had to be good.
Dolph Goldenburg (28m 32s):
I have to ask you another question now because I did not know about this. She’s woken up by a piper every morning? Does that mean the person is outside below her window or is the person in her room?
Alan Mackie (28m 42s):
When I was living in London, I knew her personal piper. He used to have to get up quite early in the morning and he would walk up and down the garden in Buckingham Palace and play a tune. It sounds really silly, doesn’t it? I don’t know, is the president woken up by bagpipe?
Dolph Goldenburg (29m 7s):
I have so many questions, but I have to ask this one. Is that a full-time job for your friend being the palace bagpiper?
Alan Mackie (29m 15s):
He was an army piper. So he piped in the army and when he was stationed in London. He would go from the barracks early in the morning and play before breakfast.
Dolph Goldenburg (29m 27s):
I was thinking what a cushy job, your day is over by 6:45am, but if he’s in the army, his day was not over at 6:45am. Alright. Well, Alan, thank you so much for joining us and sharing so much about evaluation. And also, frankly, about the bagpipes. I learned things I did not know about the bagpipes.
Dolph Goldenburg (29m 56s):
Listeners, if you want to reach out to Alan his URL is getthedata.co.uk. His firm works on both sides of the Atlantic so don’t worry about the fact that it’s a .uk URL. In addition to reaching out to him, there are two other things I want to make sure you check out. The first is check out the blog. They also have webinars every quarter. So make sure you click on the webinar link and you can register for an upcoming webinar or even watch a prior webinar. So, Alan, thank you again for coming on the podcast.
Alan Mackie (31m 16s):
It’s been a great pleasure.
Dolph Goldenburg (31m 24s):
Listeners, if you are Googling the palace bagpiper and so we’re not able to write down Alan’s URL that’s okay. Go to successfulnonprofits.com. We’re going to have the URL for his firm and we are also going to link to his LinkedIn page and his Twitter feed. Don’t forget listeners, if you’re thinking about some board development work, go over to successfulnonprofits.com and reach out to me. We would love to partner with you on strengthening your board.
Dolph Goldenburg (31m 50s):
You all know, I love to read reviews, the good, the bad and in between. I love the reviews of the podcast and we’ve had a few over the last month. I wanted to read one from Josh. He said, “Whether you’re well-established as someone who can translate creative energy and the impact you want to have on the world, or getting started as a catalyst for change. This is a must listen podcast for you. Highly recommend listening and subscribing.” Josh, thank you so much for the review. I am grateful. You are helping other people find our podcast and get this resource. So listeners, please pick up your phone and write a review. I appreciate it.
Now, if you enjoyed this conversation with Alan Mackie, you should go back and listen to Episode 3, Measuring Outcomes with Khurram Hassan. Quick note though, episode three is not available on the stream. We’ve removed about 50 or 60 of the first episodes because the audio quality is no longer up to what we find acceptable. But you can go to successfulnonprofits.com and still play that episode. That listeners, is our show for this week. I hope you have gained some insight to help your nonprofit thrive in a more competitive environment.
Dolph Goldenburg (33m 20s):
And a quick reminder that I am not an accountant or attorney and neither I nor the Goldenburg Group provide tax legal or accounting advice. This material has been provided for informational purposes only and is not intended to provide and should not be relied on for tax, legal or accounting advice. Always consult a qualified, licensed professional about such matters.
** We have edited this transcript because how you listen is not how you read. If you have a problem with this, remember you got this for free!