“Research kills opinion” is the mantra of today’s guest, Rachel Clemens, Chief Marketing Officer of Mighty Citizen. Rachel develops and analyzes donor surveys that uncover the truth about what donors are thinking and feeling about an organization. Today she shares best practices about design, implementation, and follow-up.

*****Timestamped Highlights*****

(5:42) The nuts and bolts of a donor survey
(9:24) To ask or not to ask; that is the question
(11:35) Surprise, surprise!
(12:55) Always offer the “Other”

(14:20) #1 Boring #2 Dull
(17:09) Open-ended questions are great in personal communication, not so much in a donor survey
(19:39) Dolph resurrects the will to live
(21:26) The beauty of the survey
(23:15) Transparency and timeliness
(24:13) The Good Will Jar
(26:45) Swap Your Shop!
Links:
Rachel’s agency: www.mightycitizen.com
Survey crafting tools: www.mightycitizen.com/donorsurveys
Sign up for “all of Mighty Citizen’s thinking”
Read the Transcript for Episode 105 Below or Click Here!

---

Transcript – Episode 105 – The Donor Survey: Best Practices for Real Results with Rachel Clemens

Dolph Goldenburg: Welcome to the Successful Nonprofits™ Podcast. I’m your host Dolph Goldenburg. Today, we’re going to be talking about fact versus opinion, and I got a short story to tell you. Yesterday, my husband and I walked down to the pharmacist to get our flu shot. Now, I should also explain we’re recording this in mid-October, probably won’t be released until I would guess November, December. So, don’t flip out when you’re on just getting my flu shot now. I got it in October, and then this morning I watched CBS. Yes, I know. That makes me old automatically, but I watched CBS this morning, and they said that last year, 53 percent of those surveyed did not get a flu shot. And what’s more, when asked, “Why?” 48 percent said they don’t trust the vaccine. Forty percent said they don’t believe that it prevents the flu, and 26 percent said that it’s not effective.

Now, obviously, that adds up to over 100 percent. So, some people said multiple things, but I felt like this might be the perfect segue to fact versus opinion because you can’t beat science. It is a medical fact that if you get the flu shot, a) it will not give you the flu and b), you are more likely to not get the flu; c) if you do get it, the severity will not be as bad. So, that’s my little rant about the flu shot as I prepped for this show on facts versus opinions. That’s really truly what launched into my head. And today’s guest, Rachel Clemens has a great motto. She says research kills opinions. It is interesting to consider how much of what we know is true may in fact just be our belief. We believe it to be true, but we don’t know that it’s true. And maybe even more to the point, what we know might just be plain wrong.

So, what happens when different stakeholders in your organization are pushing seriously divergent strategies based on opinions that they hold as absolute fact brought down from the mountains of the gods? We can’t know everything. And that’s where research can light the way, and that’s where our conversation with Rachel, I think will be incredibly helpful. She is a firm believer in research. She is the Chief Marketing Officer for Mighty Citizen, and among other things, she crafts donor surveys, the results of which are often quite enlightening for the organization. Then she helps nonprofits use the data from the surveys to implement better fundraising campaigns. So, let’s cue the music and hear how exactly this works.

Hey, welcome to the podcast, Rachel.

Rachel Clemens: Thanks, Dolph. Thanks for having me.

Dolph Goldenburg: So, I love your motto: research kills opinions, and I just gave an example from public health, but can you give us a donor survey example that illustrates what you mean?

Rachel Clemens: Yeah. We had a client we did donor survey for, and they thought people gave to them because they believed in engineering. It was a STEM organization that helps girls get into science, technology, engineering, and math. They thought it was because a lot of their donors were from the STEM occupations. What they came to find out through the research was that a lot of their hours were actually people who were against the current political climate, so they saw that in their opinion, girls were not necessarily given the same chances as boys when it came to STEM education. They gave because they wanted to empower our girls and less about the actual occupations of their own careers. That was something that internally the development team felt like most of their donors came from those organizations, and that wasn’t always the case if they weren’t always working those career specialties.

Dolph Goldenburg: That stem organization, had they done a survey before?

Rachel Clemens: No, they hadn’t. This was their first donor survey.

Dolph Goldenburg: So, the first time that you had a conversation with them about doing a donor survey, what was their response?

Rachel Clemens: They were sort of skeptical about it. Honestly, you know, they had some internal disagreements among them as well, and that was sort of what triggered my recommendation for a donor survey. It was because, you know, if I come in and I ask       the stakeholders a question within the organization, and I get slightly different answers, or I get disparate answers, then I start going, okay, do we really know the answers to these questions or are these are opinions? That’s what triggered the conversation to say we should probably have a donor survey and make sure we actually know the answers to these questions.

Dolph Goldenburg: When an organization is ready to actually put together their donor survey, how do they go about doing that?

Rachel Clemens: Yeah. So, one of the first things you have to do is to figure out within the organization who needs to be in the room and who has a say in what’s going to happen with the survey, what questions are going to get asked, how we’re going to use the answers to make a change. So, one of the first things I always tell our clients is figure out who is buy-in and get them the room. And then once they’re in the room, that’s when you start talking about the goal of the survey. So, what are we trying to accomplish with the survey? For example, in the case of the STEM organization, the number one thing they wanted to know was why their donors give to them. So, okay, once we find out how and why donors give to us, what are we going to do with that information?

So, it’s not enough just to be curious and just to get answers. If you’re just curious, going to the extent of a donor survey is overkill. I would only go through a donor survey if I plan to do something with the answers that I get. One of the things we always encourage people to do is when you’re sitting in that meeting with those stakeholders, when someone comes up with a question that you want to ask, you write down that question and immediately, you follow that question with, what will we do with the answer? Once we know why our donors give, how are we going to follow up with that, what are we going to do with that information? So, for the STEM organization, their answer was to use this information to better engage their donors in future communications. When they found out that people were giving, you know, in part because of political happenings then they needed to be able to say we will use that information to talk about those happenings in the world, and they had to be okay to do that.

You have to know going in that the results may shift how you talk about your organization. It may shift your strict structure. If you’re running a donor survey around a particular program, and you find out that people don’t like the program or don’t think it’s making change, what are you going to do with that information? What kind of changes are you willing to make in regard to that program?

Dolph Goldenburg: What are some examples of questions that have been discarded after you have asked, what are we going to do with this information?

Rachel Clemens: That’s a great question. One of them that we discarded was, will you give to us again in the future? I made the recommendation to discard because it’s really hard to ask people what they’ll do in the future. We’re not good fortune tellers; we don’t even know our own, what we’ll, what we’ll do ourselves in the future. That’s one reason we struck it, but then when we asked the “what will you do with this” question, they didn’t have a great answer, meaning whether or not people said they were going to give to us, we still wanted to ask them to give. If they answer, “Yes, I’ll give it to you in the future,” that didn’t mean they were going to treat them any differently. If they answered “No, I don’t plan to get you in the future,” they still wanted to be able to make that ask.

So that’s one of the questions, for example.

Dolph Goldenburg: I have seen the question, how likely are you to give him the future? Do you see that as a useless question as well?

Rachel Clemens: I think it is, yeah. I mean, especially when the organization is putting on the survey themselves. If they had a third party put it on the survey, I think maybe you’d get better answers to that question that. If these are your donors, they’re inclined to want to give you good news. I think if you ask, “How likely are you?” they’ll say they’re likely whether they are or are not.

Dolph Goldenburg: It’s interesting… I think sometimes the theory – and again this is opinion and not fact – but I think sometimes of this theory as well when someone says they’re likely to give in the future, that actually moves the needle just a little bit closer to giving again in the future.

Rachel Clemens: There’s a lot of contradiction on whether you should make an ask in your survey. I know I’ve got a friend who she does her own surveys as well, and she likes to make an ask in the survey, and I don’t like to make an ask in the survey. My argument is if I’m here to gather information from you, I’m already asking you to give up your time. I don’t want to ask you to give money in a survey as well or to have you doubt my intentions. She would argue that actually she sees people give during surveys. This is an area where there are a lot of opinions, unfortunately, but there’s also a lot of best practices and stuff, too.

Dolph Goldenburg: So, does this mean you also avoid questions like, how likely are you to give stock in the future or how likely are you to make a planned gift?

Rachel Clemens: Well, I actually do like the planned gift question. I think there is a little bit of an ulterior motive to that. When you ask someone, “Have you planned to make a gift as part of your estate?” well, they might have known that was an option. So, I kinda like it as a way to educate the audience whether or not they have a very good answer for it.  You at least raise something that they may have even thought about.

Dolph Goldenburg: I’ve also found it a good way to educate the organization as well. I know at one point I was helping out an organization with a donor survey, and they pretty much said, “No one is ever going to give us stock. No one is ever going to make a planned gift. I do not even know why you’d ask these questions.” We surveyed, you know, I think like 1800 people. We actually got a reasonably good response rate. We’ve got about 200 or so responses and about five percent, different universes, five percent were interested in maybe doing stock at some point and five percent were maybe interested in a planned gift at some point. It educated the organization. They said, “Oh, there is a small percentage of our donors who are interested in this.”

Rachel Clemens: They just thought it was so far beyond as an online survey that’s surfacing some of that information. Yeah, there are always surprises. That’s what I love about the surveys is they are always a surprise. Something you didn’t expect.

Dolph Goldenburg: Now, I know you said that you try to tease out why people give, but do you also tease out the reasons people choose not to give or not to give again?

Rachel Clemens: A Lapsed survey can be a really good exercise, a lapsed donor survey. So, if you’ve got donors that used to give to you and having given it awhile, I think asking them why they stopped giving is a great exercise. They may not even realize that they stopped giving. Maybe they had you on a credit card, and the credit card stuff working, and they didn’t realize it. I think surveys are great for getting information, but they’re also great for educating an audience; that’s just another way to do it. I think if I’m going to do a question and it’s like, “Why have you stopped giving?” it would definitely be multiple choice. Usually, people have multiple reasons for why they don’t give, and that would also include an “other” area for them to type in a response that you may not have even thought of. I think that’s the trick, especially when you’re talking about questions where you’re providing possible answers to the user to select from a list. You have to make sure you’re really exhaustive in the options that you give them to answer, and the chances or you’re going to miss something. Someone in that survey response is going to be like, “Oh, they don’t have my thing listed here,” so make sure you give them a choice of another to cover those things.

Dolph Goldenburg: Right. Do you typically only survey donors and lapsed donors, or do you survey the people in the CRM, the donor database, that have just never given but are in the donor database?

Rachel Clemens: We have done surveys for any number of things. We’ve done surveys, even not just donor surveys but brand surveys. We’re marketing community education, so we do all kinds of things. We surveyed an audience that was made up of a performing arts center, their general email list. So, at some point they had bought a ticket; they’d raised their hand to be on the list and they had shown some engagement, but it was in varying degrees. You had donors on there; you get volunteers, season ticket holders, and you had people that have probably never bought a ticket. So, we surveyed that entire list for a brand exercise. They were going through a website redesign, thinking about what users wanted from that, and it was clear that they had a logo that law wasn’t resonating.

We said, “Look, we’re going to ask these questions about the website. What if we just throw in some questions about the brand as well about the logo? And remember it’s a performing art center.” And then we asked users, “What’s the number one word you would use to describe our logo?” We showed a picture of the logo, and the number one word used to describe the logo was boring and number two was dull. And so, this is a performing arts center. So, it just made the case really clear that we had a problem with the brand, so we’ve asked all kinds of all kinds of questions

Dolph Goldenburg: So, did it drill down at all as to why people felt it was boring or dull?

Rachel Clemens: No, not really. It was more of a gut check on the look and feel of it. It was a circle way type then. So, it was boring and dull, quite honestly. They were known for being a creative theater group basically, and it just does not come across in the brand.

Dolph Goldenburg: What did the theater group do when they got those results?

Rachel Clemens: We had talked about undergoing a brand exercise, and they weren’t sure they needed it. The results clearly showed that they did. So, we decided to evolve the logo as well. So, we went through a whole new logo exercise.

Dolph Goldenburg: Very cool.

Rachel Clemens: Yeah, it was neat. It was unexpected.

Dolph Goldenburg: What are some of the biggest challenges you run into when you’re designing surveys?

Rachel Clemens: One of the biggest challenges with surveys is asking the questions in a way that doesn’t bias the response. We all have an implicit bias just by nature of being human. When we’re asking our questions, it’s really easy to fall into a lot of the shortcomings that can happen with survey questions. Here’s an example, if you are a nonprofit that offers summer camp, one of your questions might be, what is the most affordable and most fun summer camp? Well, the problem with that question is this:

It’s actually two questions in one, so I can’t answer that question. Are you asking me about the most affordable or the most fun? They might be two different answers. That’s a common one that we run into. Another one is, how much did you enjoy our annual gala? Well, if you asked me how much I enjoyed something, I’m expecting to say I enjoyed it. Right? It’s easy to make those slip-ups. Naturally, we don’t necessarily think much of them. And then, of course, another easy one is if you’re asking someone what time range they were born? You’re offering, you know, answers in the closed question. So, what in what time range were you born? And your answers are 1940 to 1960, 1960 to 1980 or 1980 to 2000. Well, if you were born in 1960, there are actually two correct answers to that.

So, there’s just a lot of those little things that can quickly add up and go wrong.

Dolph Goldenburg: So how would you reword the question, how much did you enjoy our gala?

Rachel Clemens: I would phrase it, in one word, how would you describe our gala? And you’ll notice I keep kind of using the one word as an example, and the reason I like that is you’re allowing the user to put in their own answer to think about it and provide their own answer. It’s actually pretty easy to analyze a one-word answer. So, for every open question you ask, you’ve got to analyze all the responses to that question. So, I’ll give you a quick example. When we were doing the performing center survey, we had 2,500 responses to this survey. We asked three open-ended questions. So, that means we had to read through 7,500 answers to those three open questions. It took forever. That was definitely a lesson learned kind of early on in my career. Make sure that if you’re going to ask open-ended questions, you’re asking only one or two in the survey. I like the one-word questions because they’re easy allies, and you can build a wordle or word cloud.

Dolph Goldenburg: That’s where I was going. You build a word cloud with it.

Rachel Clemens: Exactly, and it’s easy on the user to say, Hey, if you asked me for one word to describe something, there’s a word that’s generally going to come to mind pretty easily and pretty quickly.

Dolph Goldenburg: I know you said you get 2,500 different answers, but roughly what percentage does the most frequent answer represent?

Rachel Clemens: For open questions?

Dolph Goldenburg: Yeah, so for the one-word open question.

Rachel Clemens: I want to say for the boring and dull, we kind of lump those together because they were so similar. They made up 35 percent of the responses. Yeah, it was pretty overwhelming, you know, because you were going to get a lot of answers, but because we could kind of put those together, I think simple was the third, which is not boring or dull, but it’s kind of in the same vein

Dolph Goldenburg: But it is a statement if you have a very simple logo.

Rachel Clemens: Yeah

Dolph Goldenburg: Rachel, we’re going to take a short break, and when we come back, we’re going to discuss what organizations should be doing with the data once they get the results from their survey.

So, we can now see what Rachel means when she says research kills opinions. It’s a good thing. It allows the light of reality to shine into the dark cave of opinion. A not so good thing, however, is killing creativity and enthusiasm, and sometimes, seemingly even our will to live by forcing people to endure long, disorganized and ineffective meetings. Check out the Successful Nonprofits™ Bonus Break Six Tips for More Effective Meetings. It’s a quick listen, maybe eight, nine minutes long, and it will help you and your board see that meetings don’t have to be a death sentence. That bonus break will help you bring life back to your meetings and your board. Go to www.successfulnonprofits.com to check it out.

Hey, Rachel. Welcome back to the podcast. As I promised, let’s now focus our attention to what organizations actually do with the data once it’s collected.

Rachel Clemens: Yeah, thanks.

Dolph Goldenburg: Now, you have delivered this amazing report to the organization. What do they do with it? How does it impact their internal operations and also, you know, how do they communicate it externally? But let’s start with the internal.

Rachel Clemens: In the case of the Performing Arts Center, you know, it helped us to narrow in on some things that we didn’t know we had problems with, and it helped us to identify where we had gaps. Again, they were looking at redoing their website. There were certain things they wanted the website to do; we had also found out that there were things the website wasn’t providing for users that they didn’t even know about. So, we can identify where we are today based on the survey, and knowing where we want to go helped us to build that bridge to say, okay, here’s how we’re going to get there. So, that was basically our role in this was to analyze the results and then make the recommendations for how to get them from point A to point B and how far that actually meant they had to go.

Dolph Goldenburg: Do you ever have board or staff that pushed back on what the data shows and say, “Okay, yeah, I see. I see numbers in front of me, but I don’t believe it.”

Rachel Clemens: [Rarely] I mean, that’s the beauty of the survey and that’s the other reason it’s really important to have those stakeholders in the room early. So, in the case of the Performing Arts Center, we did focus groups in conjunction with the surveys, and I highly recommend that anyway. Surveys are great for quantitative data, but it’s hard to go deep on a survey. And so, usually what we’ll do if we’re doing a survey will combine that with interviews, one-on-one interviews. We can draw a little deeper. We’ll typically do those after the survey results so that if we find any surprises in the survey, we can dive a little deeper on those one-on-one interviews and get additional information. We did focus groups where we had members, especially vocal members of the board, present in the focus group, so we were getting their opinions up front so we knew sort of what we might encounter. And we also had them fill out the survey as well so that they were experiencing what we’re asking of our users. But truthfully, if they’re involved early enough and trust the process and know exactly what steps we’re going to take, they tend not to argue with the results. If they feel like the research was done appropriately, then they are generally ready for the results and not giving much push back as long as they were involved early.

Dolph Goldenburg: Change can be hard for team members or that staff or board, but change can also be hard for donors. How do you communicate the survey data to donors so they understand the change?

Rachel Clemens: That’s one of the questions that often comes up, are we transparent, and do we provide the findings back to our donors? That’s something that the organization just needs to answer for themselves. So, when we’re meeting that first meeting we talked about with the stakeholders, one of the questions and should be, are we going to share this information back to our donors? I think in good practice is to always share that back to be transparent. Um, but sometimes we’re asking really tough questions, and we need to spend a little time with our own or figuring out what we’re going to do with it before we can report back. You don’t want to wait too long where people have kind of even forgotten about it. So, usually, in the case of the STEM organization, we ran this survey for them in July, and we’re now in October, so they have not reported back yet, but they plan to.

They made a commitment to report back to the donors when they took the survey, and so that is coming for them. I don’t know when they plan to release it, but that is something I would always ask myself often. I think it’s a great exercise to report it back to say that we heard you, and because we heard you, here are the changes that we’re going to make for you. And I’ll just say one more thing. I like to think of our relationship with our donors as having a goodwill jar between us. When your donors make a donation to your organization, they are adding to the goodwill jar. When you ask your donors to complete a survey, you were taking from the goodwill jar, making a withdrawal. Your job is to make a deposit back into the goodwill jar with the results of the survey. If they tell you that they want to receive less emails from you or that they want to be recognized in a different way, you actually have to put those things into action, and that’s your deposit back to the good little jar.

Dolph Goldenburg: I’d love to hear your thoughts on this. My general feeling like when communicating as an example, donor survey data back to donors, I think it’s not just okay, it’s probably preferred to give them some high level infographics; there are very few donors, you know, you may have three major donors who you actually want to sit down with a full report, um, you know, but most donors don’t want 20 pages of data. What they want is the one-page, high-level. Do you generally agree with that or not?

Rachel Clemens: Yeah, I will typically make recommendations that they do the top three things we learned from the survey. Sometimes, you don’t even want to share the actual numbers because you know, there’s just lots of reasons you may not want to share every little detail, but I will typically recommend that our clients do an executive summary where they say, here are the three things we learned. Here are our plans to do something about it. It’s really not enough just to say here’s what we learned, especially when you’re talking to donors. You want to make sure you tell them how you’re going to make change based on that.

Dolph Goldenburg: Right. And then, of course, then they’ll see the change coming and understand that. So, they’ll be like, “Oh, I get why I’m not getting a membership card anymore for the theater,” or whatever the change might be.

Rachel Clemens: Right, right. So now instead of being wondering and lost.

Dolph Goldenburg: Rachel, I’ve got to save time for the Off-the-Map question. I ask every guest a question that has really nothing at all to do with what we’re talking about today, but it will help our listeners and myself get to know you just a little bit better, and I am fascinated by something that I saw on your LinkedIn profile. You apparently were a partner in Swap Your Shop, and this was an exchange program for creative professionals to temporarily swap lives. So, in my head, it kind of sounds like that Wife Swap show or that Trading spaces show where you trade houses with somebody. So, talk to me about swap your shop.

Rachel Clemens: Yeah, so when I was younger in my career as a graphic designer, I lived and worked in England and Australia in two different stints. As I got and a little bit more established in my career, it got harder to just give up a job and move for three months. So, some friends and I were in the same problem were kind of riffing on what it might look like if we started an organization that helps creative professionals swap work and home. So, basically, someone in Austin, Texas could trade places with someone in Bangkok, and I would go and work from their office space, but I would keep my job here in Austin, and I would work on my clients here in Austin because so much of our jobs can be done remotely. What you’d get from trading the homeless someone and the office space is you get that social component that you don’t get if you’re just, you know, staying by herself in an Airbnb.

It’s been years since we’ve worked on it, and we had swappers between Brooklyn and Valencia, Spain, and they were written up in Entrepreneur Magazine or [CNBC Magazine]. We’re just a little scrappy startup. And it was funny, the idea was you could tell the idea was good because it just sort of took off. Like, we didn’t have to do much to size it. The media was calling us. It was very surprising. It was a great idea. Honestly, at the time that we started it, several of us were having babies and starting businesses, and it didn’t live up to the dream in the end. I still love the idea and still think maybe someday.

Dolph Goldenburg: Yeah. So, at some point, did you swap your shop?

Rachel Clemens: No, but we had an employee who wanted to swap his shop. Like, he wanted to go for us overseas. It ended up not paying out. I would have definitely let my employees do that because if you’re an employer, you got a good employee, you don’t want to lose them, and they should be able to go travel and do those things. As long as they’re good, I would have loved to have kept him on and just have some working remotely for, you know, however, up to three months or so.

Dolph Goldenburg: I always say this on the podcast, I’m incredibly fortunate because I’ve gotten to travel a lot and, in some cases, got them to spend a good bit of time, like two months in a country. For me, that would be a dream to able to say, “Okay, we’re just going to swap shops and I can still do my work, but from another country and learn what it’s like to live there.”

Rachel Clemens: Well, it’s cool, too, because like in the example of the Valencia and Brooklyn swap, they did office out of each other’s offices; that was great because they met each other’s coworkers, you know, they saw each other in passing; they ended up becoming friends because they knew so much about each other at the end of the, you know, few months. So, it was neat.

Dolph Goldenburg: That is super cool. Rachel, I hope in your next act you revive that, and I hope I’m one of your customers.

Rachel Clemens: All right, I’ll let you know.

Dolph Goldenburg:  So, I have super enjoyed talking with you today, and I really appreciate coming onto the show. Now, listeners, you can find Rachel at www.mightycitizen.com, and you can add a backslash and the word donor surveys to access free tools and training for fundraisers and communicators that are interested in putting together surveys. You will find things like templates, how to guides, presentations. You get the picture; go to www.mightycitizen.com/donorsurveys  to get more. You can also sign up to receive, as they say, all of Mighty Citizens’ thinking directly to your inbox, and that sounds pretty cool as well. Hey Rachel, thank you so much for joining us today.

If you were unable to catch all of Rachel’s contact information and free offerings in my outro just now, don’t worry. You can still go to www.successfulnonprofits.com and check out today’s show notes, and you will find all of Mighty Citizen and Rachel’s contact information there. Now, Rachel gave me a lot to think about today, and I am sure that she gave you quite a bit as well, so I’d love it if you reached out to me on Facebook, LinkedIn, or Twitter and shared with me what your takeaways from today were. I would also be truly grateful if you gave us a review on Stitcher, Libsyn, iTunes, or wherever, however we come to your ears every week. Your feedback really makes all the difference. That is our show for this week. I hope you have gained some insight to help your nonprofit thrive in a competitive environment.

(Disclaimer) I’m not an accountant or attorney, and neither I nor the Successful Nonprofits™ provide tax, legal or accounting advice. This material has been providing for informational purposes only and is not intended or should not be relied on for tax, legal, or accounting advice. Always consult a qualified licensed professional about such matters.