How to Conduct a Communications Audit, Part I

I once spoke on communications audits at an ASAE Annual Meeting and regretted it as soon as I did it. Rather than take the sensible approach and speak on very specific how-to's working with a consultant, I took the high road in that program and provided a non-profit's view of conducting them. The interesting part of our well-rehearsed panel was that I and the three association speakers all did a terrible job of explaining HOW we did what we did. Instead we shared descriptions of our program and a little about what we learned, but audience members expected something more like a recipe book and they weren't shy about expressing it.

Well, it's never too late to share what I probably SHOULD have presented back then, beginning with advice on how to conduct the qualitative aspects of an audit (in later posts, we'll share some examples). It might be odd coming from someone most people think of as a researcher, but I never rely heavily on surveys—in fact, relying solely on them can be misleading.

It's key to obtain a mix of "wide" and "deep" feedback—quantitative and qualitative that covers the cross-section of your entire audience, yet still drills down deeply with current readers/users. Although you may get conflicting findings and direction, you'll have constructive advice in specific areas from those who read and recall what you've sent them. You will also understand the general reasons that lead others to not read your communications. Both audience segments represent opportunities to increase & improve the effectiveness of your communications.  

Surveys have their uses, but in general, qualitative is better for drilling down for preferences, feedback, and comparisons to competition: 

* We have typically scheduled telephone interviews with a small sample of individuals, consciously choosing a mix of leaders/engaged members and less-engaged mailboxers if we can't screen for readership patterns up-front. These are often long conversations (25+ minutes) so recruit by scheduling appointments, warning them of the length, and we're flexible to accommodate members' schedules (i.e. evenings or weekends if necessary). Your initial survey can be a great vehicle for building a pool of interviewees, or to identify respondents for a followup survey that explores initial findings in greater depth or to test new concepts.

* "Mall intercept" can work well. You can send several interviewers onto an exhibit floor armed with a copy of your magazine or recent publication and a clipboard or a recorder to capture their feedback and measure aided recall.

* Direct observation can complement interviews to more accurately measure real behavior. You don't have to do full-blown hidden-camera ethnographic consumer work as retailers do to benefit: schedule a few visits with cooperative Lexington-area members so you can speak with them in person, look at their virtual & physical in-boxes, and talk about what they read quickly, slowly, or not at all using specific emails and print pubs as a prop to help spur their memory and to help them give you richer examples of what they read.

* Focus groups can effectively measure top-of-mind reactions. Exercises can be used to evaluate copy & visuals. We've used storyboards where donors can "vote" and tell us what envelopes they would or wouldn't open and why. Also helpful to have participants sit down and read a sample email or letter, then share with us what works and what doesn't, and describe the impressions they formed regarding the organization that crafted the communications piece.

All of these can yield pretty eye-opening results. I know most of us don't have time to do more than just one or two things, but these methods will generally prove to be worth the extra work.

How to Conduct a Communications Audit, Part I

I once spoke on communications audits at an ASAE Annual Meeting and regretted it as soon as I did it. Rather than take the sensible approach and speak on very specific how-to's working with a consultant, I took the high road in that program and provided a non-profit's view of conducting them. The interesting part of our well-rehearsed panel was that I and the three association speakers all did a terrible job of explaining HOW we did what we did. Instead we shared descriptions of our program and a little about what we learned, but audience members expected something more like a recipe book and they weren't shy about expressing it.

Well, it's never too late to share what I probably SHOULD have presented back then, beginning with advice on how to conduct the qualitative aspects of an audit (in later posts, we'll share some examples). It might be odd coming from someone most people think of as a researcher, but I never rely heavily on surveys—in fact, relying solely on them can be misleading.

It's key to obtain a mix of "wide" and "deep" feedback—quantitative and qualitative that covers the cross-section of your entire audience, yet still drills down deeply with current readers/users. Although you may get conflicting findings and direction, you'll have constructive advice in specific areas from those who read and recall what you've sent them. You will also understand the general reasons that lead others to not read your communications. Both audience segments represent opportunities to increase & improve the effectiveness of your communications.  

Surveys have their uses, but in general, qualitative is better for drilling down for preferences, feedback, and comparisons to competition: 

* We have typically scheduled telephone interviews with a small sample of individuals, consciously choosing a mix of leaders/engaged members and less-engaged mailboxers if we can't screen for readership patterns up-front. These are often long conversations (25+ minutes) so recruit by scheduling appointments, warning them of the length, and we're flexible to accommodate members' schedules (i.e. evenings or weekends if necessary). Your initial survey can be a great vehicle for building a pool of interviewees, or to identify respondents for a followup survey that explores initial findings in greater depth or to test new concepts.

* "Mall intercept" can work well. You can send several interviewers onto an exhibit floor armed with a copy of your magazine or recent publication and a clipboard or a recorder to capture their feedback and measure aided recall.

* Direct observation can complement interviews to more accurately measure real behavior. You don't have to do full-blown hidden-camera ethnographic consumer work as retailers do to benefit: schedule a few visits with cooperative Lexington-area members so you can speak with them in person, look at their virtual & physical in-boxes, and talk about what they read quickly, slowly, or not at all using specific emails and print pubs as a prop to help spur their memory and to help them give you richer examples of what they read.

* Focus groups can effectively measure top-of-mind reactions. Exercises can be used to evaluate copy & visuals. We've used storyboards where donors can "vote" and tell us what envelopes they would or wouldn't open and why. Also helpful to have participants sit down and read a sample email or letter, then share with us what works and what doesn't, and describe the impressions they formed regarding the organization that crafted the communications piece.

All of these can yield pretty eye-opening results. I know most of us don't have time to do more than just one or two things, but these methods will generally prove to be worth the extra work.

Changing Gears & Entering Retirement Gracefully

It's a little embarrassing to visit one's old blog and to realize that it's been a half year since I've written anything. I've always found it easier to post responses to the questions posed by peers on the ASAE listservs, but I realize that's a somewhat weak way of contributing to the community, promoting the business, or accomplishing whatever other objectives any of us has when we attempt to do something beyond just earning an income when we shift over from staff positions, or wherever we were, to become a full-time consultant.

Ironically, it's only at the time that I decide to retire that I also decide to become more diligent about this form of outreach. Mind you, I'm reasonably pragmatic regarding my prospects of reaching anyone through this vehicle. Blogs written by random strangers are like the Web's "message in a bottle"--compositions prepared in isolation on someone's computer and cast adrift with the resigned expectation that someone might see one message. But it's better than a podium (something that I find to be more and more archaic as a method of expression or knowledge transfer) and the blog can be a powerful tool for sharing knowledge as long as I think of it more as a wiki-type compilation & repository than a time-sensitive series of news items.

Even quantity of eyeballs can be a simple matter of attitude: I laugh when I think back to how I viewed the audiences I wrote for years ago. I had 7,000 readers of the newspaper I edited at ISU almost 30 years ago. I had "only" 1,000 paid subscribers to NAHB's Housing Economics 22 years ago. I had millions of people potentially reading my CRS donor solicitations 10 years ago. Today I generally have just a couple of client contacts reading my lengthy reports, and a dozen board members reading some abstract of it. At the same time, my income steadily increased in each position. My real or imagined audience size has almost zero impact on what is sadly my best lifetime performance metric ... and I know it's a poor metric.

I always dreamed of being able to translate and share everything I learned over the years to benefit others like me, and I'm not sure that being immersed in consulting projects was the way to do it. I've billed more than $1.5 million in services over the past six years, and the altruist in me is actually somewhat proud that we often provided services to more than 150 association and non-profit clients at mild to sharply discounted rates. From my first job in the federal government, I've always taken the approach that work is all--grinding out many projects with a wide variety of clients would serve us and the community well. I'm probably right on the first count, but it has also meant that I've missed real opportunities to look back and provide a coherent body of knowledge from which others can benefit. As speaking invitations have declined and I haven't needed to attend events to get more work, I've consciously taken on a lower profile and I'm far less effective in serving the community than I was as a VP at NACDS, or a director at ASHP. Over the past six years, I've abused the freedom of a consulting lifestyle by being free to take on more and more work, rather than achieving a reasonable balance between meaningful outreach and churning work out for specific clients.

Starting today, I'd like to change that, or at least try. 

Learning to Create a Business Inside Your Association

Where have you been?

I am shocked but not surprised that it's been almost seven months since I did a blog post here. When I look back over this very eventful time, it feels a lot like how I 'grew' in my association jobs--for some reason I was aggressively intrapreneurial, but otherwise content to keep my head down and plug away at the tasks at hand.

It's helpful for the image of a business to be 'out there' yet I find, even with these social media tools, I'm still communicating in my normal voice. On Facebook I am still the smart-aleck little brother to my former classmates at New Plymouth High School in Idaho (I knew Facebook was here to stay when all my classmates magically learned to type and started living their lives there). On Twitter (another New Year's resolution) I pipe up twice a day with whatever is on my mind but it's generally professional and whatever fits in a haiku. Apparently Rebecca set me up so every post there also feeds LinkedIn so I am more careful than I would otherwise be. But bottom line, to paraphrase Popeye, I yam what I yam, so there are limits to how much I benefit or reach the outside world that I care to reach, because of who I regard myself to be and who I am comfortable being.

Why don't we do marketing well?
I say all this in part because as I use another weekend to recharge and reflect on conversations during the week, it's striking how often our group personality is also very non-marketing oriented. We want to speak in our authentic voice, which is admirable but often just doesn't get the marketing job done. I'm beginning to return to my past as marketing clients express grave doubts and conflicts when direct marketing letters "don't sound like them." We heard this over and over at Marketing General (and I'm sure they still hear it everyday). I heard it at Catholic Relief Services, when fellow division heads, even bishops on our board would express that it's a shame someone as smart as me would be in marketing--as if it's a waste of brains to focus one's effort in sending junk mail, managing telemarketing campaigns, and scheduling drops frequent enough to upset your best donors.

I have always felt this missed the point, because there was a science and art behind what we did. We had to spend well and keep it simple to raise enough money to keep the lights on. If we didn't have the active support (and relevant past experiences) on the part of our CEO, I have no idea how we could have employed the right techniques, aggressive frequency, and memorable messaging necessary to raise the $45 to $60 million we did every year in unrestricted giving.

I tolerated the attitude quietly in several positions, then chose to acknowledge it to the point that we did a regular new employee orientation program at CRS called "Fundraising--The Necessary Evil." Over the years I've been struck by how many associations don't have a systematic membership acquisition program. At NACDS, an otherwise very well functioning, large trade association, I had a bear of a time getting our Services Corporation up and running. I'm still struck when friends and periodic clients have me on their mailing list and I get cute fake Christmas Cards with a message inside saying 'wouldn't you like to join?' Each time I feel as if those associations haven't been able to carve out some space where they can be businesslike--ROI focused, clear messaging, and generally ambitious in creating the growth and programs necessary to expand their reach and to subsidize the other, nobler pursuits.

Successful direct marketing techniques and business development methods haven't changed much over the past 30ish years and yet the average association still seems very far away from using a fraction of the tools and tricks that work. I'd invite comments and questions but I know nobody is reading this .... and those who need help the most don't believe they have a problem. So I'll just treat this first return to blogging as a message in a bottle, and later we'll provide some tips on how even the message in a bottle can be found, read, remembered, and acted upon.

Data Mining: What's The Big Deal?

I was participating in a meeting with a new client yesterday and we were discussing some preliminary results of a data-mining exercise. Since some of the team members were unfamiliar with me or the project, I explained the process we used, and then we brainstormed for an hour on what they would like to learn and how they would apply it immediately to their normal practices. As I told them at the end, it was a rare (and gratifying) meeting when "data" and "really excited" occur together in the same sentence.

What I found most fascinating about this meeting was the latent desire among staff to have the database tell them something—ANYTHING. This isn't an unsophsticated or small organization—without giving away state secrets, they have 3.26 million membership dues transactions over the past 17 years, for example, and the quality of their questions and thoughts regarding how they'd apply learnings fell somewhere between pretty good and truly outstanding. But their environment—waiting six months for IT to download their data, receiving only what they refer to as 'sales reports' and little else—is very typical in associations, and it neither rewards nor cultivates expertise working with data among association staff.

No wonder the "data driven decision making" approach recommended in ASAE's 7 Measures book never took off. Even in other ASAE publications, common advice regarding data seems to be "don't collect or store what  you don't know how to use." This might be good advice and promote efficiency, if it wasn't so easy to actually snapshot your data—all of it—in an environment where it can facilitate ad hoc queries, periodic dashboarding, developing product purchaser profiles, measuring the migration patterns of your recent graduates into full membership, exploring the relationship between product/event purchase/attendance with membership conversion, measuring repeat customer repeat rates, doing market basket analysis, creating an RFM matrix for your fundraising, etc.

WARNING: MILDLY TECHNICAL DETAILS
The key is actually very simple—download your data just once from your AMS into a series of flat files (comma or tab delimited) and import them into a decision support tool. I use SAS, whose basic product for a single user runs $3200ish in the first year then $1600 for an annual license. John Dorman and the folks down at Texas Medical prefer to use the MS SQL that comes free with their network but he describes the cost of upgrades and training a staffer with at least some expertise in programming & analysis to be a one-time expense of maybe $8-$10,000. I find that loading and reprogramming an association's file takes me 2 to 10 hours depending on the number of modules the data is stored in and how much of the data we need to simplify or eliminate (since you don't really need to know the name of the event they registered for on July 2 2003—you just need to know it's one of ten they attended early in their membership tenure before they stopped attending but continued paying their dues). Querying it .. including re-sorts, creation of new variables, categories, etc. in new datasets might take 5 to 20 minutes, even for files with hundreds of thousands of members or millions of transactions. Of course, most consulting isn't iterative: most of our reports have to be large and episodic, rather than small and applied, because we're paid to do projects rather than programs, but if you added this capability in-house (my recommendation) any association who takes this approach could have answers literally on demand without annoying the IT staff with requests or annoying everyone by slowing down a production server.
END OF MILDLY TECHNICAL DETAILS

The sad part of this is that the technology is actually ancient. It's been this cheap at least for the almost 25 years I've worked with these systems: at first nobody believed it was possible because we were using AS400 mini-cmoputers (actually mini meant 'not a mainframe;' weighing 1000 pounds and sometimes being fed by magnetic tape reels or cartridges). But once you turn the corner on this, in 1987 or 2010, seeing is believing. It's a simple process and I promise you'll never miss your 80 page reports again, nor do you have to pay $50,000 for 'data integration' or other support to integrate Cognos, Crystal Reports, or any other tool de jour. And after doing it inside of several associations I also never ran into the program of being regarded as the nerdy 'stapler guy' once we proved the ease and power of real, daily data mining.

Exit Surveys: Let Us Know How We're Doing ... And Turn Off the Lights When You Leave

I hope that most of us in associations conduct exit surveys, and we recently considered how we should increase our participation rates on the ASAE listserv.

Good objectives for exit survey include:

 Collecting valid research results
 Distinguishing between avoidable & unavoidable attrition
 Ending this stage of the member relationship on a good note, and
 Clean the database for subsequent efforts to reinstate those who will want to come "back to the fold" in the future.

Our advice in this area includes:

1) Do a multi-part contact sequence. Starting with the lowest cost-per-contact option and migrate into costlier ones with the deliverable names who are not responding. A typical sequence might be: #1 online survey, #2 mail or fax survey, and (possibly) #3 telephone.

With any online survey, we recommend using an invitation & two reminders to non-respondents over a several-week period. This constitutes a reasonable and consistent effort from which they can opt out anytime or respond, and you should suppress them from the reminders. You'll always annoy the vocal 1%, but more persistence conveys to the former member that you're serious about wanting their feedback.

2) Make sure the message comes from an high ranking executive or volunteer name as the sender and signer. The reason many of our members leave us may be tied to something big & strategic within the association or in practice in general. Rightly or wrongly, a former member may regard correspondence coming from you or the membership department in general as transactional rather than an earnest attempt to learn and take action to address the issues a former member identifies that's within their control or influence.

3) Don't worry about anonymity but rather, offer confidentiality instead. You have to know who they are for proper campaign management but you won't report any personally identifiable findings. For your own internal operations, it's often worth the time to go back through your AMS and track the history of that one lapsed member who told you an interesting story about their experience ... if you don't know who they are you can't learn nearly as much about what your association did over their membership cycle to make them feel that way or form that impression.

4) Once you're using mail, offer a response premium for the first time but make it effective. The academic literature has demonstrated for years that front-end premia always outpull back-end; in our increasingly instant gratification, Amazon two-click world, it's probably even more pronounced. Offering respondents their choice of a $25 gift certificate toward your membership/services or an AmEx gift card could appeal to some (back-end premium); it would be far more timely and appealing to provide the incentive to ALL respondents included with your survey.

It helps to guilt a response (you gave me something, okay I guess I'll tell you what I think), and it can be easy to implement through a buckslip coupon insert in your #10 survey mailing, or a perfed tab on your outbound postcard that they can redeem online.

5) Monitor deliverability through each method and include this outcome in your final metrics. For example, if you have 500 lapsed members in April, it's helpful to know that 20% are unreachable due to job changes (bounces, NCOAs, phone confirmation of moves), 45% dropped due to inadequate value, 25% due to employer no longer reimbursing for dues, 20% because other associations meet their needs better, 15% for other cost-related issues, 10% due to retirement, etc. and that 20% might seriously consider rejoining in the future. You can infer the first figure based on actual campaign performance, the remaining figures by extrapolating from your findings based on any response rate.

I think there can be far greater impact if you can link exit survey insight to your ongoing membership operation on a personal basis, by 'flagging and tagging' individuals who report specific reasons for dropping. In the long run, most former members still in the field might not actually care about their confidentiality, and some exit survey findings could lead to additional followup such as more detailed discussions with the willing.

Former members in general appreciate knowing that you're working on the issues that led them to drop their membership and the additional conversations could allow you to probe further, test possible solutions. You certainly wouldn't want to spend all your time doing labor-intensive things for a low-potential reinstatement prospect, but if you can focus specialized attention on the individuals with the most interesting and/or prevalent comments, or those with pertinent demographics or practice characteristics, you can incorporate this into a much stronger member retention process. –Kevin

To Be Interesting, Be Interested First: How Asking Questions Sets Up Effective Copywriting

In a recent ASAE listserv, a staffer asked the basic question "What words do you use for 'networking' or 'education' in your copy?" I was glad to see it because so often we use a kind of shorthand with the outside world—members and non-customers—without really thinking them through. This often reflects the fact that if we're not careful, we can easily prepare our communications in a monklike isolation from the real voice of our customer. 

There are a number of terms that we all happily plop into our copy, but we often don't realize how people in our target audience actually process these terms … or the overall message we're trying to get across. When I write copy, I always ask for permission to contact a few members and schedule phone interviews with them. For most of us in associations, it should be an easier, regular, less formal process.

Some of the actual questions I'll ask a member or customer include:
* "Do you encourage other peers to consider membership? If you have, what kinds of reactions do you get from them?:
* "Pretend that I'm a peer who doesn't hold an ___ membership. How would you describe the organization to them/me?"
* "Thank you for the description. There were a few things I didn't hear ... tell me about ______ (often networking, education, advocacy, and information)." or alternatively...
* "You mentioned networking. That's a great buzzword that seems to mean different things to different people in different fields. Tell me more about what YOU regard as networking." (Ditto for education)
* "How valuable is (networking, education) to you? Why? If you don't mind, give me an illustration of some recent (networking, etc) that you've done."
* (If necessary) "Of course, we're speaking today to help us communicate more effectively. We're talked what you think and do. How similar are you to your peers? How might they think differently about ________?"

In a few short questions this allows me to have a long conversation that helps me "get inside the market's head." The member or customer won't mind--after all, nobody has ever had a bad conversation about themselves! It can help any of us figure out how to convey the benefits we offer, far beyond just using a single word that may mean different things to different readers, and to make it more compelling and understandable with the right words or examples.

I know the answers I get through this process DO vary by field, but often 'networking' entails idea exchange, commiseration when times are tough, 'brain-picking' on very sensitive issues they won't bring up in listservs or roundtables, checking in with old friends to see how they are doing, making new friends, basic business development, gauging reactions to what a speaker/guru just said, validating their current course of action, or living vicariously through others who do things differently from that person, or who have changed their practices. Depending on the season/year/person, the focus may be on survival, success, or just human interaction with like-minded people who share similar goals and operate in the same market/environment.

In any event, I'd encourage anyone writing copy for prospects to go through the exercise of speaking directly to their audience now and again, to find out how your audience thinks about these terms: I know many of us have staff specialists who see these functions in terms of their technical training and experience who serve as our SMEs, but they provide only half the story and that half may be very different from how your audience actually thinks. Often they can have tunnel vision regarding the issue and may emphasize literal truth over the words that are more likely to truly speak to the prospect's needs, to get their attention and be memorable. (I personally think advocacy is more limited than other benefits, as the copy often reads as if it's written by lawyers!) It's not easy, but going 'straight to the horse's mouth' and using what you learn will help you pick ... and defend ... the right words, examples, or concepts that will help prospects understand what you really mean by 'networking' ... or 'education,' or 'advocacy.'
 -Kevin