How to Conduct a Communications Audit, Part I

I once spoke on communications audits at an ASAE Annual Meeting and regretted it as soon as I did it. Rather than take the sensible approach and speak on very specific how-to's working with a consultant, I took the high road in that program and provided a non-profit's view of conducting them. The interesting part of our well-rehearsed panel was that I and the three association speakers all did a terrible job of explaining HOW we did what we did. Instead we shared descriptions of our program and a little about what we learned, but audience members expected something more like a recipe book and they weren't shy about expressing it.

Well, it's never too late to share what I probably SHOULD have presented back then, beginning with advice on how to conduct the qualitative aspects of an audit (in later posts, we'll share some examples). It might be odd coming from someone most people think of as a researcher, but I never rely heavily on surveys—in fact, relying solely on them can be misleading.

It's key to obtain a mix of "wide" and "deep" feedback—quantitative and qualitative that covers the cross-section of your entire audience, yet still drills down deeply with current readers/users. Although you may get conflicting findings and direction, you'll have constructive advice in specific areas from those who read and recall what you've sent them. You will also understand the general reasons that lead others to not read your communications. Both audience segments represent opportunities to increase & improve the effectiveness of your communications.  

Surveys have their uses, but in general, qualitative is better for drilling down for preferences, feedback, and comparisons to competition: 

* We have typically scheduled telephone interviews with a small sample of individuals, consciously choosing a mix of leaders/engaged members and less-engaged mailboxers if we can't screen for readership patterns up-front. These are often long conversations (25+ minutes) so recruit by scheduling appointments, warning them of the length, and we're flexible to accommodate members' schedules (i.e. evenings or weekends if necessary). Your initial survey can be a great vehicle for building a pool of interviewees, or to identify respondents for a followup survey that explores initial findings in greater depth or to test new concepts.

* "Mall intercept" can work well. You can send several interviewers onto an exhibit floor armed with a copy of your magazine or recent publication and a clipboard or a recorder to capture their feedback and measure aided recall.

* Direct observation can complement interviews to more accurately measure real behavior. You don't have to do full-blown hidden-camera ethnographic consumer work as retailers do to benefit: schedule a few visits with cooperative Lexington-area members so you can speak with them in person, look at their virtual & physical in-boxes, and talk about what they read quickly, slowly, or not at all using specific emails and print pubs as a prop to help spur their memory and to help them give you richer examples of what they read.

* Focus groups can effectively measure top-of-mind reactions. Exercises can be used to evaluate copy & visuals. We've used storyboards where donors can "vote" and tell us what envelopes they would or wouldn't open and why. Also helpful to have participants sit down and read a sample email or letter, then share with us what works and what doesn't, and describe the impressions they formed regarding the organization that crafted the communications piece.

All of these can yield pretty eye-opening results. I know most of us don't have time to do more than just one or two things, but these methods will generally prove to be worth the extra work.