Posts Tagged ‘evaluation’

Measurement Crucial to PR’s Business Value

Tuesday, May 18th, 2010

My learned Australian colleague Geoff Barbaro waxes rant in a post from 17 May (US time), where he inveighs against measurement.  Perhaps not the concept, as much as the practice. He asks:

Do you measure how you look after your family? Do you count the meals, the trips to school, the time spent with children to evaluate effectiveness? When you buy that great new dress or suit that you love, did you then sit down and work through complex metrics to measure what you did?

So why do you think it’s different in business? I’ll tell you why, it’s because you don’t trust people to do the job you employed them to do. You don’t believe they are motivated and care about their work, so you can only make sure they are working by measuring what they do, and then argue that this is the motivational tool. Measuring because “we do what we measure” is a failure of leadership, a failure of motivation, a failure of selection, a failure to define values, a failure of engagement and a failure of communication.

Sorry, Geoff, but this is fuzzy-headed thinking about a vital enhancement to the profession of Public Relations.

I started a comment on Geoff’s blog (a fine and interesting read, btw), but found that it was all too likely that I’d hijack it. And that’s not right. So, here is my reply to Geoff’s shot across the bow. Man the torpedos!

========================

Oh, my. Nothing like an existential rant to get one’s blood up, eh Geoff?

Let’s start by differentiating terms. Measurement isn’t gotcha. It’s not “check-up-on-the-poor-employees.” Neither is it merely about outputs or activities, at least not when it’s strategic.

We in PR have long been the only department in a firm that can say to the C-suite, “trust me” and get away with it. The question on the CEO (and CFO, especially) mind these days, however, is, “What business value do I get for my investment in PR?”

We can take a SWAG (stupid, wild-assed guess) at the answer, but then we sound like witless weasels (um, we build reputation and protect…uh, no, uh, we get media coverage…no, uh, we help the organization communicate effectively, wait, ummmm.)

The fact is that most of us don’t have a clue what the quantifiable business value of PR is, and that’s why PRSA has commissioned a task force to work on that very question. It’s also one of the driving forces in modern PR. It’s created an industry specialty that people are finding value in, even though there is much sophistry and bad measurement out there.

In modern business, every department must contribute to the bottom line. So, direct sales and the support for sales is a winner, as is direct effort to improve efficiency, save money, etc. There’s also credible research about the effect on brand awareness, attitude and disposition of various PR activity. On the internal side, engagement metrics, and employee knowledge and behavioral metrics lend credence to a communicator’s value.

The trick is to a) Measure what matters; and b) Link communication outputs to business outcomes. This is, indeed, a hairy process, filled with risks — bad math the most prevalent, if you ask me.  Correlation is not causation, but frequently it’s a pretty good stand-in for it, if your math is good.  We mustn’t give up on the goal of establishing impact metrics and ROI just because it’s so much easier if we don’t!

I don’t know, Geoff, if I agree that “what gets measured gets done,” but I’m sure that if you can’t measure it you can’t manage it.

Cheers,

Sean

@commammo

Share

Theater of the Absurd in Social Media Metrics

Tuesday, May 11th, 2010

As we PR people feel our way along in social media, the marketers are declaring the End of Times for everything else. Anecdotal evidence shows that big companies are pulling big money out of traditional advertising and funneling it into social media, and that bears examination.  But as I’ve said, I’m not ready to write obits for mass marketing/advertising in favor of “marketing to a segment of one” right this very minute.

I first heard that phrase (Marketing to a segment of one) from the lips of Steve Cone, legendary marketer and then-CMO with KeyCorp. He was the architect of dropping the “Corp” and/or “Bank” from the company name in favor of the symbol you see at right.

That made Key one of just three companies in the US bearing an eponymous symbol for its name. Shell and Apple are the other two.

Key made a strategy of getting people to see the Key logo and associate it with “bank,” as in, “I need to stop by the Key on the way home.”  The idea, Cone claimed, was to stop thinking of mass marketing — with all of its efficiency and logical, numbers-driven strategy, and think of “marketing to segments, eventually to a segment of one.” So then came emerging affluents, wealth management, small business, middle market, large corporate — all of those categories based on grouping customers in some logical way, then changing strategy to target them.

This requires information about customers and prospects. When it comes to social media, that information is scattered to the four winds, unless you’re on Facebook.  Twitter’s foray into geo-location, Foursquare, and many other social media firms are trying to gather as much data about YOU as possible to facilitate what is a pretty old marketing model.

Just as at the onset of the Web Age you had hundreds of companies popping up to “help” companies enter the Internet realm, now at the onset of the Social Media age you have companies popping up to “help” companies enter this realm. The part that twists my noodle is when companies purport to know how to measure social media come up with yowlers — like the Vitrue Facebook fan value imbroglio, the Altimeter study on correlations between social media activity and stock appreciation, and now Vitrue’s assertion that frequency of mention in social media is somehow a reflection of its social media reputation.

Vitrue offers a chance to compare brands in a handy Flash gobo that produces a cool pie chart. Just for fun, I compared Ford (which Vitrue pronounces its winner) with a couple of random words — sure enough, pop “the” in there, and you find upteen thousands (OK, 134,000) ‘somethings’ and the aforementioned cool pie chart. Ooh, and there’s a bar chart too! So kewl.  W00t!

I could go on for 1,500 words, but won’t. It’s another cow pie pretending to be a metric.  Resist this assault on rational thinking.

Share

Survey: Internal Comm Effectiveness ‘Important Concern,’ But…

Thursday, April 1st, 2010

Researchers Dr. Juan Meng of the University of Dayton (Ohio) and Dr. Bruce K. Berger of the University of Alabama cut to the chase in their research presentation at the Institute for PR International PR Research Conference. Their first finding? “Though communication effectiveness has been an important concern for organizational leaders, the assessment of communication effectiveness has not been widely applied by using business outcome metrics in organizations.” Sigh.

Meng and Berger used both the results from the 2007-2008 IABC Research Foundation/Watson Wyatt international survey of senior communicators, and a series of in-depth interviews with 13 IABC Gold Quill winners to look for process links between internal communication effectiveness and organizational financial performance.

For me, this represents a sort of Holy Grail: we internal comms experts know that our work is impactful, but have lacked the hard evidence of causality that we perceive the C-suite respects and demands. I was disappointed, yet again, though that first finding is by no means the only one.  In brief, the other five are:

  • Measuring internal comm effectiveness should be standard operating practice.
  • There’s lots of measurement going on, evaluating awareness/understanding; engagement; job performance; employee behavior, and improvement in overall business performance.
  • Everyone has good reasons why measurement isn’t as robust as it should be, and they’re the usual culprits — lack of time/money/staff and the pain of finding actual cause-and-effect toward business results.
  • The measurement approaches used are employee surveys, employee participation in communication activities and manager surveys.
  • Four valuable purposes for internal communication: Explaining/Promoting programs and policies; educating about culture and values; providing information about performance and financial objectives, and helping employees understand the business.

At Goodyear, we made great progress toward true outcome measurement for internal communications, but didn’t quite get there. We did establish a strong link between employee knowledge/comprehension, intranet use and managerial behavior, but never got the chance to take everything to the organizational performance level.

At National City Corporation (the regional bank), our focus from the first day I arrived was on external measurement, for a variety of reasons. But the internal side wasn’t ignored — we were a Gallup Q12 company, and despite the wretched economic conditions and horrific, calamitous financial performance of the company, we still topped 94% participation in the Q12.  Right until the last moment, we were using Q12 results in our planning process, as well as beginning to use editorial content more strategically. But, again, we weren’t reaching the business outcomes level of measurement.

Here’s a quote from one of Meng & Berger’s in-depth interviews:

I think the biggest challenge in measurement continues to be convincing clients to spend, not so much the money, but to spend the time. As the industry develops, I don’t have a hard time in convincing them about the validity of measurement, but they are reluctant to actually take the time away from business to actually administer surveys or focus groups or some other measurement tools.

Looks like we have to continue making those tools easier to use and more valuable, even as we continue to scale the mountain tops for the Holy Grail.

Share

The Measurement Debate Continues

Tuesday, February 2nd, 2010

The estimable Shonali Burke has started a fortnightly Twitter chat — #MeasurePR — that begun 2 February, with the equally estimable Katie Paine as first guest. I caught only the last half, which featured good discussion and the usual paroxysm over advertising value equivalency. AVE is bete noir for @KDPaine and @Shonali, who both are categorical in their condemnation of the practice. A couple of participants, however, say that there still is demand on the part of clients for AVE.

The Institute for PR Measurement Commission condemned AVE last fall, AMEC (the professional organization for media evaluation firms) has declared its intent to find a logical replacement, and a recent paper offered Weighted Media Cost as an element worthy of inclusion in measurement programming. Where does this leave us?

I have no stake in this game. My personal belief is that AVEs are bad science, but I’m also sensitive to the need to help clients. AVE is easy for a client to grasp — “if we paid for the space our story ran in, it would have cost us X.”  Katie points out that doctors won’t prescribe a medicine if it’s not right for the patient. AVE isn’t life and death — but what do we do after we’ve explained the drawbacks and negatives and the client still wants it?

I can’t help but put myself in that situation — young company, trying to latch on with a client. Do I tell the client “No. I won’t do AVE” and risk having him/her say, “Well then, I’ll go find someone who will!” ?

#MeasurePR had much more great content than this AVE nonsense, and I really do wish we could collectively move on. I’m done writing about the debate, at least for now.

Looking for a quick way to improve measurement?

Start setting objectives and measuring your attainment of them. Stop worrying about generating lots of eyeballs and do some audience research to reach the right ones. Start looking for correlations between your various communication outputs (and outtakes) and business metrics, such as revenue, cost savings, cost avoidance, time saved, help desk traffic, speed of benefits enrollment, travel system savings, expense systems savings, etc…

Share

AVE is Dead. But Ad Cost Improves Correlations

Sunday, January 17th, 2010

The debate over how best to measure the effectiveness of media relations has encompassed multiple streams of thought, moving from saying “it’s impossible,” all the way to saying, “it’s quantifiable.” Unfortunately, advertising value equivalency (AVE) became a popular means of applying dollar figures to unpaid media. You take the number of column inches in print, time of mention in broadcast, or space on a Web page occupied by the mention of the company or organization in question, and ask, “How much would we have had to pay to take out an ad of equivalent size/time?”

The AVE practice has been under attack by some of us, poorly understood by others, but more widely used in PR agencies than many would like to think. It even was formally condemned by the Institute for PR Measurement Commission this fall.

AVE has major flaws — measurement experts (including one notable, even famous one) have decried the practice and detailed why frequently. I’ll not repeat the argument here. This paper provides those details in part. Instead, I’ll merely say that even with substantial adjustments to methodology, it never represented a business outcome, was based on an assumption of equivalent understanding on the part of the receiver, and was wholly unsuited to describing success in social media. That alone was a huge problem for me.

The thing is, there is substantive research that supports the idea that editorial content about a product and an ad are perceived similarly by receivers.  A paper by Dr. Don Stacks and Dr. David Michaelson (albeit based on one experiment) found ads and editorial to be equally effective in generating interest in a new product. If that’s so, evaluating the PR placement in comparison to ad cost makes sense. PR costs orders of magnitude less than advertising.

Two papers by Angela Jeffrey, Dr. Stacks and Dr. Michaelson explored the linkages between volume of media coverage and share of media coverage and business outcomes (such as unit sales, tickets sold, etc.) and included media cost data in calculations.  This set the stage for a controversial finding: Media costs improved correlations, significantly.

Now, Jeffrey, vice president of research for VMS, and Dr. Brad Rawlins, Brigham Young University, and Bruce Jeffries-Fox of Jeffries-Fox Associates, have written a brilliant paper further detailing the relationship between cost and outcomes, with four case studies.  The “Weighted Media Cost” has a strong effect.  From the paper:

…if we’re getting better results with costs for purchasing media space and time data, should we…set new parameters for its proper use?”

Emphatically, yes. The paper, written in a very approachable and intuitive style, makes a compelling case.

Read the paper if you care at all about measurement in our profession.

Share

Amanda Chapel is Still Relevant, and Important

Monday, January 4th, 2010

Mark W. Schaefer’s {Grow} blog carries an interview with Web. 2.o critic Amanda Chapel this week that asks whether the acerbic commenter is still relevant.  I believe Amanda remains most relevant. The rivers of Kool-Aid flowing in social media need to be dammed (and damned) and few of us consistently do so.

I’m grateful that Amanda included me in her list of “critical thinkers” along with Kent State prof Bill Sledzik, Ike Pigott, Joel Postman and Mark; that’s high praise from an important voice.

Look, I’m a committed capitalist, so I don’t begrudge anyone from making money, in particular, people who are early adopters and make the personal investment needed to stay just ahead of the crest of a wave. A bunch of people have done so, and are making a terrific living at it.

Some of those people don’t have anything but an expertise at sales and a gift for jargon to qualify them, and that’s a big problem in social media. Consider that we don’t even have licensing for mainstream PR and marketing — and think about how much really bad advice organizations get from those professions.”

At least in PR and Marketing there are longstanding professional associations with codes of ethics, increasingly strong academic and theoretical foundations, and a body of research-based knowledge (Cutlip, Center, Broom, 10th ed., p 120) that qualify us as members of a profession. This is despite our many weaknesses, including the presence of our own charletons.

Social media isn’t even there yet, and it needs to get there soon in order to separate the wheat from the chaff. Despite worthy efforts from Institute for PR Measurement Commission colleagues Katie Paine, Don Bartholomew and a few others, we’re still working on how best to measure social media effectiveness beyond output metrics.

We need Amanda to continue to call out snake oil salespeople, foggy logic, asinine commentary and the real danger of a lost of authoritative, professional conduct in such a fast growing area of communication practice. That she does so with wit, style and occasional vulgarity keeps the stew from being too bland.

So, count on me not only to declare Amanda relevant, but for vote #3 for the return of Strumpette — 140 characters at a time isn’t enough space.

As for “her” anonymity — I have been of two minds about it, both “yea” and “nay,” especially following my rather “eventful” introduction to Amanda last year. But in the end, I don’t think it affects credibility at all and it offers the freedom to focus on the message rather than its sender.

Finally, skepticism is not negativity, as I asserted last June. We surely are not lemmings, powerless in the thrall of the “wisdom” of the crowds, are we?

Share

A Manager Who Can’t Communicate Can’t Lead

Tuesday, December 15th, 2009

“As soon as you move one step up from the bottom, your effectiveness depends on your ability to reach others through the spoken and written word.” It’s been years since Peter Drucker offered that bon mot, but it certainly seems to be truth. The New York Times’ Corner Office feature, which runs Sundays on page two of the Business section, talks to business leaders of all stripe, and each of them has something to say about the importance of communication to their business style.

Dec. 6, Joseph J. Plumeri, Chairman and CEO of Willis Group Holdings (the insurance broker whose name now graces the former Sears Tower in Chicago), was Corner Office’s subject. He said:

I spend 25 percent to 30 percent of my time calling my associates — whether they had a family problem or pulled off a great deal and brought in a new client, or saved a client. Two-minute phone call, or handwritten note. I can’t begin to tell you how important that stuff is. E-mails are easy, but sometimes they get in the way of really feeling how somebody feels about your effort.

Is it time consuming? Yes. But that’s what you’ve got to do…

Plumeri goes on to say that helping people understand and believe in the choices the company makes is essential to realizing business vision.

On Dec. 13, Nancy McKinstry, CEO of Wolters Kluwer, a Netherlands-based information services company, says “Every culture is very different in how people make decisions” as she relates how her leadership style changed over time according to the communication styles of her team.

In the Netherlands, where our company is based, people really want to be heard early in the process. So if you just go to someone and say, “I want you to go take this product and enter this new market,” most likely the first response they’ll say is, “No, and let me tell you how that won’t work.” What they really want to say is, “I’m not going to commit yet to that objective until we have a chance to really sit down and explore how we’re going to do that, what your expectations are, and how we measure success.”

Then, when I work with my Italian colleagues and the Spaniard colleagues, what you find is they can’t always tell you how they’re going to get something accomplished, but they manage to get it done.

Shocking news, really, that one’s leadership team expects to have a clear strategy in place before acting, and wants the freedom to choose how to accomplish the goals they’re responsible for.

What concerns me is how few middle managers (or even executive managers) have undertaken the sort of self-examination that both McKinstry and Plumeri evidently have. In 20 years, I’ve met only a handful who embrace the power of participative communication. By the way, they’re the leaders who typically win in the marketplace.

Why don’t more organizations evaluate the communication strength of their leaders?  One reason is the perception that you can’t hold people accountable for “soft” skills. Yet, we know that there are very strong correlations between effective communication behavior and employee understanding and comprehension. So, if we want an informed, educated workforce which understands the business and their role in it, their managers will need to be the ones providing context and leadership.

Therefore, let’s evaluate communication skills among managers and come up with ways of helping those managers improve and thrive. It’s not too difficult a concept.

Share

Objectives most critical element in measurement

Tuesday, December 1st, 2009

One smart PR pro told me years ago that even the best road map is useless without a destination. Because so many communicators are struggling to understand the role of strategy in a world of fascinating tactics, it can seem like the universe is throwing map after map to us, shouting “you need this right now!”

When I step back from “being strategic” (quite a trick for someone once called, with some derision, “strategy boy”) it is no exaggeration to say, echoing the travel metaphor, that the best strategy is useless without an objective.

Strategic objectives need three things: 1) a benchmark. You need to look back to see where you’ve been. 2) A target. You need to look ahead and see where you’re going. 3) A time period. You need to articulate how long it’s going to take to get from where you were/are to where you want to be.  If you’re missing any of those things, chances are good that you don’t have a strategic objective.

The trick is, too often, we set objectives with no clear understanding of where we are, let alone where we were. That’s where research comes in. It’s right out of PR 101 — start with research before you launch a campaign — and we find lots of reason not to do the research. Sometimes it’s related to cost, sometimes to our own skillsets. We like to think of ourselves as creative geniuses, unencumbered by such trivialities. This attitude is especially prevalent in media relations, where our relationships and seat-of-the-pants skills can mean so much in a crisis; when things go right in our activities, that can reinforce the perception that PR is art, rather than science.

Of course, our “gut” is merely the application of our accumulation of experience, both in terms of time and in terms of education. We think we know what our employees, or our customers, for example, know/think/feel about our organization, when we could remove any uncertainty with some simple research.

But, I digress — the objective-setting process is even more important when considering social media.  Too many organizations are jumping in without a clear idea of what they want to accomplish.  More on this topic to follow.

What about your communication planning process? Does it start with objectives?

Share

Still stuck on AVEs

Monday, November 23rd, 2009

The continuing debate over advertising value equivalency reached the pages of the New York Times 22 Nov., with a “Soapbox” piece saying Hollywood studios are cutting ad budgets and using public relations as an alternative. One anecdote:

Disney recently went so far as to develop a computer program to help it determine how much monetary value was coming from such publicity efforts. It can quickly plug in data — “Access Hollywood” had a 30-second interview with a star of “The Middle,” a new ABC comedy — and the program spits out what that same 30 seconds would cost to buy.

AVEs are a sore spot in PR circles these days.  KD Paine, measurement maven extraordinaire, has campaigned against them for years as bogus figures that don’t quantify the value of media hits. The Institute for PR Measurement Commission (of which I am a member) officially condemned the practice in October, and PRSA has formed a blue-ribbon panel to address measurement generally — looking to a world without AVEs.

I believe there are certain circumstances where AVEs are useful — product publicity, for one, where features and benefits are the subject. But AVEs need to be net of cost, be based on actual charges, not simply “book rate,” and the publication has to be targeted to the specific business need. My example is Goodyear, the tire company. If they get a product review in Road and Track, it’s going to be relevant to their audience, include features and benefits, and in nearly all cases quite similar to an advertisement. What’s unaccounted for is the reader’s perception of value — AVEs are limited by an inability to include the weight of third party independence.

Look, notwithstanding this last paragraph, AVE is a bad metric 90 percent of the time, and there are other ways of evaluating media coverage that are better.

So, why does it appear that so many firms are stuck on this difficult metric? Well, AVEs are simple to understand. Here’s what it would have cost us to buy this time or this space — that’s a lot easier to grasp for a lot of people. There also is the pressure on PR agencies levered by their clients — “I understand you don’t like AVE, but I have to have a dollar figure to tell my CEO, so if you don’t give it to me, I’ll find someone who will.”

Still, I wish that more companies would stop using AVE. Oh, and that more people would understand that PR isn’t limited to publicity and press agentry. Perhaps the best reason not to use AVE is that it doesn’t measure the reputation work that represents most of what PR work is in business these days. For every stunt PR trick, there are months of quiet conversations with centers of influence, months of work on helping employees better understand their industries and organizations, and programs designed to help people grasp the significance of a company’s role in the community. There is more to our profession than being a low-cost replacement for marketing.

Share

Discussions you should read

Tuesday, November 17th, 2009

Several good ones:

Rich Becker — great discussion in the comments on social media concepts…

Brian Solis — Do we need to redefine “influencers?”

Chuck Hemann — What impact on social media use/adoption does organizational culture have ?

Paul Seaman — The Excellence theory says PR is about fostering relationships. Paul disagrees.

Share