Recovering Flacktitioner / Flackademic-turned measurement PRovider
I’ll be joining global measurement maven KD Paine at two important social media measurement events in Toronto this spring (if ever it comes). Buy a ticket, come on down and contribute to the dialogue:
Joe Thornley’s Social Media Measurement Roundtable – May 20.
MESH –Canada’s Web Conference — May 21 & 22.
…and again at Mass 2 Grass, the Canadian Marketing Association’s Word of Mouth Marketing conference on June 12. (Details TBD)
I’ve been sitting on this one a while. I’ve heard a wide variety of definitions of tone in my day. Some from the Canuck media analysis master and soon-to-be Ph.D. @ Cormex, some from the organization of which I am a booster and disciple: The Institute for Public Relations and their commission on measurement. And, a whole bunch from PR practitioners via blogs, events, cocktail chats and around the water cooler.
We’ve known this for moons, but it’s worth revisiting. The industry has got a serious clarity and consistency problem and so our day of aTONEment beckons. I don’t normally do this, but the problem rarely sits with those who are trained media analysts, the problem sits with PR practitioners who are moonlighting at it. In fairness to them, most have not been trained on how to analyze media content, generally, and analyze for tone specifically.
My definition of tone? Well, it’s really Andrew Laing’s at Cormex. I’ve been schooled by the master. I use it because it’s painfully simple and it’s clear by design and it works. It’s: “the (reporter’s) explicit or strongly implicit characterization of (the story’s) subject. Content in brackets added. Or, if you prefer, the IPR definition: “Content analysis factor that measures how a target audience (is likely to) feel about the client or product or topic; typically defined as positive, neutral-to-balanced, or negative.
Tone is tone is tone. It’s supposed to be (as much as possible) objective. Tone, one media analyst to the next should come out the same. Content analysis, methodologically, has checks and balances built in to test and correct for this. Called intercoder reliability.
What I’ve seen and heard in the last few months is discouraging. I’ve heard things like:
1. We got almost everything we wanted in the article–so that’s positive.
2. We got a call to action mentioned in the article–so that’s positive.
3. We got a front-page, colour photo of our product–so that’s a positive.
4. We were the only company in our industry mentioned–so that’s positive
5. We got into the publication we wanted–so that’s positive.
6. The piece included a key message–so that’s positive.
No. Those may speak to the presence of other important variables or indicators of the relative quality of the coverage, but the fact that they are present does not have any bearing, in my view, on tone. Tone is tone is tone. What’s potentially beneficial to the client cause isn’t necessarily tone.
Shameless self-promotion, perhaps, but content that PRoponent / PRomulgator readers might find amusing nonetheless.
1. A bit about measuring stakeholder relationships on the Inside PR podcast
2. I’ve been out evangelizing the gospel of research and measurement to students. One program wrote a particularly on-target summary. Bright students. Great questions.
So this measurement wonk took an Eastern Caribean cruise over the New Year. I’ve cruised many times with many different cruise lines. They are not all created equal. I knew that. You get what you pay for. I forgot that one. We admitedly cheaped out a bit given the pricey time of year.
So, come time to provide feedback (which every single crew member repeatedly urged us to do) I went to work on the feedback form. I’m encouraged that they asked for feedback. It’s a brilliant piece of strategy (or perhaps characteristic of the line’s complete keystone cop calamity approach to service) that they give you the feedback form after you’ve tipped and before you get the final bill.
The form itself was well-constructed and seem to ask all the right questions. The trouble is though (and hat tip to the Don Stacks of this world) their scale was set-up to be beneficial to them. They were asking for feedback but only let you provide positive feedback. Hello?!
Example:
Please rate us on our food: Exceeded expectation, met expectation, room for improvement.
What?! Where’s the ‘neither exceeded nor met’? Where’s the ‘did not meet’ and the (poorly worded but you get the idea) ‘did not come close to meeting expectations’? We can praise them to the high heavens, but only slam them with a flacid ‘room for improvement.’ It’s a bit like asking your loved one to tell you what they think of you and then turning up the car radio so you have to hear the answer.
…a cheesey but useful slogan I use in dozens and dozens of lunch ‘n learns, client meetings and speaking engagements to drive the much written about (Jim Grunig, KD Paine, Mark Weiner, multiple contributors to and members of the IPR measurement commission) theme about the importance of using a data driven / focused / centric approach to communications. Bringing science to the art if I can borrow a theme from the IPR.
It’s not (at least exclusively) about measurement. It’s about research. In fact, as many who do measurement for a living would say, I don’t see the distinction between the two. And when I say this in lunch ‘n learns, client meetings and speaking engagements, this is where I feel like I’m starting to loose half the room.
I say that we (the industry writ large) often think of measurement as something that comes exclusively at the end of a campaign and research as something (if it’s thought of more than strictly as a pitch hook) that is formative pre-campaign. And while unfortunatley and in reality there may be some truth to that, it’s also ture that we use one to accomplish the other. We use research to measure. Research IS measurement. Measurement IS research. They are one in the same. They are part of the same continuum. They are part of the same thought process.
While I’m not a huge fan of the RACE (research, analysis, communication, evaluation) formula, at least it urges us to think about the “R” and the “E.” The irony, however, is that the industry is often in such a race to get a pitch or a plan out the door and to get to the end of a campaign that it’s all too often heaved to the curb. We’re often so focused on telling that we’re not listening.. Research is listening. Critique of the RACE formula aside, there’s a reason why it’s in text books. It works.
Perhaps it’s largely the speed, among many other reasons (budget, expertise, fear) that sets up a situation where the industry gets bogged in the micro tactical, silo’d, episodic measurement muck. It’s difficult but necessary to pull up and out of that micro measurement muck and look at the macro and strategic. And that macro and strategic view is hardly rocket science. It’s a management by objective approach that can be applied to any discipline.
When stuck in the muck, it can be useful to look at examples of projects that have done it right. Where research has been used strategically, at the right stages and in all the right ways. The award-winning case studies on the IPR’s website, for example, are a source of inspiration.
Contribute to the industry collective. Weigh in with your perspectives on and practices in communications measurement and evaluation. Take part in the international survey of PR measurement and evaluation here. Hat tip to KD Paine for the link. Can’t wait to see these results.
Well it’s that time of year again. Cross border shopping in the U.S.? No. Frantic Christmas shopping? No. Parking lot rage? No. It’s time once again for the bi-annual (and somewhat modest in comparison to others) measurement conference in Toronto, February 14 & 15. Though the agenda’s growing a bit stale (we’ll fix that) it is an informative and insightful attend for those struggling with and new to measurement issues.
Gary Getto of VMS is the primary author of a new measurement blog (the more the merier) called The Integrated Perspective. (Hat tip to PR Week’s In Brief feed for the heads up). Congrats and good luck with it, Gary. Looking forward to keeping tabs on it.
A new measurement blog’s been drawn to my attention. Well, new to me, not new to the bloggie sphere: The PR Measurist written by Michael Young, an SVP at Access Communications. Young asks, and ultimately answers seven questions that inform something that he calls the ‘Measurement Maturity Model’. Fancy moniker and a fancy chart. Not sure I’d completely agreee with all the types and colours in the model (and what’s measurable and what’s not and I’d point out that measurement in this specific context is really editorial measurement, NOT PR measurement), but it’s a reasonably thoughtful, interesting and at least level-headed read. Enjoy.
Perhaps some Canadian and Toronto-based readers of this blog attended, as I did this morning’s breakfast event put on by CISION called: Proving PRs Value Through Measurement.
Kudos to CISION for taking something like this on. If the number of attendees was their measure of success, then they clearly met their objective. It was one of the better attended measurement events in moons. Free admission and free lunch aside, it’s a topic that the practitioner set always want to hear about.
With the exception of the line in the invite that said “understand the value and attributes of various measures such as advertising equivalanecy (yikes), something I took issue with last week, it appeared promising. Promising not so much because of the wording of the invite, as it was due to the fact that a heavy hitter from the organization formerly known as Delahaye was on the presenting hot seat.
Frankly, I’d have expected more. While there were, in fairness, some noteable nuggets, there are a few things that struck me as problematic. In no particular order…
First, they called it “proving PR’s value through measurement.” I didn’t see proof and they’re not measuring PR, they’re measuring media coverage. There’s so much more to communications measurement than media coverage. I’m confident Delhaye knows that, it’s just curious that it wasn’t acknowledged. That the presentation threw terms like net “effect” and “impact” at us worrys me. We’re talking about measuring, albeit in a reasonably sophisticated, more sophisticated than most are used to seeing anyway, but not in a new way, the quantity and quality of coverage. Full stop. Using terms like effect and impact in this context is misleading because we’re really talking about the POTENTIAL to affect and POTENTIAL impact. Eyeballs don’t equal impact. It would sure make my work life easier if it did. To measure that, you’d need (one among many examples) to take the research one step further into polling and look at linking polling data to the media content analysis. What’s weird is that I know Delhaye does this kind of work and other fancy methods like market mix modelling and they are often, rightly, held in very high regard for the work they do. So why didn’t we see some of that? This was hardly putting their best foot forward.
Second, one of the problems with the CISION version of a net effect score (though I appreciate what they are trying to do and it’s atleast commendable that they reduce the number of impressions based on the quality of the article) is that it assumes that everyone that receives a copy of a publication or everyone that had an opportunity to see the aritcle, DID see it. Obvious flaw there. Doesn’t account, for example, for the different consumption patterns of heavy vs. light readers.
Third, their position on advertising equivalency was a source of debate. I understand the point that it can be useful in some contexts if we use it not in absolutes (x$) but in relative terms (it went up 10% month over month). But, I though it could have been more clearly articulated and it’s a difficult concept to get across and a tough sell when you have an audience that–rightly or wrongly–has grown accustomed to seeing it as a no no. A friend and colleague of mine likens the use of AVEs as being like smoking. We all know we’re not supposed to do it, but many of us do.
Fourth, this idea that if a message is present, it must be positive. Not so. Rare, but a message could very well be dragged into the article and countered by an industry analyst or a critic. We saw this all the time at Cormex Research. It’s an important variable to track. That’s not accounted for in the CISION methodology.
So, again, kudos to CISION for putting this on. You can’t please everyone. It’s important that the industry keep hearing about and talking about measurement if we are to continue to sharpen our collective measurement pencils. But I worry sometimes about events that claim to talk best practices and don’t fully deliver.
Often one looks to a blog to provide constructive criticism on a particular topic, perhaps in response to a contribution that someone else has made to the discourse on that topic. But once in a while something comes along that’s as uncritiquable (is that even a word?) as a Ferrari. What’s to complain about when it’s that good. No cup holder? Well the same is true of an excellent article that the Public Relations Journal has just published, Penned by measurement mavens David Michaelson and Sandra Macleod both of Echo Research, it’s called “The Application of ‘Best Practices’ in Public Relations Measurement and Evaluation Systems.” While there’s nothing earth shatteringly new here, it is encouraging to see best practices laid out so clearly and succinctly. And while this may be old hat to some in the measurement world (at least in theory if not always in practice), it is a MUST READ for the general practitioner and practitioners-to-be.
If it ain’t broke, don’t fix it. So, sans critique, I offer up a summary of the nine best practices the authors propose below (I’d provide a link to the article were I not fearful of copywright laws):
In two main categories: 1) methods and proceedures and 2) quality and substance of findings.
Methods & Proceedures:
1. Setting clear and well-defined research objectives
2. Applying rigorous research design that meets the highest standards of research methods and ensures reliable research results
3. Providing details supporting documentation with full transparency.
Quality & Substance of Research Findings
4. Designing the research to demonstrate the effectiveness of public relations activities
5. Linking PR outputs to outcomes
6. Using findings to aid in the development of better communications programs
7. Demonstrating an impact on business outcomes
8. Being cost effective
9. Having applicability to a broad range of PR activities
So I just got an e-mail invite to an event CISION (formerly known as Bowdens in Canada) is hosting:
|
Proving PR’s Value Through Measurement |
|
This workshop focuses on measurement techniques and research methods that can help solve the many challenges faced by communication professionals today.This workshop will show you how to:
All great stuff in theory (depending on what they discuss and deliver and how) so kudos to CISION for putting it on, but are they really going to talk to us about the value of using ad value?! Let’s hope they mean they will discuss why not to use it. And I want to point out what readers of this blog have known for moons: that there is so much more to the world of PR measurement beyond measuring media coverage. Register here if you’re interested.
|
|
:
Representatives from Microsoft and Cymfony are addressing the IPR measurement conference delegation with a presentation entitled: The Next Generation of Enterprise Measurement for Enterprise. (To be clear they’re talking only about media measurement). You’d expect a company like Microsoft to have a data rich / savvy / hungry culture. And they are, clearly. So when the issue of measuring (editorial and social media coverage anyway…oh, and buzz, too, though I’m not fond of that term) came up, there was a desire to arrive at a single number that represents the quality of the coverage and to track that number over time. I appreciate the context–they are the most written about organization on the planet–and I appreciate the need to solve what Microsoft calls measurement anarchy–they had multiple editorial measurement systems in place. However, I worry about the danger in reducing something as complex as PR and scads of media content analysis data to one number. As a colleague of mine points out, a single score for media coverage (however complex the methodology used to generate it) is a bit like getting a 4.7 from the Russian ice dancing judge. Wonderful, but what does that mean? The alternative? More data and more analysis, and using the ice dancing analogy, akin to sitting down with your coach and reviewing a video of your performance and discussing it in great detail. The whole point of measurement and the data it yields is to provide extra depth and insight to communications. Now, certainly not all organizations are created equal and not all organiztions will want to (or should) measure alike, so whatever works for you and whatever floats your boat, I suppose. But a little information is a dangerous thing.
In fairness to Microsoft and Cymfony, we haven’t and aren’t likely to see behind the methodological curtain. Still, what’s not clear, and this is my hope for Microsoft, is whether or not this single number can be reversed or deconstructed to go back to more rich data that may lie behind it for more in-depth analysis. Also, what’s not yet happening, Microsoft and Cymfony admit, are attempts to link this score to other market, brand, employee and reputation research.
This from Jim Macnamara moments ago at the IPR measurement summit. Jim is a panelist on the topic of measuring integrated communications. The angle of the dangle is essentially informed by a cultural studies perspective: brilliant. It’s rarely injected into a PR context (and it’s a perspective that I’m sympathetic too having done both PR and cultural studies grad work and having written a now dated thesis on the topic). Jim’s key point is really that because how an increasingly fragmented, multitasking prosumer audience receives and processes (and subverts or re-appropriates for that matter) and is influenced by a message is a multi-faceted mash-up (advertising, traditional editorial, out of home, social media, word of mouth, etc.) and in some senses cumulative, measurement must be equally complex and integrated. “Measuring PR in isolation is intellectually myopic,” noted Jim. He’s pointed to an unfortunate reality that marketing research, advertising measurement, PR measurement and–recently and increasingly–web and social media metrics have and continue to largely exist in silos. So where do we find the corrective lenses for myopic measurement in Jim’s view? Research needs to look at a wider perspective. But considerable barriers exist not the least of which is that there is big money on the table for the various companies promoting their respective media or tools. Jim notes that communications practitioners themselves (advertising, PR, social media) will not be the drivers of the change necessary to unlock this matrix of influence and unlock the black box of proprietary methodologies / algorithyms etc that some vendors hold so closely to their chests. Rather change will be lead, Jim says, by independant research firms–some of which are already taking up the challenge. Jim suggests that we need to start looking at deep interviews, experiments, social network mapping.
All this is an extension of a path that Jim’s been on for some time having recently published a paper called: A Fork in the Road of Media and Communication Theory and Practice which esentially uses, again, cultural studies, to point out that PR practice has not kept pace with communications theories (particularly those to do with reaching and audience and the complexity of the audience) proven long ago in the cultural studies body of work.
I thought, perhaps, some PRoponent / PRomulgator readers might be interested to hear that Doug Kells, former Director of Research at Cormex, (Canada’s leading media content analysis and measurement firm) is now a free agent and looking for a measurement gig.
I have a great deal of respect for Doug. We first crossed paths in a client-vendor context. Five years back I was at Bell Canada and Doug was the lead analyst on our account. We’d later work together for a time at Cormex.
Doug’s background / skill set is fairly rare, generally, but even within the measurement crowd:
* Four years as an analyst at Bowdens (was later bought by the Observer Group-turned-CISION)
*Nine years at Cormex Research where he played a sort of hybrid role that has seen him sharpen both media content analysis skills but, perhaps more importantly, all the methodological, number crunchy, SPSS syntactical and even IT stuff that one needs in a measurement firm to take simple clips from the measurement by thud factor approach to providing methodologically sound and rigorous analysis. There aren’t many of those in Canada.
Doug will make a valuable contribution to the North American measurement scene, regardless of where he ends up. I wish him luck and I ask that if anyone on the reading end of this might like to see Doug’s resume and/or talk with him, please let me know. I’d be happy to forward his CV and he can be reached here.
Many companies undertake some form of annual employee survey. A value index or quotient of some kind. But how often is it either influenced and or executed by PR or corprorate communications staff? I suspect we’d find a mixed bag out there–as I’ve noticed among clients and practitioner colleagues–as to which department ‘owns’ the measurement of employees. Most (?) often it’s HR. One could debate, I suppose, as to whether or not that’s the right place for it. If employee measurement (and really we should be talking about much, much more than strictly surveys here) is indeed the purview of HR, then here’s hoping that HR is also being counselled by communicators and by a good external measurement consultant.
All issues that may, I imagine, surface, at an upcoming conference dedicated exclusively to Human Capital Measurement (Toronto, Nov 26, 27).
For those that haven’t already been made aware, the ‘other’ measurement event is coming up in mid November. It’s sponsored by Cision and put on in partnership with IABC.
If any of you are like me, travel and events budgets can mean making tough choices about which (and how many) conferences to attend. Why do I characterize this as the ‘other’ event? While this event looks interesting–some great speakers and some key topics–it looks to be the Chicago (Second City) to the IPR Measurement Summit’s New York. That doesn’t mean it’s not valuable, (Chicago IS a wonderful, world-class city afterall) it just means, IMHO, that to choose one might mean choosing in favour of the IPR Measurement Summit. And, the audiences are, generally, different. Huge over-generalization, but, I suspect that the IPR Measurement Summit is more about the measurementarati talking to itself (which is important) and the Cision - IABC event is targeted more at the communications practitioner writ large.
Well Toronto’s 10th Annual Communications Performance Measurement conference is just around the corner. Next Monday Sept. 10 and Tuesday Sept. 11, in fact. For those that might be interested, I’ll see if–in between my session, introducing speakers and my moderator duties–I can squeeze in some semi-live blogging on some of the main themes and discussion points. First attendee to ask about advertising equivalency gets a spit ball to the head. Should be an interesting session. Here’s the roster:
I was thankful and honoured when PR Conversations invited this Measurement PRoponent / PRomulgator to opine on the current state of the art (or science?) of measurement in Canada. While it’s a topic that could easily spawn a healthy hundred-plus-page graduate thesis, the post is a reasonably succinct, high-level, non-exhaustive horizon scan of the measurement scene in Canada, broken up into bite-sized morsels to cover the topics of: industry associations; Canadian vendors; U.S vendor spillover; the client context and the spectrum of “outs”; standards, patterns ‘n packages; the agency sphere; homegrown, thought leadership on measurement writ large; Canadian measurement events and conferences; education; awards; the wrap; and the ask. Here’s the post and here’s hoping it spurrs a number of similar states of the art-type posts and comments from other parts of the world.
Bournemouth University’s Dr. Tom Watson has posted the final report on the Study of the Priorities for Public Relations Research (PR Priorities Study - final report). The initial piloting was initiated on Dr. Watson’s DummySpit blog in April and led to the setting of 26 public relations topics. The topics were sent to a Delphi study panel (of experts) in five continents covering top academics, leading practitioners and the CEOs of PR industry bodies. After three rounds of intensive email debate over a three month period, the Top Ten PR research emerged as follows (in descending order of priority):
1) Public relations’ role in contributing to strategic decision-making, strategy development and realisation, and organisational functioning
2) The value that public relations creates for organisations through building social capital, managing key relationships and realising organisational advantage
3) The measurement and evaluation of public relations, both offline and online
4) Public relations as a fundamental management function
5) Professional skills in public relations; analysis of the industry’s need for education
6) Research into standards of performance among PR professionals; the licensing of practitioners
7) Management of corporate reputation; measurement of reputation 
Ethics in public relations
9) Integration of public relations with other communication functions; the scope of public relations practice; discipline boundaries
10) Management of relationships (while it’s not mentioned in the report, I suspect the measurement of relationships is implied here as well)
