If you’ve ever watched Last Week Tonight, you know how good John Oliver is at making his audience feel outraged about a topic while simultaneously making them laugh. His segment on Civil Forfeitures is worth watching for information on a pretty concerning practice as much as it’s worth seeing him yell at a picture of Hugh Laurie for a solid minute.
In season three, Oliver spent an episode discussing scientific studies and the problematic reporting of findings, especially on popular news programs. He addresses the pressure scientists feel to uncover significant findings to keep their jobs and the tendency of the media to selectively broadcast results that appear interesting but may lack verifiable evidence.
The takeaways from the segment can be applied to PR measurement strategies. Despite the quality work many professionals are conducting to prove the value of their campaigns, inaccurate tactics still undermine the integrity of this data.
Oliver’s emphasis on the inequality of some studies in comparison to others, the need for verifiable results, and the illusion of viable results teach important lessons about conducting quality measurement.
Not all scientific studies are equal
At the outset, Oliver states that findings from scientific studies do not all carry the same weight and credibility. Some studies appear in less legitimate journals, are performed with a bias, or are tweaked with misleading headlines or summaries to provoke greater interest.
Much like these studies, not all measurement strategies are created equal. Some measurement activities have been proven to be outdated and ineffective at delivering data that allows businesses to make informed decisions.
Particularly, PR measurement professionals have long lamented AVEs (advertising value equivalencies). AVEs fail to offer an accurate standard for ad value, but clients and managers continue to request them because of their ability to generate a reportable number.
In reality, reliable measurement efforts, such as surveys, provide accurate results that allow professionals to make insightful decisions. The inconsistent nature of AVEs cannot offer the same quality of information as proven measurement activities.
Results need to be verified
Oliver points out that scientific studies require specific steps, such as peer review and human trials, to prove the validity of their findings. Additionally, a duplicate study needs to be performed by another team of scientists to ensure the results are replicable, thereby proving their accuracy.
Similarly, measurement requires some verification to confirm the data, especially data gathered by automated tools. Although tools like Google Analytics and social media analytics produce numbers that track important metrics, measurement activities should involve some degree of human review, as tools can make mistakes or provide numbers that can appear misleading without human evaluation.
Data produced by your email server, for example, may be showing a spike in the number of subscribers, but if these numbers seem abnormally high, they should be further investigated. Digging into the data and reviewing some of the subscribers could reveal an issue, such as sign-ups from spam emails or repeated subscriptions from bots.
Although it is tempting to believe that your numbers are organically positive, verification of your data through human involvement is required to confirm your results.
The illusion of viable results
P-hacking, the process of altering variables in an attempt to produce statistically significant results, undermines the integrity of studies and invalidates results. Despite the unethical nature of this practice, Oliver states that scientists p-hack data to produce results worthy of publication, as most scientists have job requirements to publish a certain number of times.
There are similar measurement practices that give the illusion of viable results, without actually proving anything of value. In particular, relying on vanity metrics without linking the activity to business objectives fails to benefit your organization. Accumulating a large number of followers on social media or a lot of likes on a blog posts seems significant, but these numbers don’t necessarily benefit your organization’s bottom line.
Linking customer activities to business goals leads to actual value. Laying out a plan that leads to conversions for your business avoids relying on misleading vanity metrics.
As Oliver wrapped up his segment, he stated that problematic reporting of scientific studies risks undermining people’s trust in science, and there is a similar concern with measurement. Recognizing proper measurement tools and practices, verifying results, and ensuring your efforts are linked with business goals, however, promotes quality measurement.