Standardizing on Email Marketing Measurements
by Stephanie Jordan
When is an email campaign successful? To answer the question marketers might review click-to-conversion rate, aggregate click-through rate, revenue per mailing, profit/margin per mailing, etc. These are common performance indicators, however, many marketers are not reviewing these. According to David Daniels, vice president and research director of JupiterResearch, marketers are failing to use metrics at even a high level to gauge the effectiveness of their mailings.
But one wonders if the lack of metric use is driven by the lack of standardizing on those very metrics. Dave Lewis, vice president of market development for StrongMail Systems, reviewed Daniels' report and felt that the key takeaway was that almost 50 percent of business-to-consumer (B-to-C) and business-to-business (B-to-B) mailers are not looking at any metric on a monthly basis. "I do not know whether this metric mess, has caused people to throw up their hands and say 'I do not know what to look at, so I will do nothing'. But it is a very alarming statistic," says Lewis. "How do you judge the performance of your campaigns, regardless of how you measure campaigns, if you are not keeping your eye on the indicators?" Lewis wants to know what is driving that behavior, and feels it is something the industry should try to understand.
Lewis is a member of the Email Experience Council (EEC), an organization of global professionals driving and defining email marketing and communications practices. The EEC's new survey The State of Email Metrics & Bounce Management, published in March closely paralleled the findings in Daniel's E-Mail Marketing Measurement paper. The EEC survey project and white paper were developed under the leadership of the EEC's Deliverability Roundtable chair Deirdre Baird, president and CEO of Pivotal Veracity, and Bounce Project Committee co-chair Lewis, with participating roundtable members from Datran Media, e-Marketing Strategies, Listrak and Return Path. The EEC survey findings provide an assessment of the current state of email metrics and bounce management based on the responses from 321 Mailers and 29 Email Service Providers (ESPs) representing thousands of client-companies.
When you look at the key metrics across the industry, there is a wide variation on how the industry calculates them. The lack of consistency in calculating key performance metrics makes it impossible to establish industry benchmarks or compare results. "The EEC white paper paints an alarming picture and should serve as a wake-up call to address the inability to define, calculate, view and act on key metrics," states Baird.
When asked if the severity of the problem was a surprise, Baird replied, "Yes. The severity in respect to the lack of standardization was a surprise. However, most surprising was that two-thirds of mailers were not sure or just guessed on how the number they use for "delivered" is calculated. In a sense, "delivered" is the core email marketing metric as it is used in most other subsequent return on investment calculations and is the basis for which we compare email to other direct marketing mediums (e.g., direct mail)."
"This is one of the most significant issues facing our industry," agrees Daniels. "Primarily because it is difficult to totally comprehend the return on investment and the value of the channel. Especially, if some of the assumptions going into assessing the return are flawed to begin with. I think as an industry we need to more broadly build consensus-across the ESPs, the vendors, and the marketers themselves-to come up with a standard methodology about how we would calculate delivery." Daniels goes on to point out that once the delivery metric is agreed upon, the open, click, view can be calculated with greater confidence. "Most of the hard work will be accomplished by the delivery, because that is the common denominator; that is the number they will use to calculate all the others."
Bounce Management
The survey also reveals considerable confusion around one of the key processes effecting deliverability-bounce management. The survey confirms that currently both mailers and Email Service Providers (ESPs) are unable to maximize the effectiveness of their campaigns by maintaining clean lists, retrying temporary failures and proactively managing their marketing practices.
The survey also reveals a discrepancy between the actual and perceived value of bounce management between mailers and ESPs. The survey shows that 88 percent of ESPs and 83 percent of mailers view bounce management as having high or above average importance relative to other issues impacting campaign performance. However, ESPs assumed that only 50 percent of mailers would view it so highly.
The ISP Role
While ISPs are not responsible for sender list management practices, they do significantly impact the effectiveness of those practices through the delivery disposition data they provide. "Today that's a major problem and one that both mailers and ESPs noted in the EEC survey," comments Lewis. "ISPs and other receivers of email do not provide consistent or accurate disposition data. In some cases, they silently delete email without providing any notice of their action. While rationalized on the basis of not informing the spammer, these practices present serious list management challenges to the legitimate senders of email and undermine the reliability of email as medium for communications and commerce."
Lewis notes there are initiatives underway at the Mail Anti-Abuse Working Group (MAAWG), an industry trade group principally comprised of ISPs and other receivers, to address the consistency of email delivery notices and policies. Sender organizations, such as the Email Sender & Provider Coalition (ESPC), as well as many individual companies are engaged with the ISPs on this issue as well. "While improved ISP transparency is exceedingly important, it's essential to recognize that it will take time to adopt and implement the appropriate changes," he says. "In the meantime, senders need to make use of the bounce data they do receive from the ISPs to manage their lists. Unfortunately, one of the major findings in the EEC survey was that many mailers and ESPs aren't in a good position to do so today. They lack the visibility into their delivery data to understand the nature of their bounces and what might be causing them, and with inconsistent definitions being applied, they may be removing good records while retaining bad ones. These are issues that legitimate senders must address to better manage their lists today and to take advantage of more accurate and complete ISP data once it becomes available."
Baird agrees that ISPs should be involved in the list management practices of senders on specific issues that are of relevance to enforcing end-recipients' right to choose with whom they communicate and where the ISP is in the best (or only) position to make those preferences known. "One example of this is Spam Complaints. Since the endrecipient and ISPs are the only ones who know which recipients no longer wish to receive emails, it is incumbent upon the ISP to share these complaints with senders so that they may remove the names from their file," believes Baird.
According to Lewis, the relationship between the sender and receiver communities has vastly improved over the past five years. Yet, there remains a great deal more that needs to be done, and collaboration between the two communities is the key. "Uniform practices and standards on the sending and receipt of email should be at the top of the agenda for both," states Lewis. "But we should be realistic in our expectations. Given the enormous diversity within each community, it is unlikely that we will ever achieve complete unanimity. Yet, that should not deter us from adopting basic principles and policies for the responsible sending and receipt of email as standards for our industry, such as what a bounce code means and how it should be handled. While the two communities should work together to improve the email medium, the place to start is for each to focus on its own issues too." Lewis goes on to say that senders need to focus on standardizing metrics and definitions, as well as improve bounce management capabilities. Receivers need to focus on standardizing email delivery notices and practices and ensure consistent adherence to both.
Key Discovery
Baird notes a key finding is there is a failure to communicate effectively when discussing metrics with one another. "For core metrics-such as delivered, open rates, and click rates-there are a half-dozen or more methods commonly being used to calculate them. We might as well be speaking different languages."
As both the JupiterResearch and EEC studies reveal: email metrics and bounce management are two specific areas that have not kept pace with email's evolution. While essential to evaluating and enabling successful email programs, these critical practices vary widely among email marketers and the service providers that support them.
Email continues to be the only direct marketing channel without a standardized set of metrics. "I have to wonder if mailer activity can even be considered direct marketing without the use of constant measurement of campaigns," adds Lewis. "Everything we know about direct is that it should be measurable. It should be an accumulative experience. One that you can apply lessons learned from one campaign to another." The survey results validate the need for metrics as well as common definitions and uniform practices. As an industry, Baird believes the next steps are to identify the most useful metrics and identify new labels that are more explicit in respect to what they mean.
"Email is now ten years old," states Lewis. "It is time, maybe past time, that we get serious about defining some standard metrics by which things should be judged." SJ/TMP