Gilbane Digital Content Conference: Measuring Success is Complicated

Nov 29, 2017


For all the talk about data and measurement at the Gilbane Digital Content Conference, it’s not always easy to understand what to do with the numbers once you have them. Lynette Chen, MaassMedia senior consultant, tackled what it means to try and measure success at “the edge of innovation” in her session, relying on a Guardian case study.

The Guardian Mobile Innovation Lab, is a small interdisciplinary team of editors, reporters, product managers, engineers, and more, who concentrate on creating and deploying new tools. MaassMedia helped the Guardian Mobile Innovation Lab develop an app that sent notifications to subscribers’ lock screens with leaderboards, quizzes, and real-time media alerts about the Olympics.

The challenging part turned out to be measuring the success of this two week experiment. The group set out to evaluate KPIs within a metrics framework. But one of the most vexing questions turned out to be a relatively simple one: When a user dismisses the notification, is it a negative interaction or a positive one?

Originally, the team assumed it was a negative interaction. And when they saw a high rate of dismisses they thought this experiment might have been a failure. But when they compared those numbers to the number of people who were willing to sign-up again for similar alerts, the team found a discrpency. About 70% of people were willing to re-enroll, even though they had frequently dismissed the notifications. This prompted the team to go back and add more questions to their survey and found out that people weren’t dismissing notifications because they found them annoying or irrelevant, but because they were now informed and didn’t need more information. In other words, the notifications were useful. Once they figured this out, the evaluation of the experiment drastically changed.

Chen offered a few takeaways from the experiment that could help your team during your next innovative experiment:

  • Create a framework around project goals
  • Build upon that framework over time
  • Be flexible, persistent, and inclusive during analysis and interpretation
  • Re-evaluate goals or definitions of success if necessary

Related Articles

The Gilbane Digital Content Conference kicked off with a deep dive into the customer psyche with back-to-back keynote presentations. One thing is clear: the future is here and it knows what you're thinking and doing.
Adaptive content. You probably hear those words a lot, but like a lot of buzzwords, you may still be struggling to understand what they mean. Juhee Garg, Senior Product Manager, Technical Communications Products, Adobe, was on hand to explain to the keynote audience at Gilbane Digital Content Conference.
Everyone has heard the cliché, "every company is a technology company". While we know that's not true, it is true that every company is powered by technology. It used to be that an internet connection and a Microsoft office subscription was good enough for small companies, and for larger companies, same thing but add some servers. Now, though, with the influx of data that needs to be managed, a solid IT infrastructure is more than necessary.