CIS 434 Module 8 Metrics and Evaluation Example

Proving It Worked: A Metrics and Evaluation Plan That Uses Holdout Tests Instead of Trusting the Dashboard

Student Name

Business Administration Program, Aspen University

CIS 434: Internet Marketing

Instructor Name

Month Day, Year

Proving It Worked: A Metrics and Evaluation Plan That Uses Holdout Tests Instead of Trusting the Dashboard

Over seven modules, Rattlesnake Creek Fly Shop has framed an opportunity among traveling anglers, chosen a trip-ready strategy, designed the experience and interface, combined online and offline levers and planned the work. The last stage asks whether any of it is working and which parts deserve more money next year. This paper sets the measures, targets and tests. The shop and every figure are fictional and written for this course.

Objectives

The program has three objectives for its first season, each traced to the strategy. First, 120 trips booked by out-of-state anglers by April 30, up from 75 last year. Second, $180,000 of kit sales across the website, store and expos, up from about $110,000 of fly sales online last year. Third, a 60% return rate among guided clients, the assumption behind the customer value estimated in Module 2, up from about 48%.

Measures by Stage

Why the Dashboard Is Not Enough

The website, e-mail tool and search account each report results, and each will claim credit. Li and Kannan (2014) built a model of how customers move through several online channels, such as search, display, e-mail and referral sites, before buying from a hospitality firm, and tested its predictions in a field experiment. They found that the common practice of giving all credit to the last click before a purchase misstated what each channel contributed: some channels that rarely received the last click played an important role earlier in the journey, while others that often received it would have been followed by the purchase anyway. Simple reports therefore send money to the wrong places.

Lewis and Rao (2015) went further. Pooling twenty-five big field tests of web display campaigns run for large retailers and brokerages, they showed that the sales of individual customers vary so much on their own that even experiments with millions of people often could not estimate an ad campaign's return precisely. A campaign that was clearly profitable and one that lost money could produce data that looked almost the same. If large firms with huge samples struggle to measure returns, a small shop reading its dashboard cannot hope to.

Making Tests Practical

Johnson et al. (2017) proposed ghost ads, a method for advertising experiments in which the system records the ads that members of a control group would have seen had they been in the treatment group. Comparing exposed customers with their counterparts who would have been exposed removes much of the noise that comes from people who would never have seen the ad, making experiments more precise and far cheaper because the control group does not have to be shown unrelated ads. The broader lesson is that a well-designed comparison group matters more than the size of the sample.

The shop cannot run ghost ads, but it can apply the principle: compare like with like, assigned at random where possible. Before each test begins, the coordinator will write down in one sentence what result would count as success, so that a disappointing number cannot be explained away afterward. Results will be kept in a single spreadsheet with one row per week, holding bookings, kit sales, river report visits, weather notes and any river closures, so that the owner can see at a glance whether a change in the numbers lines up with a change in the program or with something on the river.

Three Holdout Tests

The winter note test is the strongest because assignment is random and the outcome, a booking, is easy to observe. With only 200 clients, it can detect a large effect but not a small one, which is acceptable: a note that adds only a few bookings is not worth the guides' time anyway. The search test relies on comparing weeks, which can be distorted by weather or river conditions, so the shop will record each week's flows and closures. The expo comparison is the weakest, because cities differ in many ways, and the shop will treat it as a guide, not proof. Repeating it for a second winter, perhaps swapping which city gets a booth, would make the answer more trustworthy. None of the three tests needs special software; each needs only a list, a coin and the discipline to leave the holdout group alone until the season ends.

What the Results Will Decide

Conclusion

The plan measures the program at each stage of the customer relationship against specific targets, and it refuses to let software reports assign credit on their own. Li and Kannan show that last-click reports misstate channel value, Lewis and Rao show how hard precise measurement is even for large firms, and Johnson, Lewis and Nubbemeyer show that careful comparison groups make tests workable. Three small holdout tests will tell Rattlesnake Creek which parts of its internet marketing to keep, grow or drop, completing the seven-stage framework.

References

Johnson, G. A., Lewis, R. A., & Nubbemeyer, E. I. (2017). Ghost ads: Improving the economics of measuring online ad effectiveness. Journal of Marketing Research, 54(6), 867-884. https://doi.org/10.1509/jmr.15.0297

Lewis, R. A., & Rao, J. M. (2015). The unfavorable economics of measuring the returns to advertising. The Quarterly Journal of Economics, 130(4), 1941-1973. https://doi.org/10.1093/qje/qjv023

Li, H., & Kannan, P. K. (2014). Attributing conversions in a multichannel online marketing environment: An empirical model and a field experiment. Journal of Marketing Research, 51(1), 40-56. https://doi.org/10.1509/jmr.13.0050