Advertising Principles - Evidence-based principles

Group Process for Evaluation

The evaluation of ad agencies and proposals rests heavily of judgments. Fortunately, much is known about how to improve judgments.

Avoid reliance on the best expert:Judgment plays a crucial role in evaluating agencies and proposals. Judgments can be improved if done independently by a number of experts (say 5 to 20 experts with different areas of expertise), typically using equal weights or medians. They can be combined by Delphi.  As popularized in the book,  TheWisdom of Crowds, combining judgments is one of the most reliable and effective ways to improve evaluations and estimations. For further evidence, see Combining Forecasts: An Application to U.S. Presidential Elections.

Use checklists and structured guides: Checklists can improve evaluation and decision-making for complex problems. It is widely accepted in areas that affect our lives, such as in flying airplanes. It would be unthinkable to fly a large commercial airline without the use of a checklist, even though the pilots have had years of experience. Consider medicine. Would you want a surgical team to operate on you if they had no checklist? For example, a 2008 study evaluated the effects of the implementation of a 19-item checklist for hospital procedures. A before/after experimental design was used for thousands of patients in eight hospitals in eight cities around the world. In the month after the operations, the checklist led to a reduction in death rates from 1.5% to 0.8% (Haynes, Alex B., et al. (2009), “A surgical checklist to reduce morbidity and mortality in a global population,” New England Journal of Medicine, 360 (January 29), 491-499.

Avoid face-to-face meetings: A combination of typically unweighted, individual judgments is superior to judgments from face-to-face meetings. For more information, see How to Make Better Forecasts and Decisions: Avoid Face-to-face Meetings. Interestingly, these experimental findings conflict with common wisdom.

Multiple Anonymous Authentic Dissent (MAAD): Experimental studies have shown that authentic dissent (that is, when people argue for what they believe) is more effective than the use of the Devil’s Advocate, which in fact has not been found to be useful for evaluation. However, those who dissent in a group pay a high price. Thus, the MAAD technique is recommended as a way for Glossary Link campaign developers to find ways to improve their campaign. This involves asking experts, including all those involved with a given proposal, to act as dissenters. Each person independently writes all potential defects that he perceives in a proposed campaign. They send their defect lists anonymously to an administrator who organizes, edits, and circulates the list to the group. Each expert assumes that each objection has merit and describes ways to deal with the objections, and then sends them (again unsigned) to the administrator. Finally, the administrator summarizes and provides the suggested improvements to the group so that they can make appropriate revisions to the campaign. This process can be repeated. To ensure group members respond in a timely fashion, the MAAD process can be conducted during a meeting by taking a time-out for individual work. For more information, here is an excerpt from Scott Armstrong’s Persuasive Advertising.


© Copyright J. Scott Armstrong and Kesten C. Green. All rights are reserved.

Contact us with any suggestions here