Blog Endorsement and Ethics Rant

From what I can tell, Annie Pettit loves her job in market research methodology in the online space, just like me. She is a former VP at IPSOS and now she works for a firm called Research Now. From a recent post of hers, she thinks we all should. She has some key insights about market research and stats in the online space; I rarely see something I disagree with. Her posts aren't the technical cutting edge sort of things I usually read like Healthy Algorithms or the Gelman blog. They are a good, plain spoken highlighting of industry fundamentals. They have started to become a motivational tool for me and others.

She seems to be passionate about ethics, as she blogs about it occasionally (e.g. http://lovestats.wordpress.com/2011/02/04/top-9-lame-excuses-to-behave-unethically-mrx/). I am big on this topic too, and we need it badly in market research. I have seen someone advertise they can do a model fit to "insure [sic] the test group outperforms the control group." For me that is exhibit A. Our role in methodology, insights, analytics, statistics, or whatever you call it is not to pull enough levers until you get something that agrees with what you already decided before all the math and all the coding began. This happens all too often at my organization. I see a lot of people tend towards  "story time" as described by Andrew Gelman and Kaiser Fung  in which you do some stats procedure, maybe find an unexpected correlation or a step change, and just start weaving explanations from thin air as to the cause and call it a day. Those explanations have a special name in this field: hypotheses. You have only begun your analysis, those new hypotheses need to have methodologies constructed to test them as well. Don't leave out the highly relevant interaction term because it is hard to explain; don't leave that covariate out of your matching analysis because it makes the treatment effect look smaller; don't commission market research just to pat yourself on the back when your campaigns or product ideas look good and bury it when it suggests your efforts aren't as effective as you'd hoped. Without an ethical approach to analytics, it is just black boxes and wizardry to observers and will always have a reputation as such.


As discussed in many other places, "The Scottish author Andrew Lang once said of an unsophisticated forecaster, “He uses statistics as a drunken man uses lamp-posts…for support rather then illumination.” " Don't be that guy.

Thanks for your blog Annie; I hope we cross paths professionally one day. We need more evangelists for ethics and methodology in our space.