{"id":313,"date":"2023-10-18T11:37:22","date_gmt":"2023-10-18T15:37:22","guid":{"rendered":"https:\/\/pressbooks.bccampus.ca\/businessanalytics\/chapter\/__unknown__\/"},"modified":"2024-02-08T14:34:54","modified_gmt":"2024-02-08T19:34:54","slug":"__unknown__","status":"publish","type":"chapter","link":"https:\/\/pressbooks.bccampus.ca\/businessanalytics\/chapter\/__unknown__\/","title":{"raw":"What is p hacking?","rendered":"What is p hacking?"},"content":{"raw":"

p-hacking or data dredging<\/h1>\r\n[caption id=\"attachment_314\" align=\"aligncenter\" width=\"108\"]\"green https:\/\/xkcd.com\/882\/[\/caption]\r\n\r\n \r\n\r\nComic:\u00a0 XKCD, https:\/\/xkcd.com\/882\/<\/a>\r\n

What is p-hacking?<\/h2>\r\nRecall the iced-coffee promotion.\u00a0 Normally, the average sales per customer in our store is $4.18.\u00a0 I recently ran a promotion where the store offered half-priced iced coffees.\u00a0 My hope is that this changes customer buying habits enough to raise the average sale per customer.\u00a0 We also really want to succeed!\u00a0 Our boss was unsure about the promotion, and if it turns out not to have an effect, we're\r\n\r\nNULL HYPOTHESIS: The promotion didn't work, so sales are the same or decreased:\r\n\r\nH<\/mi>0<\/mn><\/msub>:<\/mo>\u03bc<\/mi>\u2264<\/mo>$<\/mo><\/mrow>4.18<\/mn><\/mstyle>{\"version\":\"1.1\",\"math\":\"H_0: \\mu \\leq $4.18\"}<\/annotation><\/semantics><\/math>\r\n\r\nALTERNATIVE HYPOTHESIS: The promotion worked, and average sales increased!:\r\n\r\nH<\/mi>A<\/mi><\/msub>:<\/mo>\u03bc<\/mi>><\/mo>$<\/mo><\/mrow>4.18<\/mn><\/mstyle>{\"version\":\"1.1\",\"math\":\"H_A: \\mu > $4.18\"}<\/annotation><\/semantics><\/math>\r\n\r\nRemember, we need to come up with a criteria to decide when to make a decision to implement iced-coffee afternoons across the entire chain.\u00a0 We use the typical 5% level of significance.\u00a0 That is, we will reject the null hypothesis, and conclude that there is enough evidence to show that the promotion raises the average customer bill, if p < 0.05.\r\n\r\nIn class we noted that if we use a significance level of 0.05, we would expect a result this strange\u00a0if the null hypothesis was true<\/em><\/strong> only 5% of the time.\u00a0 But what if I...\u00a0 did the experiment 20 times?\u00a0 So, every day I pick out 50 random customers, and measure their average spend.\u00a0 Even if the promotion didn't do anything, we would expect that at least one of these samples would have a sample mean that was far away enough from the true population mean of $4.18 to give a small p-value.\r\n\r\nHere's the result of my experiment (run on Excel using the norm.inv() command to pull from the correct distribution.\u00a0 I used a standard error of\r\n\r\n$<\/mo><\/mrow>0.84<\/mn><\/mrow>50<\/mn><\/msqrt><\/mfrac><\/mstyle>{\"version\":\"1.1\",\"math\":\"\\frac{$0.84}{\\sqrt{50}}\"}<\/annotation><\/semantics><\/math>\r\n\r\n, just like last week.\u00a0 If you open the file, you will rerun the randomizing, and should get different results.\r\n\r\n\"Screengrab\r\n\r\n \r\n\r\nThen I show the best result to my boss - if this screengrab, I have an impressive result of p = 0.0088, which is less than 1%, a strong result!\u00a0 Note that in 7 of the 20 experiments, my sample mean was actually lower than $4.18.\r\n

Publication Bias<\/h2>\r\nI hope that it's clear that only showing the good results to my boss would be unethical, and would not benefit our business.\u00a0 But in the world of research, it's often unusual to publish something that has a negative conclusion - \"not enough evidence to suggest anything\" is an unexciting headline.\u00a0 That means that this ends up happening more than we would like.\r\n\r\nFor more info, try googling \"reproducibility crisis\", to see how publishing only good results has gotten the field of psychology into hot water.\r\n

Data Dredging, or what about the green jellybeans?<\/h2>\r\nThis is a little bit different.\u00a0 In the comic at the top, the researcher didn't test jellybeans over and over.\u00a0 Instead, she kept on testing different hypotheses, from the same dataset.\u00a0 Much as testing the\u00a0same false hypothesis<\/em> 20 times gave us significant results, testing\u00a020 different, but all false hypotheses<\/em> might give us something that is statistically significant (ie, p < 0.05) by random chance.\u00a0 Certainly, if we test 100 or 1000 or even 10,000 hypotheses all at once, we would probably come up with something significant.\r\n

More to explore:<\/h2>\r\n