The Hawthorne Effect is a term coined to explain inconclusive results from a set of studies performed at Western Electric Company’s Hawthorne Works on worker productivity from the 1920s and 30s. Essentially, researchers were confused with the productivity results from two specific parts of the study—changes in illumination levels and worker break time—which improved productivity only during the study. Workers knew they were being studied, thus improved productivity regardless of the changes implemented by the researchers. The Hawthorne Effect is used to describe positive results from research as influenced by the workers, not by the actual independent variables studied. Researchers today now work to reduce this effect through a number of ways, but it is still a tricky process.

The Big Data movement in conjunction with the declining cost of storage and increasing fidelity of data has business intelligence professionals poring over corporate data. They are looking for the next big adjustment they can make to the business that will give them some kind of competitive advantage with an opportunity to earn economic rents. But if it was just that simple, wouldn’t everyone have a functional big data analytics program?

The Data Center, by Tu Holmes

The Data Center, by Tu Holmes

With no scientific backing or official research of my own, I would suggest that the Hawthorne Effect could apply to big data analytics as well. It’s a bit of a stretch to apply this, but run with me for a second here. Let’s take two very prominent examples. The first comes from a study of the orientation of toilet paper rolls in homes—the old roll over or roll under debate. Based on the book The First Really Important Survey of American Habits by Barry Sinrod and Mel Poretz, they found that 60% of respondents that earned $100K or more prefer the paper-over-the-roll method and 73% of respondents that earned $40K or less prefer the paper-under-the-roll method (salary figures adjusted for 2014 inflation). If someone who earns less than $40K/yr decides to swap their toilet paper orientation, it doesn’t guarantee them another $60K in annual salary. But if enough people do it, the results change and you have to evolve your study of the data to account for new patterns that may arise.

Similarly, if you find a town with a median income of $100,000 and move there, it doesn’t guarantee that you will improve your take-home pay.

Data analysis looks into the past to derive patterns that we can use to predict the future. If we change our behavior based on the analysis, the models must be altered to accommodate the change. Essentially, applying the Hawthorne Effect, metrics will improve with changes, but if you don’t alter the analysis to take into account the change (this starts to add some game theory into the mix), you may end up in a situation where you have a short term improvement that is not sustained or even reverts over time.

This post originally appeared on BrandenWilliams.com.