But in the real world, it rarely is straightforward.    First of all, hypothesis tests rarely actually match the hypothesis.   The most common hypothesis test is known as the “nil hypothsis” — you are testing that a value is exactly equal to zero.  Yes, that’s right; most hypothesis tests are testing a continuous value against an exact point value.   Anyone familiar with probability densities should recognize that the probability of a value being exactly zero is vanishingly small.    Usually, what the hypothesis really wants to test is that some value is too small to be important.    BUT, researchers seem reluctant to specify what that “too small” value is.   The way we handle that is with the p=0.05 value; if there is less than a 5% chance that this value would be seen given that the true value is 0, then we call it “small enough.”

However, this misses another really important point: much research isn’t theory and hypothesis driven like that.  To see this, we need to elucidate the theory -> hypothesis -> test pattern a bit.   Usually, once you have a theory you have to find some data that can be used to test that theory.  Often researchers will conduct a lab experiment, and carefully measure the appropriate values.    Key word here is “appropriate”; researchers will figure out what kind of data they need, and then go get it.

Exploratory analysis turns this on its head.  You start with data, and then have to figure out what it is you really have.    You are severely constrained about how you operationalize variables.   You need to make arguments for why the operationalization you choose is a good one, and those arguments can’t be ‘convenience.’  You also end up trying lots of different statistics to see which ones make sense and which ones don’t.   This introduces a minor issue: if you have a 5% significance level, then you expect 1 in 20 statistics to be erroneously significant.  If you do 100 tests, you should have approximately 5 erroneously significant results.   But there are well-known techniques for dealing with this issue; I am more interesting in thinking about the other difficulties with EDA.

One challenge is understanding how to use traditional hypothesis tests, like t-tests.  With EDA, it is really easy to do LOTS of comparisons and look for statistically significant results.  But this misses the point; you aren’t just looking for correlations, you are looking for interesting relationships that suggest larger theories.   This means that you have to spend a lot of time looking at the results of each test, and figuring out what it is potentially saying.   What is this variable really measuring, and what higher-level concepts might it represent?   Look at the actual different in means, rather than just the p-value, and ask if what this difference represents; is it actually big and meaningful, or is it small or obvious?   Think about what higher-level concepts this might actually be testing, and then see if this really is a good test.  Often, EDA works by realizing that a given test you ran might be actually testing an interesting concept, but not really.  So you try a different test that actually does test that interesting concept — in this way, you move from one test to another to another, continually getting more and more interesting.

Another challenge relates to statistical models such as regression models.  In traditional theory-driven hypothesis testing, the statistical model is usually fairly straightforward; so much so that many statisticians assume that you already know what model you want and teach from there.  However, with EDA, you are doing more exploratory work, trying out different models and seeing how well they fit and explain your data, and what you can learn from them.  Complicated decisions, such as which variables to include, and what parametric assumptions to make (logistic, probit, poisson, zero-inflated, etc.) turn into multiple model runs and an attempt to interpret each variant.   Also, very importantly, it also requires work to figure out what good / interesting dependent variables should be, and a very difficult question is often “should this be an DV or an IV?”  In the end, when doing EDA you spend a lot of time trying out different models, including different combinations of predictors, and then spend a lot of time trying to interpret each model using hypothesis tests, graphs, and table to try to understand what the model is telling you.

Finally, another challenge involves understanding what variables you have.   Even this isn’t straightforward.  Often, you can benefit from post-processing some of your data to create new variables.   For example, if you have time-series data, you can create lagged variables that can help control for certain types of variation.   Often, it is beneficial to aggregate variables by subject and use that aggregation as a predictor.   Or to take a continuous predictor, decide on a cutoff, and make it a binary predictor (or multiple cutoffs and an ordinal predictor — few, medium, lots).

Exploratory data analysis is a distinct skill from traditional hypothesis-driven statistical analysis.  And it is used with different types of data.   I encounter EDA regularly when looking at existing log data from Internet websites; the data here is extremely rich and complex, but it requires a lot of work to figure out what you can learn from it.    Another place that seems to deal with this issue frequently is ecology; they collect lots of “field data” — and they often mean actual fields — and then use EDA techniques to study it.  I’ve found (mainly through the help of Emilee Rader) that ecological statistics can be really helpful in my work of understanding behavior on the Internet.

Now, Boice’s 4 step plan is actually quite interesting:

Step 1: Automaticity.  I wrote about automaticity last week.  The basic idea is to make writing, and especially starting writing, as automatic as possible.  Much like getting in a car and driving is very automatic, Boice wants us to make sitting down to write an automatic function.  He has some specific suggestions, which I discussed last week, such as spontaneous writing and generative writing.

Step 2: Externality. Writing is not purely an internal thing that you do solely because you love it.   At least, doing it for that reason alone doesn’t lead to being very productive.   Boice strongly suggests that the best way to be productive and make writing easy is to use external pressure to get yourself to write.   There are two types of suggestions he makes.   First, give yourself a positive reward for writing; make it so that the only way to get that reward is to write.   Don’t let yourself do something enjoyable that you do everyday (like shower) until you’ve completed a minimal amount of daily writing.   Another form of this advice is to track your writing; it feels good to put down a checkmark saying “I wrote today.”  Second, create an environment that is only for writing: a writing chair / writing desk in the corner of your office, or a room where you write.   And only use that environment for writing.  That way, you quickly and easily get in the mood to write when you enter that environment.   I find this last one difficult because I move around so much; I write from home, my wife’s apartment, my office at work, and occasionally the conference room at work.  However, I am going to try to configure at least my *software* environment in this way; software such as WriteRoom can greatly help with this.

Step 3: Self-control. In order to be productive as a writer — to produce a large quantity of writing — it helps to actually enjoy the process of writing.  If you don’t enjoy it and you are frequently dreading it, then eventually you will find a way to stop doing it regularly, even if you practice the other steps.   To combat this, Boice suggests that you work on controlling the thoughts you have about writing.  Don’t let yourself talk in your head about the difficulty or dreariness of writing; replace those thoughts with more upbeat thoughts about productivity and importance.   This is straight out of modern cognitive-behavioral therapy.  Replace the bad thoughts with good thoughts.  However, it is important that the good thoughts be true — be statements you actually believe.  So don’t try to convince yourself you really enjoy writing; remind yourself about the benefits of writing and the reason you are writing.   Surprisingly enough, if you take this seriously it actually can work.

Step 4: Sociality.  Share your writing, as early as possible, with other people.   Work on making writing a more social act.   Set aside a time to write with other people.  Share your early generative writing and first drafts with others for feedback.   Provide feedback on others.  In the end, it is hard to improve as a writer without feedback, and if you only get feedback on finished products then it is hard to improve the earlier intermediate outputs.   Making writing into a social act is what allows people to not only produce more, but actually improve the quality of the writing produced.

In practice, following all four steps can be rather difficult.  It takes a lot of willpower and introspection to regularly practice step 3.  In addition to the courage to share early work with others, it requires a lot of time and effort to find people willing to play the part of the “other people” in step 4.  Self-discipline and willpower are required in great and regular abundance to practice step 2.   In the end, following Boice’s 4 step plan is hard; it is at least as hard as writing in the first place.   But, it provides concrete advice; rather than “just write” it actually provides advice that is useful.   Also, it has been proven to work well for a wide variety of academics.  I still haven’t decided if this is the process I want to use to write, or what parts of it are worth adopting.  But it is certainly worth trying.

Automaticity to me is a really interesting concept.  The idea is that you make writing “automatic” — something that you become so used to doing that you can do it regardless of your mood.   Much like driving a car, you want sitting down and writing to be easy to do because you just do it.   To me, this automaticity reminds me a lot of the concept of “flow” from  Mihály Csíkszentmihályi.   Flow is a state of being where you are fully immersed in an activity and are doing it without distraction, and with both energy and control.  Most people have experienced states of flow before; it is often called being “in the zone” or “in the groove.”

Looking back, I’ve developed some strategies on my own for developing this kind of automaticity and flow.   The one I used most was talking to myself.  I’d start trying to explain something to myself.  Frequently, if I was excited by the concept I was explaining, I’d get really in to and the explanation would just flow forth from my brain.  Slight variants of this would be trying to explain something to my dogs, or to another person.   But I found that explaining it to myself often worked just as well.  Then, of course, the challenge was remembering it so I could write it down when I was done.  Often I’d lost parts of the explanation as I tried to remember it; I didn’t want to stop the flow in order to write anything down.

Boice suggests a different approach for achieving flow.  Really, he has two suggestions.   First, practice what he calls “spontaneous writing” and what some others call “free writing”: sit down and spend 5-10 minutes (use a timer) and just write.   Whatever comes into you head, write it down.   Yup, that’s right, write down that silly strange monologue that is running through your head.  Even if it is just “I don’t know what to write; my brain doesn’t work.” that’s OK.  Just don’t stop writing until the 5-10 minutes is up.  This helps achieve a bit of automaticity, but does it achieve flow?  It is hard to tell; usually I don’t really get into it enough to really be in a state of flow.

Next, Boice suggests “generative writing.”  In generative writing, you bring a little bit of the right brain back in, but only in certain ways.   You do spontaneous writing, but on a specific topic.  You let your right brain control the direction you are heading — steer your writing toward the correct topics — but you don’t let it edit or wordsmith or worry about audience.  Yet.   By removing the majority of the editorial functions, it frees you up in much the same way that spontaneous writing does.  But, I think this actually better produces a state of flow because I am writing about something I’m really interested in and want to see written.  And also because usually I’m writing about something that has been floating around in my head for a while.

Once done with generative writing, Boice suggests that you read back through it and see what you find interesting and useful.  Did you have a good way of explaining something?  Does the writing have a good structure?  You don’t have to keep it all, and you can certainly edit it later, but it is a good way of generating a first draft, especially when stuck.

Boice’s suggestions of spontaneous writing and generative writing are his attempts to make writing “automatic.” He believes that you can teach yourself to sit down and write automatically, then it is relatively easy to be very productive in writing.   One of the biggest difficulties that writers have is getting past that “blocking” feeling; not only does that feeling cause you to not write, but it also creates anxiety that causes you to put of future writing projects.   Writing in a fairly automatic way helps to eliminate that blocking feeling and get you moving; it also, therefore, helps remove the anxiety and makes it easier to start writing projects earlier (which is a good thing).

So, next time you are feeling blocked (or even if you aren’t), try out Boice’s suggestion for making writing more automatic: start by practicing spontaneous writing and then follow it up with generative writing.

Many social sciences use a model of “rational actors” that completely think through options and make decisions consistent with their preferences.  This model of human decision making is particularly prevalent in the field of economics, where it underlies most of the results and thinking.  However, many other social scientists strongly dislike this model of decision making, arguing that it is has unrealistic expectations on humans and is demonstrably false.   They argue that the rational actor model requires humans to think through infinite streams of “what would he do” arguments, have perfect and extensive knowledge about their environment, and otherwise use complicated thinking patterns.  They also cite numerous studies that are mostly in psychology and even behavioral economics that illustrate cognitive biases and other instances where people act irrationally.

What follows is some of my thoughts about the utility and usefulness of the rational actor model of human decision making.

First, it is valuable to separate two ways that he rational actor model is used: as a normative and as a descriptive model.   As a normative model, the rational actor model illustrates one recommendation for how decisions should be made.  Many people generally consider rational decisions to be the “best” or “most correct” decisions because they take into account all of the different factors that influence the decisions and balance them appropriately.  Economists will often point out that irrational decisions open up the opportunity for arbitrage: taking advantage of the decision maker’s irrationality through repeated interactions.

While many people may (at least reluctantly) agree with the rational actor model as a normative model of decision making, many fewer people believe the rational actor model is accurate as a descriptive model.  Indeed, as a description of how decisions are made it is almost certainly false.  Most people need to be trained (often through economics courses) to be able to think through what a “rational” actor would do; they do not do so naturally.    It is unreasonable, therefore, to believe that most people actually do think through decisions using this complicated “rational” way of thinking.

However, just because they don’t necessarily think rationally doesn’t mean that they don’t act rationally.   There are many outside influences on behavior that may have shaped human behavior towards rationality without cognitively being rational.   Indeed, if the rational actor model is a good normative model, it would make sense for humans to end up figuring out ways to act rationally even without thinking rationally.  It is plausible that evolution has shaped a number of human behaviors to be rational.   When you think about it, it makes sense for evolution to shape much human behavior to be as optimal (rational?) as possible without requiring overly-costly thinking; rational behavior without the difficulty of rational thinking could easily be an optimal outcome of evolutionary processes.   Additionally, there are many equilibrium process — arbitrage, competition, etc. — that further shape behavior towards rationality.  Indeed, many behaviors are well-approximated by the rational actor model, including but not limited to many market behaviors such as price/quantity predictions, macro-scale economies, etc..

The fields of psychology and behavioral economics has done an excellent job document a wide variety of cognitive biases: patterns in human decision making that consistent deviate from the rational actor model.    It is true that the rational actor model is not a perfect theory of human behavior; however it does apply in many situations.  I think a good analog is Newton’s theory of motion.  Newton theory of motion isn’t a description of how objects decide where to go; it is simply a description of the result behavior.   Planets don’t use Newton’s equations to decide where to go next, but Newton’s equations are still effective at predicting where the planets will be tomorrow nevertheless.  This is similar to the rational actor model: the RAM doesn’t necessarily say how humans come to their decisions, but it does accurately predict what those decisions will be.

Newton’s theory of motion is also a good analog for another reason: it too is imperfect.   There are many things that Newton’s theory gets wrong, particularly at the extremes when objects are moving very fast (relativity) or are very small (quantum mechanics).  But just because it is imperfect doesn’t make it useless; it is still a very useful theory for predicting the behavior of objects in motion in many circumstances, and is still taught in school for that very reason.   For many years (centuries?) there were acknowledged flaws in Netwonian mechanics, but still it was the dominant theory because it was useful and no better theory existed.

I believe the rational actor model is very similar.  Yes, there are numerous acknowledged flaws in the theory; people do not always behavior rationally.   However, right now it is still the best and most comprehensive model of human decision making that we have.  Many scientists are striving to figure out just when and were the model works well, and when and where it should not be used.   I suspect that at some point there will be a Kuhn-ian paradigm shift where a new model of decision making will supplant the RAM as the primary / best model of human decision making, must like relativity and quantum mechanisms supplanted newtonian mechanics.  But until this new theory emerges, the rational actor model is still the best model.

(And now this is when this essay becomes particularly defensive.)  There are many critics of the rational actor model.  Indeed, I have had entire papers rejected simply because I used the rational actor model, which the reviewers believed was “wrong”.  Yes, the rational actor model has flaws, but so does any theory.   What most critics fail to do is to argue that either a) there exists a better / more useful theory of human decision making that should be used instead, or that b) not having a theory, and not making predictions, is better than using the rational actor model.  Obviously, if the predictions are completely wrong, then b) is true.   The “Rick Wash” theory of decision making (where I assume that everyone would make the same decision that I think I would make in that situation) is an incorrect theory of behavior, and probably leads to so many incorrect predictions that not having a theory is better than using the “Rick Wash” theory.  However, the rational actor model has proven to be quite useful in many situations.  It has been immensely useful in understanding the behavior of markets, in understanding political relationships and behavior between and within nation-states, and in understanding behavior within organizations.  Much of the advise used to make major decisions for our economy is based on predictions from the rational actor model.

As a theory of human decision making, the rational actor model is certainly not perfect.  It does not explain how decisions are made, and it doesn’t accurately describe all human decisions.  However, it is a very useful model of behavior because it provides strong and simple guidance on how to think through decisions in a way that provides reasonable and relatively accurate predictions about complicated human behaviors.

Most of the time I (and other researchers like me) look at individual-level outcomes within the site.   Do individual users increase their contributions to the site?   Do individual users participate more by leaving comments or rating items?  Most of the outcomes I identified in a previous post fit in this category.  This is the easiest category of outcomes to study, since within-site outcomes are easy to measure, and there are usually lots of individuals using the site.

A second type of outcome is group-level or system-level outcomes within the site.  These are outcomes that concern group/system level properties.  For example, we could ask the question “is there a sufficient number of users contributing for the site to be self-sustaining?” or “even though some users increase and others decrease their contributions, is the total quantity of information on the site increasing?”   My recent paper on minimum thresholds looks at this second question.   These questions are different because they look at the site as more than simply an aggregate of individuals; they assume that there exist group-level properties of interest.  Often looking at these outcomes involve looking at tradeoffs across individuals, such as the “some people decrease contributions but others increase” tradeoff in the minumum threshold research.   These types of outcomes are hard to study because each social media system is N=1; the system can only have a single outcome.  I have obviously done some work on these types of outcomes, and I intend to do more in the future.  I think economic modeling is actually a powerful tool for this because it can look at group-level properties and tradeoffs.

A third type of outcome is the individual-level spillover outcome.  Often people end up changing other parts of their life as a result of their use of a social media system.   The field of communications calls these “media effects.”  For example, people can feel more connected and have more friends because of their Facebook use.  Spillover effects don’t necessarily exist, but they can be an excellent reason to study social media systems when they do exist.  However, it is generally quite difficult to study spillover effects; simply having access to data from the social media system is not sufficient to study spillover effects.  You must also have data from the individual users about life outside of the social media system.

Finallly, the fourth type of outcome is a group-level spillover outcome.  These are outcomes that result to larger social structures as a result of social media use by the individuals in the group.  For example, a number of business have adopted various social media systems (IBM has gone full out, producing their own social networking system Beehive, their own social bookmarking system Dogear, and their own microblogging system).  These businesses hope that these systems have enterprise-level spillover effects.  (They also hope these effects will be generally positive for the business, like making people more productive, or encouraging innovation through collaboration.)  Unfortunately, these are the hardest types of effects to measure because they have all the difficulties of both group-level effets and spillover effects. Often research into these effects are case studies that follow how one particular system has impacted a given organization.

When thinking through the various ways that social media systems can have an impact, this taxonomy of effects might be useful.

I’m not sure I agree.  Admittedly, I’ve done my share of large-scale data analysis and statistics.  I think the advent of advanced computational tools is allowing us to do things that we couldn’t do otherwise.   But I think being a “data scientist” scientist isn’t enough.   Theory is what drives the questions that we ask, and what allows us to put our findings into perspective.  Yes, much like Google, you can tell which color of blue works best for links with a large-scale experiment.   But do you really want to have to do that experiment every time you deploy a web page?

Basically, theory speeds up research.   You probably can all the research you with without having a strong theoretical background.   But if you do understand the theory, you can get a LOT more done, and get it done faster.  Having good theory-driven questions allows you to focus on only the data and statistics that are most important.  In Emilee Rader and my work on del.icio.us, it wasn’t till we had the theory-driven question about how tags are created that our quantitative analysis started to produce really interesting results.  Having theory allows you to focus on the results that are most interesting, and understand why they are most interesting.   And having theory allows you to not re-do analyses that are almost certainly going to come up the same as expected.

Being a data scientist will allow you to get research done.  But if you want to get more research done, learn theory also.