Skip to content

Boom Blox

I haven’t been updating this blog much in the last couple of weeks because I’ve been doing the final writing push for my dissertation.   After spending 12+ hours a day writing the dissertation, it is hard to sit down and write more for a blog.  But now that is done, and Emilee and I took the whole weekend off of work.  And we spent most of the weekend playing the new Boom Blox Bash Party game for the Nintendo Wii.

One of the interesting things about this new game is that it supports user-contributed content: anyone can use the built-in level creator to create new levels and upload them to a central content server.  You can also download and play levels that others have created.  As expected, I found a few really fun levels and a bunch of not-so-good levels.  Fortunately, the Boom Blox online system allows users to rate levels with 1-5 stars, and you can sort the levels by rating when looking for new levels.

This is a great example of how this user-contributed content is becoming pervasive.  Even games like Boom Blox have a user-contributed content portion of the game.  And I think that this feature could have benefited from hiring someone who studies incentive-centered design.  There is a basic contribution question here: why should users contribute levels?  And more importantly, what types of levels should users contribute?  So far, the creators of the game have been rewarding complex Rube Goldberg type levels by highlighting them on YouTube.  It would be interesting to think about different ways that the game can reward other types of levels; for example, it could pick the highest rated levels of each type currently in the game and make them available as a downloadable “level pack.”  This might encourage users to create interesting new levels to extend the game’s current gameplay.

Another ICD issue is the rating system.  Right now, after playing an online level, the user is asked to rate it with 1-5 stars.   But, why should the user provide a rating?   And, more importantly, what metric should he or she employ?  A level might get 5 stars on “cool” but 1 on “playability”.  By thinking about what behavior you want people to do, you can design the system to elicit that behavior.  For example, you might want people to rate levels based on how much fun they are to play over and over. Do elicit that type of rating, the designers might allow people to save levels from online (which is currently supported) and automatically sort them by the rating I gave it by default.   Levels I give 5 stars are shown first, then 4 stars and on down.  This way, levels I have an incentive to rate highly the levels I want to play again (to make them easy to find), and rate poorly the levels I don’t.

This is actually an example of the side effect mechanism I’ve talked about before.  Users have a private reason to rate levels — to make it easier to find the ones they want to play again.   And the incentive to rate levels is aligned with the goals of the consumers, who want to use the ratings to know what levels are best to play.

Tagged
2009 08 17 rwash Real World Observations Comments (0) Permalink

Facebook uses a Minimum Threshold

I recently blogged about minimum threshold mechanisms: set a minimum threshold for contribution and exclude anyone from using the system who doesn’t meet that threshold.  I recently encountered a great example of a popular social media system using a minimum threshold to encourage contribution: Facebook.

In order to use Facebook, a new user must do three things: 1) sign up for an account, 2) join a network, and 3) contribute a list of her friends (her social network).  Users who don’t do all of these don’t receive access to any of the information on Facebook.  Notice #3 (and to some extent #2); it includes contributing information to the system that benefits others.   Facebook doesn’t have lurkers in the classic sense of someone who doesn’t contribute anything.  They decided that it was worthwhile to lose the true lurkers in exchange for the increased contributions.   By knowing everyone’s social network, Facebook can offer additional services (like strong privacy controls and better friend suggestions) that they wouldn’t be able to offer if the social network information was less complete.   This is a great example of the minimum threshold in action; you gain a little bit of information from everyone at the expense of losing the lurkers.   For Facebook, they figured out a “little bit” of information that allows them to really make their system valuable.

If you are skeptical, compare Facebook with Twitter.   Twitter does not have a minimum threshold of contribution.   There are true lurkers on Twitter; many people read twitter without accounts.  These people have never contributed to twitter but still use it.  Twitter values these lurkers and as such does not require a minimum contribution like Facebook does.

And sure enough, a number of the predictions of my minimum threshold paper have come true on Facebook.  A number of people have chosen to join Facebook, but only contribute the minimum.  I’ve seen a number of users who are my “friends” who never contribute pictures or status messages; all they do is contribute their social network information so they can get access to the information of others.   Also, as Facebook has gotten larger, more people who would have preferred to lurk have decided to contribute the minimum threshold.   Larger systems provide more value, and hence more people are willing to make that minimum contribution in order to join.

The Facebook example also illustrates a useful point for using the minimum threshold mechanism: everyone who remains in the system will have contributed at least the minimum.  If the threshold is chosen carefully, the system can take advantage of the knowledge that everyone has contributed this information.   On Facebook, all users have to contribute social network information.  Facebook is then designed to take advantage of the fact that all Facebook users have provided their social network information to the system; many features such as the privacy controls assume the existence of contributed social network information.  This well-chosen threshold enables many useful features.

Tagged
2009 07 20 rwash Real World Observations Comments (0) Permalink

Bootstrapping Problem in Social Media

The first problem that any social media system faces is how to get the first few contributions from users.  This is known as the bootstrapping problem.  The basic problem is that each user gets value from the system that depends on how much useful information is currently in the system; but when the system is first created there isn’t any information contained therein, and therefore no one will want to use it.

Technically, this is an instance of a positive network externality; each user that joins the system makes the system more valuable to everyone else, but that user don’t take that value into account when deciding if he or she should join.  In this way, social media systems share this property with many other information systems.  Who had the first fax machine?

Bootstrapping problems are particularly difficult to deal with precisely because there isn’t much to work with; since very few users are yet on the system there isn’t much value that can be thrown at users to motivate them to join.  In general, there are a couple of different strategies that might work to overcome this problem:

  1. Personal Benefits.   Give users a personal reason to use the system that doesn’t depend on other users.   I have discussed the example of delicious.com and its use of personal benefits and side effects for motivating users to contribute.  This side-steps the network externality; however providing both personal and network benefits adds that much more complexity to the system.
  2. Setting Expectations.   Users join the site not only because of who is currently on the site, but who they believe will be on the site in the future.   Finding some way to set the expectations of a number of users that the site will be worthwhile can sometimes work.   If enough people believe that the site will have a lot of users, then indeed these expectations will be fulfilled and it will have a lot of users.  Major companies like Google, Apple, Yahoo!, and Microsoft often use marketing and PR to set these kinds of expectations.
  3. Paying High-value Users.  Paying users is one way to get them to use the site.  Since they don’t have the network benefits yet, you can supplement by providing some external source of value at least until the site is self-sustaining.   However, all users of the system are not the same; some people are worth a lot more than others.   For example, recently Twitter benefited a lot when both Aston Kutcher and Oprah Winfrey joined and publicized the site.  If you are going to pay users, try to find the high-network-value users that cause lots of people to want to join.

None of the solutions I present here are all that good.  They each have their place.   But the bootstrapping problem is a open problem in social media (and often information technology in general) because there are many situations where these won’t work, or won’t work very well.

All social media systems need some way to get started. The bootstrapping problem describes just one thing that makes getting started difficult.

One interesting note is that the bootstrapping problem is fundamentally a strategic problem.  It is a case where each users’ decision affects many other users, and vice versa.  If I decide to not join and wait and see who else joins, then my absence from the system can influence others to not join.   Likewise, if I do join, then that decision can induce others to also join.  These types of strategic problems are often particularly hard to reason about precicely because of these complex interdependencies between user decisions.

2009 07 17 rwash Big Thoughts Comments (0) Permalink

Data as Science

Recently there has been speculation that the rise in computation power will put us scientists out of business, or at least seriously change our business.   Chris Anderson has probably the most extreme view with his End of Theory article.  A more reasonable approach is the Risk of the Data Scientist by Nathan Yau.  The basic argument in both cases is that with modern computation and access to extremely large datasets, it is possible to computationally understand the underlying relationships.  Chris Anderson argues that this is pretty much all you need to do science; traditional theory building and testing isn’t necessary.  Nathan Yau doesn’t go that far, but he does argue that the hot new job in science will be the kind of person who can do this.

I’m not sure I agree.  Admittedly, I’ve done my share of large-scale data analysis and statistics.  I think the advent of advanced computational tools is allowing us to do things that we couldn’t do otherwise.   But I think being a “data scientist” scientist isn’t enough.   Theory is what drives the questions that we ask, and what allows us to put our findings into perspective.  Yes, much like Google, you can tell which color of blue works best for links with a large-scale experiment.   But do you really want to have to do that experiment every time you deploy a web page?

Basically, theory speeds up research.   You probably can all the research you with without having a strong theoretical background.   But if you do understand the theory, you can get a LOT more done, and get it done faster.  Having good theory-driven questions allows you to focus on only the data and statistics that are most important.  In Emilee Rader and my work on del.icio.us, it wasn’t till we had the theory-driven question about how tags are created that our quantitative analysis started to produce really interesting results.  Having theory allows you to focus on the results that are most interesting, and understand why they are most interesting.   And having theory allows you to not re-do analyses that are almost certainly going to come up the same as expected.

Being a data scientist will allow you to get research done.  But if you want to get more research done, learn theory also.

2009 07 13 rwash Big Thoughts
Methodology Notes Comments (0) Permalink

End-User Innovation

Twitter is a fascinating service. To many people, it is not clear what exactly it is supposed to be used for.   Chats / conversations?   There are better services for that, like email, IM, and discussion boards.   Blogging?   There is better software for that too.   Twitter seems to be something new that can’t easily be put into existing bins, at least according to its advocates.

To me, one of the most interesting features of twitter was mentioned briefly by Twitter co-founder Biz Stone in his Freakonomics interview.  He says

Many of the features we have launched were created by users including @replies, and there are more to come.

Twitter has developed a culture of end-user innovation.   This is when the users of a social media system find innovative new uses for the system itself.  Those new uses can then be developed into explicit features, or they can remain implicit.  End-user innovation is extremely valuable because it allows users to customize the experience of using the system to make it more useful.

But from the point of view of designing this type of social media, what kind of design features can encourage end-users to come up with innovative uses of the system?  This is the incentive version of the this problem; how can we design a system that encourages end-user innovation?

The most interesting part of this problem is that we do not know ahead-of-time what innovations the end-users will come up with, but we still want to encourage them.  In many ways, this is similar to the user-contributed content problem; we don’t know what content users will contribute but we still want to encourage contribution.   But innovations have slightly different properties.   They are often patterns of contributions that others can pick up also.   For example, on twtitter, using @username to reply, or using RT to mean re-tweet, and even using #hash-tags are just patterns in contributions that some users innovated that have now been coded into the system.  Innovations are also less explicit; there is no place for me to type in my innovation like there is place for me to type in my content.  I just have to do it and convince others to also.

We know some things about encouraging innovation from the economics literature.  The major way we encourage innovation is the various forms of intellectual property.  In particular, innovations are usually encouraged through patents: monopoly power over user of the innovation for a fixed period of time so that the creator can make lots of money off his innovation.   That doesn’t work so well in social media; giving the person who came up with @replies monopoly power to control who can use them might stroke his or her ego, but I don’t think it is a good way to encourage people to innovate.  It also strongly discourages adoption of innovations and derivative innovations, and discourages the system from codifying the innovation.

Another way that might work better is prize mechanisms.   The classic example is the Ansari X-Prize: the $10 million dollar prize for the first private team to produce a working spaceship.   A number of teams competed to win the prize and a large amount of innovation resulted.   Similar prizes could be instituted in social media; for example the most innovative use of a tweet; best way to organize groups of individuals on flickr, etc.  If you can come up with measurable goals, prizes might be a workable way to encourage innovation in social media systems.

But, there are still a lot of open questions in this area.   What kinds of system features discourage innovation and force people into a single way of using the system?   What kinds of features encourage innovation.  How do you balance flexibility (which is important for innovation) and supporting specific uses?  When is innovation harmful to a social media system?  How do you set goals and prizes for innovation?

2009 07 10 rwash Big Thoughts Comments (0) Permalink

Why Points Work

I previously wrote about an interesting system of awarding “points” for contributions that IBM introduced into its social networking system.  Despite explicitly not being worth anything, the points turned out to be a strong motivation to contribute more information.  Unfortunately, the paper that describes this doesn’t really give any theoretical reasons why points might work well as motivators.   Which is OK, to an extend.   What theory really does is it helps us to extend our knowledge to as-yet-untried circumstances.   The paper can’t really tell us what would happen if we change the definition of “Busy Bee” from 500 to 700 points, or if we removed or expanded the leader board.    Theory is what allows us to make intelligent guesses about these design options.

I have three ideas why these points might work as motivators.   First, people might directly value having points.   Even though the points aren’t externally worth anything, people might be conditioned to believe points are valuable in their own right, and be willing to work for them.   I don’t think this theory explains much of what was seen; it doesn’t account for why achieving status categories is important for example.   And I just don’t believe that people are so naive that they are willing to work for worthless compensation.

Second, points can serve as a relative measure for social comparison.  People naturally compare themselves to other similar people, and are willing to work to increase or decrease their relative standing.  One of the things about social comparison is that there has to be some way to evaluate and compare people.  In real life, we usually know enough about other people that comparison isn’t hard.  But online, we have very little information to use for comparison.  Points are a good way to compare across people, and earning more points can make us look better. Social comparison has been extensively studied in the social psychology literature.  We often want to move up when we are feeling inferior for some reason, and earning more points can help with that.  Likewise, if we are content with our current social status, then we won’t feel a strong motivation to earn points, and the points won’t work as a motivator.

Third, earning points can work as a signal to others.   Signaling is a theory from economics that describes how it makes sense for people to work hard to earn something as worthless as points because it helps set them apart from other people.  Signaling works when there are important but difficult-to-observe differences between people.  For IBM Beehive,  people have different willingness to collaborate and cooperate with others to get work done.   This difference is important to the company since both collaboration and cooperation are important for maximizing productivity.  The other thing that is necessary for signaling to work is that this difference has to result in a difference in how easy it is to earn points.  Highly social people find collaborating easy, and also find commenting on Beehive easy; anti-social employees don’t particularly like to do either.

When these conditions hold, then the relative number of points actually has some meaning.  People who are very social will decide to earn lots of points to signal their sociability and skill at collaboration and cooperation.  People who are less social will find it more difficult to earn a similar number of points, and therefore be less willing to do so.  People will self-select into the high-points-earners and the low-points-earners based on how difficult it is to earn points.   Then, once this happens, the points gain meaning.  If you have a lot of points, then you must be one of the people who is very social and good at collaborating.  And since the points have a meaning that is related to something the company cares about, it is worthwhile to earn points in order to signal that you are a good employee.  Points can help people to signal, or set themselves apart from others.  This implies that only the “good” employees (for some definition of “good”) are going to be motivated by points, though.

One interesting thing here is that both of the last two theories only work for subsets of the population; social comparison only works for people who want to increase their social standing, and signaling by definition only works for people who find earning points easy. Points are not a general purpose motivator.  But for the people that they do influence, they can be a powerful motivator.

R Farzan, JM DiMicco, DR Millen, B Brownholtz, W Geyer, C Dugan. (2008) “Results from Deploying a Participation Incentive Mechanism within the Enterprise.” Full paper, Conference on Human Factors in Computing Systems (CHI 2008), Florence, Italy, April 2008.

2009 07 08 rwash Research Pointers Comments (0) Permalink

Points as Motivators

Unsurprisingly, if you give people a reward for contributing to a user-contributed content system, users repond by increasing their contributions.  But how much of a reward is really needed to induce contributions?  The social software research team at IBM tried one of the simplest reward systems: they awarded people “points” for contributing to their internal social networking website.   You can earn 5 points for contributing a photo, 15 points for commenting on someone else’s status, and 100 points for filling in information on your profile page.

That’s it.   Points couldn’t be traded for prizes.   Points didn’t earn you promotions or good end-of-the-year reviews.  Points were just that: points.  Most economists would probably say that the points will mostly be ignored; since they can’t be used to get something valuable, no one should care about getting them.

Well, the folks at IBM did an experiment, and found that the points indeed caused people to increase their contributions.  But only to a certain degree.   The system assigned people into categories based on the total number of points they had earned.  Earning less than 200 points makes you a “New Bee”, and between 500 and 2000 points makes you a “Busy Bee.”   Most people it seems tried to earn their way up to being a “Busy Bee” and then would stop contributing.   Yes, this is more than they probably would have otherwise contributed, as the group of users who didn’t have points didn’t contribute as much.

I took a number of lessons from this work.  First, there can be a lot of value in valueless rewards like points.  Just because the reward doesn’t have any explicit value doesn’t mean that it won’t work as an incentive.   Second, “status classes” (like “New Bee” and “Busy Bee”) can be really powerful motivators.  Often people will work hard to get up to the next higher status class.   I seem to remember some other paper on status classes that suggests most people will be near the bottom of their status class, because once they get near the top they will do the extra effort to push themselves up to the next higher class.  Third, leader boards (top-ten lists) can also be important pieces of the motivation.   A number of their users reported trying to get more points to make it into the top ten users on the system.

And finally, it is really important to include some sort of dynamic changes in the points system.  One finding from this experiment is that the points system caused a one-time bump in contributions.   Everyone earned their way up to the status class they cared about, and then stopped contributing.  Because points never disappeared and never decayed, they were able to stay in that status class forever.  If a system wants to encourage continued contribution, then points can’t last forever.  Either the definition of the status class has to keep changing, or points have to expire or decay in value.  For example, the status class could be based on the number of points earned in the last month.

However, even without expiration, points might be valuable. They served as an encouragement for everyone to get up to a minimum level of contribution.  This is similar to the minimum threshold mechanism, but without the harsh punishment for undercontribution.  Points might help encourage new users to keep trying out the system and contributing for longer than they otherwise would have, raising the total contributions by raising the average initial contribution of each user.

R Farzan, JM DiMicco, DR Millen, B Brownholtz, W Geyer, C Dugan. (2008) “Results from Deploying a Participation Incentive Mechanism within the Enterprise.” Full paper, Conference on Human Factors in Computing Systems (CHI 2008), Florence, Italy, April 2008.

R Farzan, JM DiMicco, DR Millen, B Brownholtz, W Geyer, C Dugan. (2008) “When the experiment is over: Deploying an incentive system to all the users.” Full paper, Symposium on Persuasive Technology, In conjunction with the AISB 2008 Convention, Aberdeen, Scotland, April 2008.

2009 07 06 rwash Research Pointers Comments (0) Permalink

Cost of Implementing Results

From an interview with the Chicago economist Kevin Murphy:

What really does matter is the cost of treatment. If treatment costs are $10 trillion, the project has a negative net present value even if the research is free. With $2 trillion in treatment costs, the net gain from success is $3 trillion, so that we would get a good return even if the probability of success was one in 30. So when you think about research, it’s not the dollars you spend that matter—what matters is the cost of implementing the treatment that might be discovered. The downside to research is not failure, but unaffordable success. [Emphasis mine]

“The downside to research is not failure, but unaffordable success.”  He makes an excellent point here that applies well beyond health care or economics.  Applied research isn’t useful if it is too costly to implement.  I know we’ve seen this frequently in the computer science / HCI world.  Researchers will often say that they know how to do something that just isn’t being done by industry.  They ignore the fact that implementing the research can be very costly and not worth it.  The first example that comes to mind is complex cryptography.  We know how to do all kinds of interesting things with cryptography: secret sharing, private information retrieval, and sometimes even public key cryptography.   Industry often finds needs for these technologies.  But these technologies, as they currently stand, are too costly to implement.  Implementing the current version of private information retrieval would be enormously complex and difficult to use.  Even public key cryptography can be so slow that people don’t want to use it.

The same thing happens in HCI. For example, design techniques like participatory design often get ignored because they are so costly to implement.  Many prominent systems were designed without using these sophisticated design techniques that have come from the research community.

When doing research, one important piece of the puzzle is the cost of implementation.   If implementing the results of the research is cheap, then it is more likely to see widespread use.  A new technique or technology that works just as well as existing technologies but is easier to implement is definitely valuable research, and should be considered a contribution. When writing papers, I try to keep the “cost of implementation” in mind as I write; for example my minimum threshold paper tries hard to make the results of the work directly relevant and easy to use by system designers.

However, it is rare to see researchers work on improving the cost of implementation.   Implementation cost is very hard to measure, even for the most applied of research.  It often depends on many uncontrollable aspects such as who is implementing it, what complementary technologies and techniques are used, how much experience they have using it, etc.   Since it is hard to quantify implementation cost, it is hard to make the argument that some new technology or technique is an improvement along this dimension.  And without a solid argument along these lines, papers won’t be published and researchers won’t get raises / promotions / tenure / jobs.  As a result, researchers rarely undertake projects designed at improving the implementation cost.  Instead, most researchers focus on developing the “new hotness” — some novel idea that looks at something completely new and “unsolved” problem — rather than improving on existing solutions.

Of course, this opens up an opportunity for observant researchers.   Look for places where you can measure implementation cost in some way.   If you can measure it for a given technology or technique, then you can work to find improvements in existing solutions that will still lead to valuable papers.

HT: Marginal Revolution

2009 07 03 rwash Big Thoughts Comments (0) Permalink

Pot Luck Dinner

Pot luck dinners are dinners were everyone is expected to bring a dish of food for everyone to share.  When everyone who attends does this, you end up with a wide variety of food and usually more than enough food for everyone there.  Pot luck dinners also happen to be a great real-world example of the minimum threshold mechanism: you don’t get access to the rest of the food unless you bring a dish of your own.    And there is a threshold; it just usually isn’t stated explicitly.  If I brought exactly one chocolate chip cookie to a pot luck, you can be sure I’d get some dirty looks and I would not be invited again.

Pot luck dinners also illustrate another property of the minimum threshold mechanism: the food usually sucks.  Each person puts forth the least amount of effort they can get away with.  Often you end up with lots of store-bought potato salad or really easy-to-make homemade dishes.  Why spend more effort on your pot luck dish if a cheap tub of potato salad from the grocery store is good enough?

However, not all the food borders on the inedible.  Some people make great dishes, and those usually disappear moments after arriving. The “chef” who brought the dish usually derives some form of extra, personal benefit by bringing something great.   Often they get known for good pot-luck dishes and get invited to lots of parties.   When I was single I used to make really fancy pot-luck dishes as a way of catching the interest of the single women at the party; my favorite was individual cups of white chocolate mousse  drizzled with a raspberry sauce.  Now that I’m married, I don’t put near that much effort into pot-lucks.  (For those keeping score at home, I don’t think my wife has ever had my white chocolate mousse.  I had figured out that that didn’t work before I met her.)

Tagged
2009 06 29 rwash Unusual Examples of Mechanisms Comments (0) Permalink

But What About Quality?

Every time I present my research that looks at how to induce greater contributions to social computing systems, one of the first questions I get is “But what about quality?”  How do we make sure that the contributions are actually worthwhile?  This is an extremely important question for real social computing systems, and a number of real systems have failed not beause they didn’t have e nough contributions but because the contributions they did get weren’t very good.  While none of my work has explicitly been about quality, it is closely related.

Quality of contributions is a tricky property to improve.   In the end, the quality of the contributions is really up to the users who are contributing.  If all of the user contributions are really low quality (e.g. trolls) then there is not much the system can do with it.   There are two main methods of dealing with quality:

First, you can attempt to filter high-quality contributions from low-quality contributions.  This filtering can be programatically as long as there is some method of measuring quality automatically.  Or, you can have users rate others’ contributions and use those ratings to identify high-quality contributions.   Amazon.com does this with its 5-star ratings; users rate each review with 1 to 5 stars, Amazon averages all these ratings and provides an overall quality rating.  It then uses these ratings to better suggest new items.   Slashdot also uses user-supplied ratings in its comment moderation system.  Users are given a certain number of “moderator” points that they can use to vote certain comments either up or down.   But user ratings also add a new user-contribution problem complete with its own contribution and quality issues; Slashdot has had to introduce “meta-moderation” where users can vote on whether other users have been using their moderator points well.

Second, rather than trying to identify the quality of contributions, it may be possible to provide an incentive for users to increase the quality of their contributions.  Both the side effect mechanism and the minimum threshold mechanism can be used to encourage higher quality contributions if they are designed properly.  And there might be other mechanisms that more directly deal with quality issues.  This is an interesting open area for research.   Certainly, any solution of the first type that tries to identify high-quality contributions and then promotes those contributions might also, indirectly, induce users to try to produce higher-quality contributions.  Wikipedia allows users to give “barn stars” to other users for high-quality contributions; the rationale for these is that as a public display they encourage higher quality article writing and editing.  It is interesting to see if there are any direct quality improvement mechanisms that do not also work for quantity.

2009 06 26 rwash Big Thoughts Comments (0) Permalink