Skip to content

Open-source tools for running online field experiments

Dean Eckles points me to this cool new tool for experimentation:

I [Eckles] just wanted to share that in a collaboration between Facebook and Stanford, we have a new paper out about running online field experiments. One thing this paper does is describe some of the tools we use to design, deploy, and analyze experiments, including the2012 US election voter turnout experiment. And now we have open sourced an implementation of these ideas.

We were inspired by Fisher’s quote — “To consult the statistician after an experiment is finished is often merely to ask him to conduct a post-mortem examination.”

The idea is that one way to consult a statistician in advance is to have their advice built into tools for running experiments — a similar idea to how you emphasize the importance of defaults in data visualization tools.

We have a shorter blog post about this work, a paper, “Designing and Deploying Online Field Experiments” (to appear in Proc. of WWW very soon), and the software and documentation, PlanOut.

We’d be very interested in your thoughts on any of this, as perhaps would your blog readers. I also think many might be interested in using the software to run experiments themselves — that’s our hope!

Looks good to me. It’s great to see this sort of stuff out there, not just in textbooks but really getting used.

P.S. Brian Keegan writes in:

I’m a post-doc in David Lazer’s computational social science group at Northeastern. I noticed that you were going to discuss open-source tools for running online experiments on Thursday, so I wanted to offer a shameless plug for a platform that we’re developing here in hopes it might fit into the themes of your post.

Volunteer Science ( is a web laboratory platform for conducting group and network experiments that’s built on open standards and open code. We’re very much interested in recruiting others to develop experiments for the platform as well as expanding the number of users who volunteer.


  1. Daniel Gotthardt says:


    Somehow the formatting went bad. Btw.: Do you intend to discuss facebook’s mood manipulation experiment at some point in the future?

  2. Rahul says:

    Naive question: I understand an online expt. & a field expt. But what’s an online field expt?

    • Daniel Gotthardt says:

      Bakshy et al. wrote:

      Randomized field experiments are central to contemporary design
      and development processes for Internet services. In the most
      popular case, practitioners use “A/B tests” that randomly assign
      users to one of two variations of a service

      I think the point is that online communities are observed in their natural environment while the treatment is (randomly) assigned.

    • Dean Eckles says:

      Unfortunately, ‘online’ has way too many meanings in related contexts, so this can be a bit confusing.

      Basically all we denote by ‘online field experiment’ is a field experiment that is run on an Internet service. This would be distinct from other online experiments, such as those on Mechanical Turk, that might not be regarded as field experiments per se. Of course, what counts as “the field” is relative.

      • Rahul says:

        So, if I want to see if a blue background web page will get more online sales than green, is that an online field experiment?

        OTOH, if I survey women about the color of their clothes & menstrual cycles that’s not an online field experiment?

        • ZC says:

          The survey likely doesn’t constitute an experiment of any kind. I’m not Dean, but I’d classify your first example as an online field experiment, yes. I think the key difference is having the subjects not be in a contrived setting as the lab typically is.

  3. K? O'Rourke says:

    Neat (some of us that get to do only a few RCTs a year with long delays from REB are jealous)

    And nice that this is being pointed out in the paper “results of a field experiment from one time and one population may not generalize to new times and populations”

  4. […] 10 – Open-source tools for running online field experiments by Andrew Dean Eckles points me to this cool new tool for experimentation: I [Eckles] just wanted […]

  5. JVM says:

    In collaboration with my lab at NYU, I was involved in building a tool called PsiTurk that made it fairly easy to run large online experiments using Amazon’s Mechanical Turk, just thought I’d shout it out here:

    A lot of cognitive Psych researchers have found it useful. The team has done a ton of great work on it since I left the lab.

  6. Came across this thread while searching to determine if VolunteerScience is open sourced. Do you know if there are currently any open-source solutions that enable fast development of experiments in which multiple people interacting simultaneously? ( Eg, we do network experiments: )

    As opposed to some existing excellent tools (like TurkServer and PsiTurk) that are well suited to one-player games but seem to provide no multi-player support. Also are very turk-focused.

Leave a Reply