Ideally, when scientists read published results from an experiment, they want to be able to repeat the experiment and confirm the outcomes. Consequently, ideally results are published so that that everything can be reproduced, describing exactly what they did and how they did it. In practice, as scientific procedures become more complex, this has become difficult, especially in computational sciences.

Nowadays, replicating a result may require a worker to have the same software as well as the same initial data, settings and methodology as did the original scientists. And yet the traditional medium for publishing results—a scientific paper just a few pages long—can’t possibly give a researcher everything they need to replicate this new class of experiments.

“As it is now often practiced, one can make a good case that computing is the last refuge of the scientific scoundrel,” said Randy LeVeque, a prominent mathematician in a keynote presentation at an international mathematics conference in 2006. He complained that published results of computational experiments often lack detailed descriptions of the methods used. “Scientific and mathematical journals are filled with pretty pictures these days of computational experiments that the reader has no hope of repeating,” he said.

We develop and use Madagascar as an solution to this problem, in association with other groups arounf the world. Our approach to reproducibility is to keep computational experiments together with data sets and past versions of the software. It requires documenting how the software changes and repeating experiments on newer versions to see how they compare with earlier runs.

Michael Tobis