Bet You Can’t Do That Again

There is a problem raging in the sciences, referred to as the replication crisis, in which scientific studies are found to be either difficult or impossible to reproduce. In 2016, a poll of 1,500 scientists found that 70% reported a failure to reproduce at least one other scientist’s experiment. Even more surprising, 50% reported failing to reproduce at least one of their own experiments. Widespread findings of this sort do not exactly inspire confidence in the conclusions of scientific research.

Research into artificial intelligence (AI) is in no way immune to reproducibility problems. With the explosive growth in Embodied AI — a subdiscipline that equips AI algorithms with physical bodies for interaction with the environment — in recent years, problems with reproducibility have been amplified. A rapid development of many diverse tools, often with overlapping functionality, has fragmented the community and made even simple tasks a challenge to complete — causing problems with both reproducibility as well as entry into the field.

Recognizing these problems, the Allen Institute for AI has released AllenAct, a free, open source framework for research in Embodied AI. The framework provides support for a growing collection of simulated environments, tasks and algorithms, reproductions of state-of-the-art models, and includes extensive documentation and pre-trained models.

Written in Python and PyTorch, AllenAct allows experiments to be defined in code. Hyperparameters are specified in configuration files and can be versioned and tracked over time as an experiment progresses. Flexible data analysis pipelines can also be created that allow for a well-defined and repeatable set of steps to occur when running an experiment.

A large selection of tutorials, code examples, and pre-trained models help new projects to get off the ground quickly. More time can be devoted to solving the problem of interest, rather than spending time getting tools installed and learning the particulars of a new software package. Several visualization packages are also included to assist a user in interpreting results.

The Allen Institute for AI expects to continue development of AllenAct for at least the next several years, and has a number of ongoing projects using the framework, so support for the software should be strong for some time to come. For those wanting to give it a try, AllenAct is available on GitHub.