CodaLab Competitions is a powerful open source framework for running competitions that involve
result or code submission. You can either participate in an existing competition or host a new
Most competitions hosted on Codalab are machine learning (data science)
competitions, but Codalab is NOT limited to this application domain. It can accommodate any
problem for which a solution can be provided in the form of a zip archive containing a number of
files to be evaluated quantitatively by a scoring program (provided by the organizers). The
scoring program must return a numeric score, which is displayed on a leaderboard where the
performances of participants are compared.
Codalab was created in 2013 as a joint venture between Microsoft and Stanford University.
Originally the vision was to create an ecosystem for conducting computational research in a more
efficient, reproducible, and collaborative manner, combining worksheets and competitions.
Worksheets capture complex research pipelines in a reproducible way and create "executable
Some competitions have been organized using worksheets, but the competition platform
and the worksheet platform have both a large user base and can be used independently. In 2014,
ChaLearn joined to co-develop Codalab
2015, University Paris-Saclay is
community lead of Codalab competitions, under the direction of Isabelle Guyon, professor of big
data. Codalab is administered by CKCollab and the LRI staff.
Codalab in research
Codalab is used actively in research. In 2016/2017, 88 new challenges were launched. Recent high
profile challenges organized with Codalab include the 2 million
Euro prize of the EU, organized
by the See.4C consortium, the CIKM
which attracted 493 participants,
(633 participants) and the ChaLearn
AutoML challenge 2017 (687 participants).
Since 2016, Codalab offers the possibility of organizing machine learning challenges with code
submission. The simplest machine learning challenges require only the submission of results,
which are compared to a solution (or key) by a scoring program. Result submission challenges are
less computationally expensive than code submission challenges. However, they offer less
possibilities. In particular, code submission allows conducting fair benchmarks by executing
submitted code in the same condition for all participants.
Codalab has been providing free resources for challenge organizers who want to run high impact
events, within a pre-approved agreed upon budget. New since version 1.5: organizers can hook up
their own compute workers to the backend of Codalab to redirect the code submissions, enabling
growth to big data competitions running at the expense of the organizers. For very special
dedicated projects, Codalab can be customized since it is an open source project.
We are preparing a challenge on
Automatic Deep Learning (AutoDL)
challenging participants to design code eliminating the need of human expertise to choose the architecture and hyper-parameters of
deep neural networks. The challenge is co-organized and sponsored by Google. The protocol will be tested on Codalab in October.
The LAL and CERN are organizing a challenge to reconstruct
particle trajectories in high energy physics detectors. After the success of the
first phase with result submission only, a second phase with code submission will be run on Codalab. TrackML is an officially selected
challenge of the NIPS 2018 conference.
February 2018: 2 million Euro Big Data EU prize powered by Codalab.
February 2018: Isabelle Guyon presents Codalab at the newly formed Institute of Convergence DataIA
January 2018: Paris-Saclay master students create challenges for L2 students.
January 2018: Paris-Saclay instructors create reinforcement learning homework.
December 2017: Codalab exceeds 10000 users with 480 competitions (145 public)
December 2017: Codalab presented at the Challenges in Machine Learning workshop [slides].
Version 1.5 is out!
November 2017: Explore the new features: scale up your code submission competition with
your own compute workers (full privacy, dockers); organize RL challenges and hook up simulators
providing data on demand (with your own "ingestion program"); use the ChaLab wizard to create
competitions in minutes.