What the hell is that?

Organized by vision - Current server time: Oct. 27, 2020, 12:36 p.m. UTC

Previous

Development
Oct. 22, 2017, 6:53 p.m. UTC

Current

Final
April 30, 2018, 6:53 p.m. UTC

End

Competition Ends
Never

What the hell is that ?

Brought to you by Vision group

 

Domains of computer vision

Computer Vision is a field related to artificial intelligence allowing the analysis and understanding of the image by an acquisition system such as a camera or more generally, a sensor.

In today's world where we are looking for autonomous systems that can understand and evolve in any environment, this field has become one of the most active areas of artificial intelligence research with impact in as many diverse fields. We can cite, for example, the development of intelligent vehicles (and drones) carried by Google  and many car companies (PSA, Audi, Tesla, ...), medical imaging (decision support, biomedical instrumentation), the recognition of handwriting or deep experimental analysis of scientific experiments (fundamental physic, simulations, ...).

 

For this last area, there are some diversified projects like Astronometry or LSST project to automatically detect celestial body or Varcity, a project from ETH Zurich to create a simulation of dynamic cities containing events and traffic flow from image. Beyond technological innovation, this field has an important impact on how we interact with technology by stimulating decision support for our everyday actions such as locating a place with a photo, restaurant classification or just evaluate the filling status of a refrigerator for example.

 

 

In the context of this challenge, we will study one of the problems of this field that is image classification. This problem is recurrent because at the base of any problem of decision, specially for intelligent vehicules. Indeed, the initial step (before evaluating any action) is to understand and evaluate the evolution environment of the system. Therefore, it is essential to understand the specificities and difficulties of this problem. To illustrate this problematic, we chose autonomous vehicules context and we propose you to study the image source CIFAR-10.

 

This dataset contains groups entities that can interact with the vehicle environment like animals( cat, horse, dog, ...) and vehicles (bike, car, truck, ...). This approach will allow you to get acquainted with the fundamental notions of this field of application and will initiate you to the best practices and methodologies to use in the general context of machine learning.

 

To contact us:  vision@chalearn.org

References and credits:

  • MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications" ,An- drew G. Howard, Menglong Zhu et al, 2017.
  • "CNN Features off-the-shelf: an Astounding Baseline for Recognition", Ali Sharif Razavian et al, 2014
  • Learning Multiple Layers of Features from Tiny Images, Alex Krizhevsky, 2009.
  • LSST project website: http://dm.lsst.org/#myCarousel
  • Astronometry website: http://astrometry.net/
  • VarCity website: https://varcity.ethz.ch/
  • This challenge was generated using ChaLab.

     

     

What the hell is that ? - Evaluation

The problem is a multiclass classification problem. Each sample (an image) is characterized by 256 features obtained by a Convonlution Neural Network (CNN). You must recognized what is on the picture. 


You are given for training a data matrix X_train of dimension num_training_samples x num_features and an array y_train of labels of dimension num_training_samples. You must train a model which predicts the labels for two test matrices X_valid and X_test. 


To prepare your submission, remember to use predict_proba, which provides a matrix of prediction scores scaled between 0 and 1. The dimension of the matrix is 1 x num_classes. Each line represents the probabilities of class membership, which sum up to one. Preparing your submission with the starting kit is the easiest. 

There are 2 phases:

  • Phase 1: development phase. We provide you with labeled training data and unlabeled validation and test data. Make predictions for both datasets. However, you will receive feed-back on your performance on the validation set only. The performance of your LAST submission will be displayed on the leaderboard.
  • Phase 2: final phase. You do not need to do anything. Your last submission of phase 1 will be automatically forwarded. Your performance on the test set will appear on the leaderboard when the organizers finish checking the submissions.

This sample competition allows you to submit either:

  • Only prediction results (no code).
  • A pre-trained prediction model.
  • A prediction model that must be trained and tested.

The submissions are evaluated using the bac_multiclass metric. This metric computes the balanced accuracy (that is the average of the per class accuracies). The metric is re-scaled linearly between 0 and 1, 0 corresponding to a random guess and 1 to perfect predictions.

What the hell is that ? - Rules

Submissions must be made before the end of phase 1. You may submit 5 submissions every day and 100 in total.

This challenge is governed by the general ChaLearn contest rules.

 

This competition is organized solely for test purposes. No prizes will be awarded.

The authors decline responsibility for mistakes, incompleteness or lack of quality of the information provided in the challenge website. The authors are not responsible for any contents linked or referred to from the pages of this site, which are external to this site. The authors intended not to use any copyrighted material or, if not possible, to indicate the copyright of the respective object. The authors intended not to violate any patent rights or, if not possible, to indicate the patents of the respective objects. The payment of royalties or other fees for use of methods, which may be protected by patents, remains the responsibility of the users.

ALL INFORMATION, SOFTWARE, DOCUMENTATION, AND DATA ARE PROVIDED "AS-IS" THE ORGANIZERS DISCLAIM ANY EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. IN NO EVENT SHALL ISABELLE GUYON AND/OR OTHER ORGANIZERS BE LIABLE FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF SOFTWARE, DOCUMENTS, MATERIALS, PUBLICATIONS, OR INFORMATION MADE AVAILABLE THROUGH THIS WEBSITE.

Participation in the organized challenge is not-binding and without obligation. Parts of the pages or the complete publication and information might be extended, changed or partly or completely deleted by the authors without notice.

Development

Start: Oct. 22, 2017, 6:53 p.m.

Description: Development phase: create models and submit them or directly submit results on validation and/or test data; feed-back are provided on the validation set only.

Final

Start: April 30, 2018, 6:53 p.m.

Description: Final phase: submissions from the previous phase are automatically cloned and used to compute the final score. The results on the test set will be revealed when the organizers make them available.

Competition Ends

Never

You must be logged in to participate in competitions.

Sign In
# Username Score
1 Zhengying 0.8682
2 Uriopass 0.8670
3 regard 0.8638