Evaluation

Introduction of the problem

The problem is a regression problem. The goal is to predict the time a patient survives after his admission in a hospital. To do so, each sample (aka each patient) is characterized by the following information:

• Age
• Ethnicity (with a one-hot representation)
• Gender
• Blood pressure (systolic and diastolic)
• Glycated hemoglobin concentration
• Body Mass Index (or BMI)

This give you a total of 10 features to predict the following couple:

• Target: the survival time to predict
• Event: if the data is censored (the patient left the study or the study stopped too early) 1 else 0

You are given for training a data matrix X_train of dimension 19297 x 10 and an array y_train of labels of dimension 19297 x 2. You must train a model which predicts the labels for two test matrices X_valid and X_test.

Evaluation metric

To evaluate the performance of your model, you will need a metric: the concordance index. The formula is:

In other word, the concordance index is a “global” index for validating the predictive ability of a survival model. It is the fraction of pairs in your data, where the observation with the higher survival time has the higher probability of survival predicted by your model.

The index is not calculated for every observation/subject. So the c-index can not be interpreted as the risk of a subject. High values mean that your model predicts higher probabilities of survival for higher observed survival times.

Different steps

There are 2 phases:

• Phase 1: development phase. We provide you with labeled training data and unlabeled validation and test data. Make predictions for both datasets. However, you will receive feed-back on your performance on the validation set only. The performance of your LAST submission will be displayed on the leaderboard.
• Phase 2: final phase. You do not need to do anything. Your last submission of phase 1 will be automatically forwarded. Your performance on the test set will appear on the leaderboard when the organizers finish checking the submissions.

During all this sample competition, you are allowed to submit either:

• Only prediction results (no code)
• A pre-trained prediction model
• A prediction model that must be trained and tested

The submissions are evaluated using the above metric: the concordance index.

Rules

Submissions must be made before the end of phase 1. You may submit 5 submissions every day and 100 in total.

This challenge is governed by the general ChaLearn contest rules.