Fashion IQ Challenge 2019

Organized by yupenggao - Current server time: Oct. 27, 2020, 12:53 p.m. UTC

Previous

Test Phase
Sept. 1, 2019, midnight UTC

Current

Development Phase
June 15, 2019, midnight UTC

End

Competition Ends
Sept. 30, 2019, 11:59 p.m. UTC

Introduction

This is the Fashion IQ challenge submission and leaderboard site. The Fashion IQ challenge features a novel computer vision task: retrieving images by natural language based feedback. Advancement on this task can lead to image search engines with more natural and effective interface and can pave the way towards conversational image search retrieval systems.

Please check the challenge website for more details of the challenge.

Please check the Fashion IQ challenge 2020 website

News and Annoucement

Sep-30-2019: The challenge is over. Please send your technical report to yupeng.gao@ibm.com by Oct 5th.

July-20-2019: We released the baseline code for the challenge.

July-16-2019: We are truly sorry that we loss all the entries due to Codalab system failure. The server is now back to normal for submission.

June-15-2019: The dev phase has started. We offer 4k USD for prizes.

June-01-2019: We released the Fashion IQ dataset.

Contact Us

Please leave your comments and questions in the Forum Tab. Our team will respond in a speedy fashion. Alternatively, you can send your questions to yupeng.gao@ibm.com

Evaluation Metrics

The retrieval results are evaluated by recall on the test splits of the dataset. For each of the three fashion categories (dresses, tops, shirts), Recall@10 and Recall@50 will be computed on all test queries. The overall performance is computed by taking the average of Recall@10 and Recall@50 on the three fashion categories.

 

Submission Format

Please save your results in three files: "dress.predict.json", "toptee.predict.json", "shirt.predict.json" for both the development submission and the test submission. Please zip all three files into one "zip" file and submit it to the leaderboard of the respective phase. For each query, you will only need to keep the top 50 image IDs of the retrieval ranking list, since we only need to compute recall@10 and recall@50.

The submission should use the following format:

[{

"candidate": image_ID(str),

"ranking": image_ID(str),image_ID(str),image_ID(str),....

},

{

"candidate": image_ID(str),

"ranking": image_ID(str),image_ID(str),image_ID(str),....

},

... ...

]

Additional Information

  • We provide the evaluation script for your reference.
  • This is a sample submission. It contains "val.zip" for the dev phase and "test.zip" for the final test phase. Please unzip the sample submission and submit "val.zip" to dev leaderboard to check if your account works properly.

Terms and Conditions

General Rules

  • We will offer cash awards to top contenders of the leaderboard. 
  • Participants must submit a technical report to describe their methods by Oct. 5th. The technical reports will be linked on the workshop website. 
  • Participants can leverage external data or pretrained networks. Any and all external data used for training must be specified in the submitted technical report.
  • Results should follow the correct format and must be uploaded to the evaluation server through the CodaLab competition site.
  • Each entry needs to be associated to a team and its affiliation.
  • The organizers reserve the right to disqualify entries which is incomplete or illegible, late entries or entries that violate the rules.
  • The best entry of each team will be public in the leaderboard at all time. 

Dataset Access and Usage

  • The datasets are released for academic research only and it is free to researchers from educational or research institutions for non-commercial purposes. When downloading the dataset you agree not to reproduce, duplicate, copy, sell, trade, resell or exploit for any commercial purposes, any portion of the images and any portion of derived data.
  • To obtain the attribute labels associated with Fashion IQ dataset, please send an email to yupeng.gao@ibm.com 
  • Please cite the following paper if are using Fashion IQ dataset: 

The Fashion IQ Dataset: Retrieving Images by Combining Side Information and Relative Natural Language Feedback. Xiaoxiao Guo*, Hui Wu*, Yupeng Gao, Steve Rennie and Rogerio Feris. Arxiv. 2019

Development Phase

Start: June 15, 2019, midnight

Description: In this phase, you can submit the result of validation set and see your rank in leaderboard.

Test Phase

Start: Sept. 1, 2019, midnight

Description: In this phase, you can submit the result of test set and the leaderboard will show the result of testing set.

Competition Ends

Sept. 30, 2019, 11:59 p.m.

You must be logged in to participate in competitions.

Sign In
# Username Score
1 suits 0.5254
2 Shuan 0.4435
3 superraptors 0.3832