Shared Task: Lifelong Learning for Machine Translation
This task aims at developing autonomous lifelong learning Machine Translation systems. The initial training process is standard. The system is free to learn its model parameters using all provided bitexts that correspond to the WMT News Tasks data along with their year tag. The system is allowed to update its models using the incoming stream of source text (unsupervised adaptation), creating a better model to handle the next document. In addition, a simulated active learning protocol will allow the model to obtain help from a (simulated) human domain expert. This active learning will be simulated by obtaining the reference translation for a training example. See the details in the simulated active learning section. The evaluation will differ from standard evaluations since it will be performed across time, taking into account the cost of the simulated active learning. For this year the language pairs are:
- French-English
- German-English
CHECK THE GITHUB REPOSITORY
JOIN THE SLACK CHANNEL ALLIES_LLMT_WMT
GOALS
The goals of the shared task on lifelong learning for MT are:
- To develop systems that can self-develop relying solely on domain expert data and are then freed from the necessity of machine learning expertise
- To investigate the continuous training/adaptation of MT systems
- To provide additional publicly available corpora for machine translation and machine translation evaluation
- To explore methods to perform efficient active learning via a controlled simulated environment
- To foster research on unsupervised adaptation of MT systems
- To assess the effectiveness of document-level approaches
- To evaluate systems across time
IMPORTANT DATES
| Evaluation period | July 19-29, 2021 |
| Deadline for final system submission | July 29, 2021 at 23:59 (Anywhere on Earth) |
| System description | August 5, 2021 |
| Camera-ready version due | September 15, 2021 |
| Conference in Punta Cana | November 10-11, 2021 |
SIMULATED ACTIVE LEARNING
The autonomous system will be able to ask questions to domain experts when its confidence in the proposed output is low. Only one type of question can be asked, namely: "what is the translation of the sentence S". Each request of active learning will have a cost which is proportional to the number of source word of the query. The quantity of active learning data required to reach the performance level will be taken into account in the lifelong evaluation metric (see section evaluation)
DATA
LICENSING OF DATA
The data released for the WMT20 translation task can be freely used for research purposes, we just ask that you cite the WMT20 shared task overview paper, and respect any additional citation requirements on the individual data sets. For other uses of the data, you should consult with original owners of the data sets.
TRAINING DATA
We aim to use publicly available sources of data wherever possible. Our main sources of training data are the Europarl corpus, the UN corpus, the news-commentary corpus and the ParaCrawl corpus.
| Year | 2008 | 2009 | 2010 | 2011 | 2012 | 2013 | 2014 | 2015 | 2016 | 2017 | 2018 | 2019 |
| FR-EN | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ||||
| DE-EN | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
DOWNLOAD
- BEAT environment: allies_llmt_beat
- Parallel data: the data is available in the following github repository allies_llmt_data
File FR-EN DE-EN Notes Europarl v10 ✓ ✓ Now with metadata and timestamps. News Commentary v15 ✓ ✓ Now with metadata and timestamps.
SUBMISSION
What to submit? A file containing the translations for the test period along with the log file of the process (output of BEAT).
EVALUATION
Evaluation will be done automatically.
BLEU score across time including active-learning penalisation will be used.SYSTEM DESCRIPTION
You will be asked to describe your submission in a system description paper exhibiting the main aspect of your approach.
Evaluation will be done automatically.
BLEU score across time including active-learning penalisation will be used.