Massively multilingual machine translation has shown impressive capabilities, including zero and few-shot translation of low-resource languages. However, these models are often evaluated from or into English, where the most data is available, and assuming that models generalise to other pairs and low-resource languages.
In the first edition of the Multilingual Low-Resource Translation shared task, we focus on multilinguality in the cultural heritage domain for two Indo-European language families: North-Germanic and Romance. We want to explore how information in one language can be transferred to other related languages and, for this, we evaluate translation quality in low-resourced language pairs, but explicitly encourage the use of data of the high-resourced language-pairs in the same family. We would like to answer the question: do we need English and/or Spanish for high quality translation of related languages? If so, which is the best way to combine the data: pivot, cascade, multilingual, via pre-training? To explore the topic, we present two tasks with different characteristics, one per language family (see below). Even if data in the highest resourced languages is allowed and encouraged for training, translation quality will be only evaluated on the other pairs. Participants are also encouraged to make available to the community any additional resources useful for the task.
Given the importance of named entities in the cultural heritage domain, we provide participants with parallel/multilingual lexicons from Wiktionary, Wikidata and Wikipedia titles. One can also create its own resource from Wikimedia, we just extract the data for the shared task languages to facilitate participation.
| North-Germanic |
Romance |
Info |
||||
|---|---|---|---|---|---|---|
| is-nb-sv | bilinguals | ca-it-oc-ro | ca-it-ro | bilinguals | ||
| Wikidata | all cleaner | all | all cleaner | all cleaner | all | README |
| Wiktionary | ---- | all | ---- | ---- | all | README |
| Wikipedia Titles | ---- | ---- | all cleaner | all cleaner | all | README |
europeana.test.sv.xml used in the evaluation had accidentally an additional line break at line 618 ("I min uppsats har jag valt att undersöka hur och i vilket syfte pedagoger använder sig av musik i förskol \\ an." should be a single line and "förskolan." a token). This has been taken into account in the evaluation. The full test set corrects this mistake.
Automatic evaluation results will be reported per language pair and in average. The final ranking will be done according to the average translation quality per subtask, that is, per family and not per pair. When possible, we will also perform human evaluation at document level on a subset of paragraphs/sentences. If you plan to participate, please, contact us so that we can plan the human evaluation. If you are native in any of the target languages involved and are interested in the evaluation, please, also contact us!
Since the official evaluation will be done per family, participants can participate in only one of the tasks if preferred, but they are expected to submit translations for all the languages involved in the task.
sysid="myTeamPrimary". Please, use this same name as the first part of the file name for your submission (see below).
Two submissions per group and task are allowed: a primary and a contrastive system. Participants will have to fill a form at submission time describing the main characteristics per system.
File naming. Name your files with your system name (including type of submission primary vs. contrastive), language family and translation direction. Examples:
myTeamPrimary.romance.ca2it.xml myTeamContrastive.romance.ca2it.xml myTeamPrimary.germanic.is2sv.xml Send your translations by email to cristinae aatt dfki.de and fill this short online form per system (for each combination Romance/Germanic/Primary/Contrastive). You will receive a confirmation email in few hours.
| Release of initial training data | April 5, 2021 |
| Additional training data deadline | May 19, 2021 |
| Release of test data | June 29, 2021 |
| Results submission deadline | July 6, 2021 |
| Paper submission deadline | August 5, 2021 |
| Paper notification | September 5, 2021 |
| Camera-ready version due | September 15, 2021 |
| Conference at EMNLP | November 10-11, 2021 |
This shared task is funded by the European Language Resource Coordination ELRC (SMART 2019/1083) and LT-BRIDGE (H2020, 952194), and supported by the Directorate-General for Language Policy, Ministry of Culture. Government of Catalonia. We are thankful to Europeana for providing source texts in Icelandic, Norwegian and Swedish.