EMNLP 2011 SIXTH WORKSHOP
ON STATISTICAL MACHINE TRANSLATION

Shared Task: Machine Translation

July 30 - 31, 2011
Edinburgh, UK

[HOME] | [TRANSLATION TASK] | [FEATURED TRANSLATION TASK] | [SYSTEM COMBINATION TASK] [EVALUATION TASK] | [RESULTS]
[BASELINE SYSTEM] | [BASELINE SYSTEM 2]
[SCHEDULE] | [PAPERS] | [AUTHORS]

The recurring translation task of the WMT workshops focuses on European language pairs. Translation quality will be evaluated on a shared, unseen test set of news stories. We provide a parallel corpus as training data, a baseline system, and additional resources for download. Participants may augment the baseline system or use their own system.

This year we also have a featured translation task: translating Haitian Creole SMS messages into English. This task includes real-world data in the form of emergency response messages that were sent in the aftermath of the devastating 2010 Haitian earthquake.

GOALS

The goals of the shared translation task are:

We hope that both beginners and established research groups will participate in this task.

TASK DESCRIPTION

We provide training data for four European language pairs, and a common framework (including a baseline system). The task is to improve methods current methods. This can be done in many ways. For instance participants could try to:

Participants will use their systems to translate a test set of unseen sentences in the source language. The translation quality is measured by a manual evaluation and various automatic evaluation metrics. Participants agree to contribute to the manual evaluation about eight hours of work.

You may participate in any or all of the following language pairs:

For all language pairs we will test translation in both directions. To have a common framework that allows for comparable results, and also to lower the barrier to entry, we provide a common training set and baseline system.

We also strongly encourage your participation, if you use your own training corpus, your own sentence alignment, your own language model, or your own decoder.

If you use additional training data or existing translation systems, you must flag that your system uses additional data. We will distinguish system submissions that used the provided training data (constrained) from submissions that used significant additional data resources. Note that basic linguistic tools such as taggers, parsers, or morphological analyzers are allowed in the constrained condition.

Your submission report should highlight in which ways your own methods and data differ from the standard task. We may break down submitted results in different tracks, based on what resources were used. We are mostly interested in submission that are constraint to the provided training data, so that the comparison is focused on the methods, not on the data used. You may submit contrastive runs to demonstrate the benefit of additional training data.

TRAINING DATA

The provided data is mainly taken from version 6 of the Europarl corpus, which is freely available. Please click on the links below to download the sentence-aligned data, or go to the Europarl website for the source release.

Additional training data is taken from the new News Commentary corpus. There are about 45 million words of training data per language from the Europarl corpus and 2 million words from the News Commentary corpus.

Europarl
  • French-English
  • Spanish-English
  • German-English
  • Czech-English
  • French monolingual
  • Spanish monolingual
  • German monolingual
  • Czech monolingual
  • English monolingual
News Commentary
  • French-English
  • Spanish-English
  • German-English
  • Czech-English
  • French monolingual
  • Spanish monolingual
  • German monolingual
  • Czech monolingual
  • English monolingual
News
  • French monolingual
  • Spanish monolingual
  • German monolingual
  • English monolingual
  • Czech monolingual
United Nations
  • French-English
  • Spanish-English
French-English 109 corpus
  • French-English
Crawled from Canadian and European Union sources.
CzEng
  • Czech-English
The current version of the CzEng corpus (version v0.9) is available from the CzEng web site (note: same as last year).

You may also use the following monolingual corpora released by the LDC:

Note that the released data is not tokenized and includes sentences of any length (including empty sentences). All data is in Unicode (UTF-8) format. The following tools allow the processing of the training data into tokenized format:

DEVELOPMENT DATA

To tune your system during development, we suggest using the 2008 test set of 2051 sentences. The data is provided in raw text format and in an SGML format that suits the NIST scoring tool. We also release the 2009 test set of 2525 sentences, a system combination tuning set of 502 sentences. and the 2010 test set.

News news-test2008
  • English
  • French
  • Spanish
  • German
  • Czech
  • Hungarian
This data is a cleaned version of the 2008 test set.
News news-test2009
  • English
  • French
  • Spanish
  • German
  • Czech
  • Hungarian
  • Italian
News news-test2010
  • English
  • French
  • Spanish
  • German
  • Czech

DOWNLOAD

** NB Same file as last year

TEST SET SUBMISSION

Punctuation in the official test sets will be encoded with ASCII characters (not complex Unicode characters) as much as possible. You may want to normalize your system's output before submission. You are able able to use a rawer version of the test sets that does not have this normalization.

To submit your results, please first convert into into SGML format as required by the NIST BLEU scorer, and then upload it to the website matrix.statmt.org.

SGML Format

Each submitted file has to be in a format that is used by standard scoring scripts such as NIST BLEU or TER.

This format is similar to the one used in the source test set files that were released, except for:

The script wrap-xml.perl makes the conversion of a output file in one-segment-per-line format into the required SGML file very easy:

Format: wrap-xml.perl LANGUAGE SRC_SGML_FILE SYSTEM_NAME < IN > OUT
Example: wrap-xml.perl en newstest2011-src.de.sgm Google < decoder-output > decoder-output.sgm

Upload to Website

Upload happens in three easy steps:

If you are submitting contrastive runs, please submit your primary system first and mark it clearly as the primary submission.

EVALUATION

Evaluation will be done both automatically as well as by human judgement.

DATES

Release of training dataDecember 15, 2010
Test set distributed for translation taskMarch 14, 2011
Submission deadline for translation taskMarch 20, 2011
Paper due dateMay 19, 2011

supported by the EuroMatrixPlus project
P7-IST-231720-STP
funded by the European Commission
under Framework Programme 7