The recurring translation task of the WMT workshops focuses on European language pairs. For the years 2015-2017 our core language pairs will be English to and from Czech and German. In addition, we will introduce a guest language each year that provides a particular translation challenge. For 2015 the guest language will be Finnish. We provide parallel corpora for all languages as training data, and additional resources for download.

GOALS

The goals of the shared translation task are:

• To investigate the applicability of current MT techniques when translating into languages other than English
• To examine special challenges in translating between European languages, including word order differences and morphology
• To investigate the translation of low-resource, morphologically rich languages
• To create publicly available corpora for machine translation and machine translation evaluation
• To generate up-to-date performance numbers for European languages in order to provide a basis of comparison in future research
• To offer newcomers a smooth start with hands-on experience in state-of-the-art statistical machine translation methods

TASK DESCRIPTION

We provide training data for five language pairs, and a common framework (including a baseline system). The task is to improve methods current methods. This can be done in many ways. For instance participants could try to:

• improve word alignment quality, phrase extraction, phrase scoring
• add new components to the open source software of the baseline system
• augment the system otherwise (e.g. by preprocessing, reranking, etc.)
• build an entirely new translation systems

You may participate in any or all of the following language pairs:

• French-English
• Finnish-English
• German-English
• Czech-English
• Russian-English

We also strongly encourage your participation, if you use your own training corpus, your own sentence alignment, your own language model, or your own decoder.

If you use additional training data or existing translation systems, you must flag that your system uses additional data. We will distinguish system submissions that used the provided training data (constrained) from submissions that used significant additional data resources. Note that basic linguistic tools such as taggers, parsers, or morphological analyzers are allowed in the constrained condition.

Your submission report should highlight in which ways your own methods and data differ from the standard task. We may break down submitted results in different tracks, based on what resources were used. We are mostly interested in submission that are constraint to the provided training data, so that the comparison is focused on the methods, not on the data used. You may submit contrastive runs to demonstrate the benefit of additional training data.

TRAINING DATA

The provided data is mainly taken from version 7 of the Europarl corpus, which is freely available. Please click on the links below to download the sentence-aligned data, or go to the Europarl website for the source release. Note that this the same data as last year, since Europarl is not anymore translted across all 23 official European languages.

Additional training data is taken from the new News Commentary corpus. There are about 50 million words of training data per language from the Europarl corpus and 3 million words from the News Commentary corpus.

A new data resource from 2013 is the Common Crawl corpus which was collected from web sources. Each parallel corpus comes with a annotation file that gives the source of each sentence pair.

We will be releasing development data for the Finnish-English task, and for the French-English task. The reason for the French-English release is that the text type has changed this year, to a more informal type gathered from user-generated comments.

You may also use the following monolingual corpora released by the LDC:

• LDC2011T07 English Gigaword Fifth Edition
• LDC2009T13 English Gigaword Fourth Edition
• LDC2007T07 English Gigaword Third Edition
• LDC2006T17 French Gigaword First Edition
• LDC2009T28 French Gigaword Second Edition
• LDC2011T10 French Gigaword Third Edition

Note that the released data is not tokenized and includes sentences of any length (including empty sentences). All data is in Unicode (UTF-8) format. The following tools allow the processing of the training data into tokenized format:

• Tokenizer tokenizer.perl
• Detokenizer detokenizer.perl
• Lowercaser lowercase.perl
• SGML Wrapper wrap-xml.perl

DEVELOPMENT DATA

To evaluate your system during development, we suggest using the 2014 test set. The data is provided in raw text format and in an SGML format that suits the NIST scoring tool. We also release other test sets from previous years.

The news-test2011 set has three additional Czech translations that you may want to use. You can download them from Charles University.

DOWNLOAD

• Parallel data:

  File	Size	CS-EN	DE-EN	FI-EN	FR-EN	RU-EN	Notes
  Europarl v7	628MB	✓	✓		✓		same as previous year, corpus home page
  Europarl v8	179M			✓			new for this year corpus home page
  Common Crawl corpus	876MB	✓	✓		✓	✓	same as previous year
  UN corpus	2.3GB				✓		same as previous year, corpus home page
  News Commentary v10	122M	✓	✓		✓	✓	updated
  109French-English corpus	2.3 GB				✓		same as previous year [md5|sha1]
  CzEng 1.0	115MB	✓					same as previous year, corpus home page (avoid sections 98 and 99)
  Yandex 1M corpus	121MB					✓	corpus home page; v1.3 now in original case
  Wiki Headlines	9.1MB			✓		✓	Provided by CMU..

• Monolingual language model training data:

  Corpus	CS	DE	EN	FI	FR	RU	All languages combined	Notes
  Europarl v7/v8	32MB	107MB	99MB	95MB	107MB		446MB
  News Commentary	12MB	15MB	18MB		15MB	16MB	73MB
  News Crawl: articles from 2007	3.7MB	92MB	198MB		6.0MB		302MB	News Crawl Extracted article text from various online news publications. The data sets from 2007-2013 are the same as last year's.
  News Crawl: articles from 2008	191MB	313MB	672MB		244MB	2.3MB	1.5GB
  News Crawl: articles from 2009	194MB	296MB	757MB		233MB	5.1MB	1.6GB
  News Crawl: articles from 2010	107MB	135MB	345MB		99MB	2.5MB	727MB
  News Crawl: articles from 2011	389MB	746MB	784MB		317MB	564MB	3.1GB
  News Crawl: articles from 2012	337MB	946MB	751MB		218MB	568MB	3.1GB
  News Crawl: articles from 2013	395MB	1.6GB	1.1GB		474MB	730MB	4.3GB
  News Crawl: articles from 2014	380MB	2.1GB	1.4GB	52M	598MB	801MB	5.3GB
  News Crawl 2014 (first release - not recommended as it overlaps with newstest2014)	380MB	2.1GB	1.4GB	52M	598MB	801MB	5.3GB
  News Discussions. Version 1 from 2014/15			1.7GB		146MB			New for 2015

• Development sets (21 MB)
• Development sets - updated 2015-01-27 (Translations checked)
• NEW: Test sets (source and reference) (2MB)

TEST SET SUBMISSION

Punctuation in the official test sets will be encoded with ASCII characters (not complex Unicode characters) as much as possible. You may want to normalize your system's output before submission.

To submit your results, please first convert into into SGML format as required by the NIST BLEU scorer, and then upload it to the website matrix.statmt.org.

SGML Format

Each submitted file has to be in a format that is used by standard scoring scripts such as NIST BLEU or TER.

This format is similar to the one used in the source test set files that were released, except for:

• First line is <tstset trglang="en" setid="newstest2015" srclang="any">, with trglang set to either en, de, fr, es, cs or ru. Important: srclang is always any.
• Each document tag also has to include the system name, e.g. sysid="uedin".
• CLosing tag (last line) is </tstset>

The script wrap-xml.perl makes the conversion of a output file in one-segment-per-line format into the required SGML file very easy:

Format: wrap-xml.perl LANGUAGE SRC_SGML_FILE SYSTEM_NAME < IN > OUT
Example: wrap-xml.perl en newstest2015-src.de.sgm Google < decoder-output > decoder-output.sgm

Upload to Website

Upload happens in three easy steps:

1. Go to the website matrix.statmt.org.
2. Create an account under the menu item Account -> Create Account.
3. Go to Account -> upload/edit content, and follow the link "Submit a system run"
   • select as test set "newstest2015" and the language pair you are submitting
   • select "create new system"
   • click "continue"
   • on the next page, upload your file and add some description

If you are submitting contrastive runs, please submit your primary system first and mark it clearly as the primary submission.

EVALUATION

Evaluation will be done both automatically as well as by human judgement.

• Manual Scoring: We will collect subjective judgments about translation quality from human annotators. If you participate in the shared task, we ask you to perform a defined amount of evaluation per language pair submitted. The judgements will be collected in "HITs", where each HIT consists of ranking the outputs of 5 different systems for 3 sentences. Each team will be expected to complete 100 HITs per language pair. The evaluation will be done with an online tool.
• As in previous years, we expect the translated submissions to be in recased, detokenized, XML format, just as in most other translation campaigns (NIST, TC-Star).

ACKNOWLEDGEMENTS

WMT15 receives support from the European Union under the projects MosesCore (grant number 288487), Cracker and QT21.
We thank Yandex for their donation of data for the Russian-English task.