ACL 2007 SECOND WORKSHOP
ON STATISTICAL MACHINE TRANSLATION

Shared Task: Machine Translation for European Languages

June 23, in conjunction with ACL 2007 in Prague, Czech Republic

[HOME] [SHARED TASK] | RESULTS] | [BASELINE SYSTEM] | [PROCEEDINGS] | [SCHEDULE]

The shared task of the workshop is translation between European language pairs. Translation quality will be evaluated on a shared, unseen test set. We provide a parallel corpus as training data, a baseline system, and additional resources for download. Participants may augment this system or use their own system.

Goals

The goals of the shared translation task are: We hope that both beginners and established research groups will participate in this task.

New Challenges This Year

Task Description

We provide training data for four European language pairs, and a common framework (including a language model and a baseline system). The task is to improve methods current methods. This can be done in many ways. For instance participants could try to Participants will use their systems to translate a test set of unseen sentences in the source language. The translation quality is measured by a manual evaluation and various automatic evaluation metrics. Participants agree to contribute to the manual evaluation about eight hours of work.

You may participate in any or all of the following language pairs:

To have a common framework that allows for comparable results, and also to lower the barrier to entry, we provide a common training set, language model, and baseline system.

We also strongly encourage your participation, if you use your own training corpus, your own sentence alignment, your own language model, or your own decoder.

Your submission report should highlight in which ways your own methods and data differ from the standard task. We may break down submitted results in different tracks, based on what resources were used.

Provided Data

The provided data is taken from a new release (version 3) of the Europarl corpus, which is freely available. Please click on the links below to download the sentence-aligned data, or go to the Europarl website for the source release. If you prepare training data from the Europarl corpus directly, please do not take data from Q4/2000 (October-December), since it is reserved for development and test data.

Additional training data is taken from the new News Commentary corpus. There are about 35-40 million words of training data per language from the Europarl corpus and 1 million words from the News Commentary corpus.

Europarl
  • French-English
  • Spanish-English
  • German-English
  • French monolingual
  • Spanish monolingual
  • German monolingual
  • English monolingual
News Commentary
  • French-English
  • Spanish-English
  • German-English
  • Czech-English

Note that in difference to previous years the released data is not tokenized and includes sentences of any length (including empty sentences). Also, this year all data is in Unicode (UTF-8) format. The following tools allow the processing of the training data into tokenized format:

Development Data

To tune your system during development, we provide development sets of 2000 sentences (Europarl) and 1057 sentences (News Commentary). The data is provided in raw text format and in an SGML format that suits the NIST scoring tool.

Europarl dev2006
  • English
  • French
  • Spanish
  • German
This data is identical with the 2005 development test data and the 2006 development data.
News Commentary nc-dev2007
  • English
  • French
  • Spanish
  • German
  • Czech
This is newly released data.

Development Test Data

To test your system during development, we provide development test sets of 2000 sentences (Europarl) and 1064 sentences (News Commentary -- the Czech-English set is slightly smaller). We encourage people submitting research publications to the workshop to report results on these sets. The data is provided in raw text format and in an SGML format that suits the NIST scoring tool.

Europarl devtest2006
  • English
  • French
  • Spanish
  • German
This data is identical with the 2005 test data and the 2006 development test data.
Europarl test2006
  • English
  • French
  • Spanish
  • German
This data is identical with the 2006 test data (in-domain part).
News Commentary nc-devtest2007
  • English
  • French
  • Spanish
  • German
  • Czech
This data is identical with the 2006 test data (out-of-domain part).

Download

The test data is part of the dev sets in future years, so please check recent WMT workshop sites for it.

Evaluation

Evaluation will be done both automatically as well as by human judgement.

Dates

March 21: Test data released
April 6: Results submissions (by email to pkoehn@inf.ed.ac.uk)
April 13: Short paper submissions (4 pages)

supported by the EuroMatrix project, P6-IST-5-034291-STP
funded by the European Commission under Framework Programme 6