Goals

• To investigate the applicability of current MT techniques when translating into languages other than English
• To examine special challenges in translating between European languages, including word order differences and morphology
• To create publicly available corpora for machine translation and machine translation evaluation
• To generate up-to-date performance numbers for European languages in order to provide a basis of comparison in future research
• To offer newcomers a smooth start with hands-on experience in state-of-the-art statistical machine translation methods

New Challenges This Year

• News Translation: Besides the traditional Europarl task, we will present a test set of news stories as a test set. These news stories will be taken from major news outlets as the BBC, Der Spiegel, Le Monde, etc. during the time period of November-December 2007. Due to time constraints, we will not be able to provide a development set.
• Hungarian: For the first time, we will include the challenge of translation from and into Hungarian.
• Non-English Language Pair: We include one language pair, Spanish-German and back, that does not involve English either as source or target.

Task Description

• improve word alignment quality, phrase extraction, phrase scoring
• add new components to the open source software of the baseline system
• augment the system otherwise (e.g. by preprocessing, reranking, etc.)
• build an entirely new translation systems

You may participate in any or all of the following language pairs:

• French-English
• Spanish-English
• German-English
• Spanish-German
• Czech-English
• Hungarian-English

We also strongly encourage your participation, if you use your own training corpus, your own sentence alignment, your own language model, or your own decoder.

Your submission report should highlight in which ways your own methods and data differ from the standard task. We may break down submitted results in different tracks, based on what resources were used. We are mostly interested in submission that are constraint to the provided training data, so that the comparison is focused on the methods, not on the data used. You may submit contrastive runs to demonstrate the benefit of additional training data.

Training Data

Additional training data is taken from the new News Commentary corpus. There are about 35-40 million words of training data per language from the Europarl corpus and 1 million words from the News Commentary corpus.

Europarl French-English Spanish-English German-English Spanish-German French monolingual Spanish monolingual German monolingual English monolingual

News Commentary French-English Spanish-English German-English Czech-English

Hunglish Hungarian-English The corpus was created as part of the hunglish project by the joint work of the Media Research and Education Center at the Budapest University of Technology and Economics and the Corpus Linguistics Department at the Hungarian Academy of Sciences Institute of Linguistics.

CzEng Czech-English You will need to to download the current version of the CzEng corpus (version v0.7) from the CzEng web site. Please re-download the data, if you accessed it before February 5, 2008.

Note that in difference to previous years the released data is not tokenized and includes sentences of any length (including empty sentences). Also, this year all data is in Unicode (UTF-8) format. The following tools allow the processing of the training data into tokenized format:

• Tokenizer tokenizer.perl
• Detokenizer detokenizer.perl
• Lowercaser lowercase.perl
• SGML Wrapper wrap-xml.perl

Development Data

Europarl dev2006 English French Spanish German This data is identical with the 2005 development test data and the 2006 development data.

News Commentary nc-dev2007 English French Spanish German Czech This is identical with the 2007 development data.

Hunglish hung-dev2008 Hungarian English Note that this dev set is extracted from the official Hunglish corpus. The training corpus we provide (above) does not overlap with the dev set.

Overview of Development and Test Data

Europarl Development Data

Other Development Data

Test Data

• Parallel corpus training data (631 MB) Updated Feb 9 for minor problems
• Monolingual language model training data (276 MB)
• Development sets (11 MB)
• Tools (3 KB)
• Test (1.5 MB)

Evaluation

• Manual Scoring: We will collect subjective judgments about translation quality from human annotators. If you participate in the evaluation, we ask you to commit about 8 hours of time to do the manual evaluation. The evaluation will be done with an online tool.
• As last year, we expect the translated submissions to be in recased, detokenized, XML format, just as in most other translation campaigns (NIST, TC-Star). The official BLEU scoring tool will be the NIST scoring tool.
• We will also be applying other automatic evaluation metrics to submissions, and will be reporting results about how well they correlate with human judgments. If you would like to have your evaluation metric tested in this way please contact Chris Callison-Burch.

Dates

supported by the EuroMatrix project, P6-IST-5-034291-STP funded by the European Commission under Framework Programme 6