Shared Task: Metrics

Metrics Task Important Dates

System outputs ready to downloadJune 3rd, 2018 June 22th, 2018
Start of manual evaluation periodJune 11th, 2018
End of manual evaluationJuly 2nd, 2018 (tentative)
Submission deadline for metrics taskJuly 2nd, 2018 July 15th, 2018 (AoE)
Paper submission deadlineJuly 27th, 2018
Notification of acceptanceAugust 18th, 2018
Camera-ready deadlineAugust 31th, 2018
Conference in BrusselsOctober 31st—November 1st, 2018

Metrics Task Overview

This shared task will examine automatic evaluation metrics for machine translation. We will provide you with all of the translations produced in the translation task along with the human reference translations. You will return your automatic metric scores for translations at the system-level and/or at the sentence-level. We will calculate the system-level and sentence-level correlations of your scores with WMT18 human judgements once the manual evaluation has been completed.

Goals

The goals of the shared metrics task are:

Details Recorded

Submissions to this year's metrics task should include in each submission:

Since 2016, the system-level evaluation includes evaluation of metrics against large sets of references (10k synthetic, "hybrid" MT systems). If your system-level metric is not terribly computationally expensive, please provide also your scores for the 10k hybrid MT systems.

This year, there are no additional domains (e.g. the medical domain last year).

Task Description

We will provide you with the output of machine translation systems and reference translations for language pairs involving English and the following languages

You will compute scores for each of the outputs at the system-level and/or the sentence-level. If your automatic metric does not produce sentence-level scores, you can participate in just the system-level ranking. If your automatic metric uses linguistic annotation and supports only some language pairs, you are free to assign scores only where you can.

We will assess automatic evaluation metrics in the following ways:

Summary of Tracks

The following table summarizes the planned evaluation methods and text domains of each evaluation track.

Track Text Domain Level Golden Truth Source
DAsys news, from WMT18 news task system-level direct assessment
DAseg news, from WMT18 news task segment-level direct assessment

Other Requirements

If you participate in the metrics task, we ask you to commit about 8 hours of time to do the manual evaluation. You are also invited to submit a paper describing your metric.

Manual Evaluation

To take part in the manual evaluation, please sign up yourself for some "accounts" listed in this Google sheet by entering your name into the column "Group". (We particularly need evaluation out of English.)

Each such account contains 2x100 sentences to evaluate and takes approximately 1 hour to complete. Ideally, you should thus complete 8 accounts.

To access the annotation, use this URL pattern:

Make sure that after following this URL, the upper right corner on the web pages says the correct USERNAME. Browser cookies tend to keep the previous account logged in, in which case you need to click the wrong username on that page and select "Sign out" from the drop-down menu. The URL should then log you in with the correct account.

Paper Describing Your Metric

You are invited to submit a short paper (4 to 6 pages) describing your automatic evaluation metric. You are not required to submit a paper if you do not want to. If you don't, we ask that you give an appropriate reference describing your metric that we can cite in the overview paper.

Download

Test Sets (Evaluation Data)

WMT18 metrics task test sets are ready, apologies for the delay.

There are three subsets of outputs that we would like you to evaluate:

newstest2018
This is the basis of the metrics task, segment-level evaluation of MT outputs.
testsuites
These are the additional sets of sentences translated by WMT18 translation systems to allow detailed inspection of system's (linguistic) properties. There will be no manual evaluations collected for these translations, but on the other hand, your automatic scoring will help the testsuite authors to interpret the performance of MT systems on their testsuite. We would like you to score these.
hybrids
Since we are trying to establish better confidence intervals for system-level evaluation, we artificially create more than 10k system outputs per language pair and test set. You need to evaluate hybrids if you system-level score is not a simple average of segment-level scores.

The package of inputs for you to evaluate thus comes in three versions:

Here is a bash script that you may want to run around your scorer to process everything:

cd wmt18-metrics-task-nohybrids
for testset in `ls -d system-outputs/* | cut -d/ -f2`; do
  for lp in `ls -d system-outputs/$testset/* | cut -d/ -f3`; do
    echo "PROCESSING TESTSET $testset, LANGUAGE_PAIR $lp"
    ref=references/$testset-${lp:0:2}${lp:3:5}-ref.${lp:3:5}s
    src=sources/$testset-${lp:0:2}${lp:3:5}-src.${lp:0:2}
    echo "  REF: $ref  SRC: $src"
    for hyp in system-outputs/$testset/$lp/*; do
      echo "  EVALUATING $hyp"
      <YOUR EVALUATION TOOL> --reference=$ref --hypothesis=$hyp --source=$src
    done
  done
done

Training Data

You may want to use some of the following data to tune or train your metric.

DA (Direct Assessment) Development/Training Data

For system-level, see the results from the previous years:

For segment-level, the following datasets are available:

Each dataset contains:

RR (Relative Ranking) from Past Years

Although RR is no longer the manual evaluation employed in the metrics task, human judgments from the previous year's data sets may still prove useful:

You can use any past year's data to tune your metric's free parameters if it has any for this year's submission. Additionally, you can use any past data as a test set to compare the performance of your metric against published results from past years metric participants.

Last year's data contains all of the system's translations, the source documents and human reference translations and the human judgments of the translation quality.

Submission Format

The output of your software should produce scores for the translations either at the system-level or the segment-level (or preferably both).

Output file format for system-level rankings

The output files for system-level rankings should be called YOURMETRIC.sys.score.gz and formatted in the following way: 

<METRIC NAME>   <LANG-PAIR>   <TEST SET>   <SYSTEM>   <SYSTEM LEVEL SCORE>   <ENSEMBLE>   <AVAILABLE>
Where: Each field should be delimited by a single tab character.

(This year, we no longer collect the timing information.)

Output file format for segment-level rankings

The output files for segment-level rankings should be called YOURMETRIC.seg.score.gz and formatted in the following way: 

<METRIC NAME>   <LANG-PAIR>   <TEST SET>   <SYSTEM>   <SEGMENT NUMBER>   <SEGMENT SCORE> <ENSEMBLE>   <AVAILABLE>
Where: Each field should be delimited by a single tab character.

Note: fields ENSEMBLE and AVAILABLE should be filled with the same value in every line of the submission file for a given metric. Inclusion in this format involves some redundancy but avoids adding extra files to the submission requirements.

How to submit

Submissions should be sent as an e-mail to wmt-metrics-submissions@googlegroups.com.

As a sanity check, please enter yourself to this shared spreadsheet.

In case the above e-mail doesn't work for you (Google seems to prevent non-member postings despite we set it so), please contact us directly.

Metrics Task Organizers

Ondřej Bojar (Charles University)
Yvette Graham (Dublin City University)
Qingsong Ma (University of Chinese Academy of Sciences)