Shared Task: Metrics

Metrics Task Important Dates

System outputs ready to downloadMay 10th, 2019
Start of manual evaluation periodMay 15th, 2019
Paper submission deadlineMay 17th, 2019 (indeed earlier than the final submission of your scores)
Submission deadline for metrics taskMay 25th, 2019 (AoE)
End of manual evaluationMay 27th, 2019 (or longer if struggling for confidence)
Notification of acceptanceJune 7th, 2019
Camera-ready deadlineJune 17th, 2019
Conference in FlorenceAugust 1st—2nd, 2019

Metrics Task Overview

This shared task will examine automatic evaluation metrics for machine translation. We will provide you with all of the translations produced in the translation task along with the human reference translations. You will return your automatic metric scores for translations at the system-level and/or at the sentence-level. We will calculate the system-level and sentence-level correlations of your scores with WMT19 human judgements once the manual evaluation has been completed.

Goals

The goals of the shared metrics task are:

Details Recorded

Submissions to this year's metrics task should include in each submission:

Since 2016, the system-level evaluation includes evaluation of metrics against large sets of references (10k synthetic, "hybrid" MT systems). If your system-level metric is not a simple arithmetic average of segment-level scores and it is not terribly computationally expensive, please provide also your scores for the 10k hybrid MT systems.

This year, there are no explicitly labelled additional domain but the German-French pair is targeted at EU elections and the "Test suites" part of newstest2019 does contain various domains for some language pairs.

Task Description

We will provide you with the output of machine translation systems and reference translations for the following language pairs:

Additionally, Quality Estimation Task 3 "QE as a metric" runs joinly with the metrics task. For "QE as a metric", you need to provide the same outputs as standard metrics participants (see below) but you must not make use of the references.

You will compute scores for each of the outputs at the system-level and/or the sentence-level. If your automatic metric does not produce sentence-level scores, you can participate in just the system-level ranking. If your automatic metric uses linguistic annotation and supports only some language pairs, you are free to assign scores only where you can.

We will assess automatic evaluation metrics in the following ways:

Summary of Tracks

The following table summarizes the planned evaluation methods and text domains of each evaluation track.

Track Text Domain Level Golden Truth Source
DAsys news, from WMT19 news task system-level direct assessment
DAseg news, from WMT19 news task segment-level direct assessment

Other Requirements

If you participate in the metrics task, we ask you to commit about 8 hours of time to do the manual evaluation. You are also invited to submit a paper describing your metric.

Manual Evaluation

The evaluation will be done with an online tool, details will be posted here.

Paper Describing Your Metric

You are invited to submit a short paper (4 to 6 pages) describing your automatic evaluation metric. You are not required to submit a paper if you do not want to. If you don't, we ask that you give an appropriate reference describing your metric that we can cite in the overview paper.

Download

Test Sets (Evaluation Data)

WMT19 metrics task test sets are ready, apologies for the delay.

There are three subsets of outputs that we would like you to evaluate:

newstest2019
This is the very basis of the metrics task, segment-level evaluation of MT outputs.
testsuites
These are the additional sets of sentences translated by WMT19 translation systems to allow detailed inspection of system's (linguistic) properties. There will be no manual evaluations collected for these translations, but on the other hand, your automatic scoring will help the testsuite authors to interpret the performance of MT systems on their testsuite. We would like you to score these.
hybrids
To establish better confidence intervals for system-level evaluation, we artificially create more than 10k system outputs per language pair and test set. You need to evaluate hybrids only if you system-level score is not a simple average of segment-level scores. We are not distributing hybrids this year upfront, contact us if you need them.

The package of inputs for you to evaluate this year comes in three versions (the directory layout and naming convention this year match the main translation task):

Here is a bash script that you may want to run around your scorer to process everything:

wget http://ufallab.ms.mff.cuni.cz/~bojar/wmt19/wmt19-submitted-data-v3-txt.tgz
tar xzf wmt19-submitted-data-v3-txt.tgz
cd wmt19-submitted-data-v3/txt-ts
for testset in `ls -d system-outputs/* | cut -d/ -f2`; do
  for lp in `ls -d system-outputs/$testset/* | cut -d/ -f3`; do
    echo "PROCESSING TESTSET $testset, LANGUAGE_PAIR $lp"
    ref=references/$testset-${lp:0:2}${lp:3:5}-ref-ts.${lp:3:5}
    src=sources/$testset-${lp:0:2}${lp:3:5}-src-ts.${lp:0:2}
    echo "  REF: $ref  SRC: $src"
    for hyp in system-outputs/$testset/$lp/*; do
      echo "  EVALUATING $hyp"
      <YOUR EVALUATION TOOL> --reference=$ref --hypothesis=$hyp --source=$src
    done
  done
done

Training Data

You may want to use some of the following data to tune or train your metric.

DA (Direct Assessment) Development/Training Data

For system-level, see the results from the previous years:

For segment-level, the following datasets are available:

Each dataset contains:

RR (Relative Ranking) from Past Years

Although RR is no longer the manual evaluation employed in the metrics task, human judgments from the previous year's data sets may still prove useful:

You can use any past year's data to tune your metric's free parameters if it has any for this year's submission. Additionally, you can use any past data as a test set to compare the performance of your metric against published results from past years metric participants.

Last year's data contains all of the system's translations, the source documents and human reference translations and the human judgments of the translation quality.

Submission Format

The output of your software should produce scores for the translations either at the system-level or the segment-level (or preferably both).

Output file format for system-level rankings

Since we assume that your metrics are mostly simple arithmetic averages of segment-level scores, your system-level outputs serve primarily as a sanity check if we get the exact same averages.

The output files for system-level rankings should be called YOURMETRIC.sys.score.gz and formatted in the following way: 

<METRIC NAME>   <LANG-PAIR>   <TEST SET>   <SYSTEM>   <SYSTEM LEVEL SCORE>   <ENSEMBLE>   <AVAILABLE>
Where: Each field should be delimited by a single tab character.

(This year, we no longer collect the timing information.)

Output file format for segment-level rankings

The output files for segment-level rankings should be called YOURMETRIC.seg.score.gz and formatted in the following way: 

<METRIC NAME>   <LANG-PAIR>   <TEST SET>   <SYSTEM>   <SEGMENT NUMBER>   <SEGMENT SCORE> <ENSEMBLE>   <AVAILABLE>
Where: Each field should be delimited by a single tab character.

Note: fields ENSEMBLE and AVAILABLE should be filled with the same value in every line of the submission file for a given metric. Inclusion in this format involves some redundancy but avoids adding extra files to the submission requirements.

How to submit

Submissions should be sent as an e-mail to wmt-metrics-submissions@googlegroups.com.

As a sanity check, please enter yourself to this shared spreadsheet.

In case the above e-mail doesn't work for you (Google seems to prevent non-member postings despite we set it so), please contact us directly.

Metrics Task Organizers

Ondřej Bojar (Charles University)
Yvette Graham (Dublin City University)
Qingsong Ma (Tencent Inc.)
Johnny Wei (University of Massachusetts Amherst)