Motivation

Task

Terminology

Unfortunately, the notion of domain is ambiguous in NLP applications. To avoid confusion we will instead use the term webdomain to refer to content from a specific website, e.g, "This page is from the statmt.org webdomain". We distinguish between webdomains using their Fully Qualified Domain Name (FQDN). Thus, www.example.com and example.com are considered to be different webdomains.

We will use source to denote English pages and target for French ones. This does not imply that translation was performed in that direction. In fact we cannot know if translation from one side to the other was performed at all, both sides could possibly be translations of a third language document.

Training data

As training data we provide a set of 1,624 EN-FR pairs from 49 webdomains. The number of pairs per webdomain varies between 4 and over 200. All pairs are from within a single webdomain, possible matches between two different webdomains, e.g. siemens.de and siemens.com, are not considered in this task.

We also provide mirrors of all the pages in the webdomains which were crawled using httrack.

All crawls in a single file: lett.train.tgz (8.7G, md5:81b99f1a95a5153009bfd902b99857ab , sha1:f05754d63ba9b3fbc1b8e82bdbb45c3c950be129)

webdomain	source pages	target pages	possible pairings	training pairs
bugadacargnel.com.lett.gz	919	779	715,901	19
cbsc.ca	1,595	904	1,441,880	20
cineuropa.mobi	23,050	15,972	368,154,600	73
creationwiki.org	8,417	203	1,708,651	22
eu2007.de	3,201	2,488	7,964,088	11
eu.blizzard.com	10,493	6,640	69,673,520	10
forcesavenir.qc.ca	3,592	3,982	14,303,344	8
galacticchannelings.com	4,231	1,283	5,428,373	9
golftrotter.com	377	361	136,097	8
iiz-dvv.de	1,160	894	1,037,040	67
ironmaidencommentary.com	6,028	635	3,827,780	41
kicktionary.de	2,752	888	2,443,776	29
kustu.com	1,544	1,511	2,332,984	13
manchesterproducts.com	15,621	9,651	150,758,271	10
minelinks.com	736	212	156,032	66
pawpeds.com	983	135	132,705	19
rehazenter.lu	201	317	63,717	16
santabarbara-online.com	1,151	1,099	1,264,949	11
schackportalen.nu	33	29	957	14
tsb.gc.ca	5,885	5,828	34,297,780	236
virtualhospice.ca	43,500	22,327	971,224,500	46
www.acted.org	3,333	2,431	8,102,523	21
www.antennas.biz	812	327	265,524	30
www.artsvivants.ca	5,487	1,368	7,506,216	12
www.bonnke.net	414	129	53,406	27
www.bugadacargnel.com	919	779	715,901	7
www.cgfmanet.org	9,241	6,260	57,848,660	25
www.cyberspaceministry.org	1,534	958	1,469,572	29
www.dakar.com	17,420	14,582	254,018,440	45
www.dfo-mpo.gc.ca	25,277	19,087	482,462,099	97
www.ec.gc.ca	12,266	15,404	188,945,464	26
www.eohu.ca	2,277	2,136	4,863,672	4
www.eu2005.lu	5,649	5,704	32,221,896	34
www.eu2007.de	3,249	2,535	8,236,215	11
www.fao.org	11,931	5,004	59,702,724	6
www.inst.at	3,203	543	1,739,229	62
www.krn.org	115	115	13,225	67
www.lameca.org	692	1,567	1,084,364	6
www.luontoportti.com	3,645	1,796	6,546,420	30
www.nato.int	40,063	8,773	351,472,699	36
www.nauticnews.com	24,325	43,045	1,047,069,625	21
www.pawpeds.com	1,011	136	137,496	43
www.prohelvetia.ch	5,209	4,421	23,028,989	7
www.socialwatch.org	13,803	2,419	33,389,457	21
www.summerlea.ca	434	338	146,692	58
www.the-great-adventure.fr	2,038	2,460	5,013,480	18
www.ushmm.org	10,472	967	10,126,424	26
www.usw.ca.lett	5,006	2,247	11,248,482	83
www.vinci.com	3,564	3,374	12,024,936	24

Data format

The training pairs are one pair per line:

Source_URL<TAB>Target_URL\n

For the crawled data we provide one file per webdomain in .lett format adapted from Bitextor. This is a plain text format with one line per page. Each line consists of 6 tab-separated values:

• Language ID (e.g. en)
• Mime type (always text/html)
• Encoding (always charset=utf-8)
• URL
• HTML in Base64 encoding
• Text in Base64 encoding

We make sure that the language id is reliable, at least for the documents in the train and test pairs. We also ensure that all known pairs have been crawled and no URLs are missing from the crawls.

Text extraction was performed using an HTML5 parser. As the original HTML pages are available, participants are welcome to implement their own text extraction, for example to remove boilerplate.

To facilitate use of the .lett files we provide a simple reader class in Python: lett.py.

Additionally, we have identified spans of French text for which we produced English translations using MT. These translations are not part of the lett files but provided seperately: translations.train.gz. The format for the source segments and target segments is

URL<TAB>Text

Test data

For testing, we will provide 203 additional crawls of new webdomains, distinct from the ones in the training data in the same format. The official test pairs were released after the completion of the evaluation campaign.

All test lett files in one file: lett.test.tgz ((16G, md5:a16f7afdcf7de3c4bc992f0451ef89a3 , sha1:6b978ee34c8803876e0dad5b760eccc3957e3a5e)
As for the train data we provide translations of all French text spans: translations.test.tgz

Test set statistics:

Evaluation

Our main metric is recall on the test set, i.e. what percentage of the test-set pairs is found in a submission after enforcing the 1-1 rule above.

Participants are expected to produce a list of possible pairings in the format of the training data. Each source url may be matched with at most one target url and vice-versa. Should a URL occur repeatedly, later occurrences are ignored. We provide an evaluation script to assess performance during development.

Submission [NEW May 1st]

Baseline

We provide a simple baseline based on URL matching. Get the code from github:

git clone https://github.com/christianbuck/wmt16-document-alignment-task.git

The baseline iterates through all URLs and strips language identifiers such as /english/ or ?lang=FR from URLs and then produces pairs of URLs that have the same stripped representation. Check test_languagestripper.py for some examples.

$ ./baseline.sh 
Read 222509/345106/224267 fr/en/other URLs from stdin
Read 1624 url pairs from train.pairs
0 urls missing from candidate URLs
160219/229406 stripped source/target urls
Found 8144 stripped source + unmodified target pairs (total: 8144) covering 107 pairs from devset
Found 9119 stripped target + unmodified source pairs (total: 17263) covering 121 pairs from devset
Found 126588 stripped source + stripped target pairs (total: 143851) covering 1131 pairs from devset
Total: 143851 candidate pairs
Keeping 119979 pairs after enforcing 1-1 rule
1103 pairs from devset

Running eval

Read 1624 reference pairs from train.pairs
Read 119979 predicted pairs from predicted.pairs
Keeping 119979 pairs after enforcing 1-1 rule
Found 1103 (67.92%) pairs from reference

Important dates

Organisers

Sponsor