WMT08 Preliminary Results and Human Eval Selection

Key
A system we should include in human scoring.
A system we haven't decided on yet.
Excluded system.

System Name BLEU NIST RATIO Notes Old Submission Path
cz-en nc-test2008
umd25.48 7.4440 0.952   submissions/umd/nc-test2008.trans.sgm
uedin23.01 6.7878 1.008   submissions/uedin/nc-test2008.cz-en.5.sgm
dcu22.17 6.6773 1.049   submissions/dcu/cz-en.DCU.nc-test.sgm
systran15.41 5.5805 0.990   submissions/systran/nc-test2008_czen_systransub.sgm
cz-en newstest2008
umd14.73 5.6624 0.870   submissions/umd/newstest2008.trans.sgm
dcu14.58 5.4556 0.993   submissions/dcu/cz-en.DCU.newstest.sgm
uedin14.47 5.5067 0.953   submissions/uedin/newstest2008.cz-en.5.sgm
de-en newstest2008
saar-contrast19.67 6.1464 1.022  Winner contrast. It looks from the filename that "gw" might mean gigaword? submissions/saar/hybrid-gw-newstest2008.de-en.en.sgm
limsi18.09 6.1140 0.941   submissions/limsi/LIMSI.newstest2008.de-en.en.sgm
uedin17.67 5.8667 0.998   submissions/uedin/newstest2008.de-en.50.sgm
rbmt417.52 5.8289 1.058   submissions/rbmt4/newstest2008.de-en.sgm
saar17.50 6.0347 0.951   submissions/saar/hybrid-tuned-newstest2008.de-en.en.sgm
saar-contrast-217.45 5.9859 0.964   submissions/saar/hybrid-untuned-newstest2008.de-en.en.sgm
rbmt316.88 5.7218 1.060   submissions/rbmt3/newstest2008.de-en.sgm
liu16.76 5.7412 1.011   submissions/liu/newstest2008-liu.en.sgm
rbmt216.71 5.8092 1.058   submissions/rbmt2/newstest2008.de-en.sgm
rbmt516.06 5.5915 1.067   submissions/rbmt5/newstest2008.de-en.sgm
rbmt615.17 5.4341 1.052   submissions/rbmt6/newstest2008.de-en.sgm
rbmt115.06 5.4962 0.952   submissions/rbmt1/newstest2008.de-en.sgm
cmu-statxfer11.87 4.7515 1.059   submissions/cmu-statxfer/newstest2008.en.cmu-statxfer-deen.sgm
de-en test2008
uedin29.07 7.2429 1.009   submissions/uedin/test2008.de-en.50.sgm
liu28.03 7.1394 0.995   submissions/liu/test2008-liu.en.sgm
limsi27.92 7.1389 0.993   submissions/limsi/LIMSI.test2008.de-en.en.sgm
saar27.78 7.0741 1.005   submissions/saar/hybrid-tuned-test2008.de-en.en.sgm
saar-contrast27.03 6.9711 0.993   submissions/saar/hybrid-untuned-test2008.de-en.en.sgm
ucl20.30 6.2672 0.851   submissions/ucl/test2008-out.de-en.sgm
cmu-statxfer18.12 5.7821 1.024   submissions/cmu-statxfer/test2008.en.cmu-statxfer-deen.sgm
rbmt316.39 5.4808 1.031   submissions/rbmt3/test2008.de-en.sgm
rbmt216.29 5.5498 1.020   submissions/rbmt2/test2008.de-en.sgm
rbmt515.67 5.3733 1.040   submissions/rbmt5/test2008.de-en.sgm
rbmt415.67 5.3812 1.041   submissions/rbmt4/test2008.de-en.sgm
rbmt614.07 5.0806 1.043   submissions/rbmt6/test2008.de-en.sgm
rbmt113.47 4.9868 0.877   submissions/rbmt1/test2008.de-en.sgm
de-es newstest2008
uedin16.60 5.6701 0.965   submissions/uedin/newstest2008.de-es.6.sgm
de-es test2008
uedin28.62 7.0193 1.005   submissions/uedin/test2008.de-es.6.sgm
uw-contrast21.18 5.7069 1.032   submissions/uw/test2008.UW.de-es.boost.output.es.sgm
uw21.17 5.7184 1.043   submissions/uw/test2008.UW.de-es.1best.output.es.sgm
en-cz nc-test2008
cu-bojar15.91 5.3768 1.025   submissions/cu-bojar/nc-mosesBIG.seg
cu-bojar-contrast-214.64 5.1475 1.031  Only 4 other entries here, should we include it? submissions/cu-bojar/nc-moses.seg
uedin12.96 4.8585 1.007   submissions/uedin/nc-test2008.en-cz.5.sgm
cu-tectomt9.28 4.5126 0.935   submissions/cu-tectomt/nc-test2008-tst.cz.sgm
pc-translator8.48 4.1051 1.010   submissions/pc-translator/nc-pct.seg
cu-bojar-contrast-14.98 3.4846 0.905   submissions/cu-bojar/nc-etct.seg
en-cz newstest2008
cu-bojar11.93 4.5566 1.073   submissions/cu-bojar/news-mosesBIG.seg
cu-bojar-contrast-29.75 4.1077 1.078  Only 4 other entries here, should we include it? submissions/cu-bojar/news-moses.seg
uedin9.64 4.0982 1.059   submissions/uedin/newstest2008.en-cz.5.sgm
pc-translator8.41 3.9583 1.033   submissions/pc-translator/news-pct.seg
cu-tectomt6.94 3.9325 0.965   submissions/cu-tectomt/newstest2008-tst.cz.sgm
cu-bojar-contrast-13.36 2.9608 0.919   submissions/cu-bojar/news-etct.seg
en-de newstest2008
rbmt414.24 5.0736 1.023   submissions/rbmt4/newstest2008.en-de.sgm
rbmt213.84 5.0259 1.040   submissions/rbmt2/newstest2008.en-de.sgm
saar13.83 5.2503 0.930   submissions/saar/hybrid-tuned-newstest2008.en-de.de.sgm
rbmt312.82 4.9071 0.992   submissions/rbmt3/newstest2008.en-de.sgm
rbmt112.59 4.7592 1.036   submissions/rbmt1/newstest2008.en-de.sgm
saar-contrast12.59 5.0849 0.931   submissions/saar/hybrid-untuned-newstest2008.en-de.de.sgm
limsi12.31 4.8699 1.030   submissions/limsi/LIMSI.newstest2008.en-de.de.sgm
uedin12.07 5.0305 0.975   submissions/uedin/newstest2008.en-de.30.sgm
liu11.60 5.0224 0.968   submissions/liu/newstest2008-liu.de.sgm
rbmt511.21 4.5870 1.037   submissions/rbmt5/newstest2008.en-de.sgm
rbmt610.70 4.5338 1.037   submissions/rbmt6/newstest2008.en-de.sgm
en-de test2008
uedin21.07 6.0962 0.993   submissions/uedin/test2008.en-de.30.sgm
saar20.96 6.0162 0.991   submissions/saar/hybrid-tuned-test2008.en-de.de.sgm
limsi20.65 6.0139 0.997   submissions/limsi/LIMSI.test2008.en-de.de.sgm
cmu-gimpel20.64 6.0614 0.995   submissions/cmu-gimpel/ende-europarl-cmu-gimpelsmith.sgm
liu20.55 6.1248 0.981   submissions/liu/test2008-liu.de.sgm
saar-contrast20.20 5.8834 1.000   submissions/saar/hybrid-untuned-test2008.en-de.de.sgm
ucl16.11 5.6062 0.889   submissions/ucl/test2008-out.en-de.sgm
rbmt412.39 4.5779 1.045   submissions/rbmt4/test2008.en-de.sgm
rbmt212.21 4.5674 1.062   submissions/rbmt2/test2008.en-de.sgm
rbmt111.33 4.3959 1.004   submissions/rbmt1/test2008.en-de.sgm
rbmt311.14 4.4013 1.025   submissions/rbmt3/test2008.en-de.sgm
rbmt59.80 4.1451 1.053   submissions/rbmt5/test2008.en-de.sgm
rbmt69.72 4.1212 1.064   submissions/rbmt6/test2008.en-de.sgm
en-es newstest2008
saar22.69 6.5781 0.953   submissions/saar/hybrid-tuned-newstest2008.en-es.es.sgm
saar-contrast22.24 6.4750 0.962   submissions/saar/hybrid-untuned-newstest2008.en-es.es.sgm
ucb21.92 6.3525 1.014   submissions/ucb/ucb_newstest2008.en2es.sgm
rbmt421.86 6.3221 1.011   submissions/rbmt4/newstest2008.en-es.sgm
limsi21.18 6.2320 1.007   submissions/limsi/LIMSI.newstest2008.en-es.es.sgm
uedin20.72 6.2909 0.968   submissions/uedin/newstest2008.en-es.6.sgm
cmu-smt20.17 6.1110 1.019   submissions/cmu-smt/CMU-SMT.newstest2008-src-en.sgm
rbmt320.12 6.1032 0.994   submissions/rbmt3/newstest2008.en-es.sgm
rbmt520.06 6.0291 1.035   submissions/rbmt5/newstest2008.en-es.sgm
rbmt620.03 6.1164 0.998   submissions/rbmt6/newstest2008.en-es.sgm
upc19.28 6.3106 0.914   submissions/upc/newstest2008.spa.xml
rbmt117.25 5.7592 1.015   submissions/rbmt1/newstest2008.en-es.sgm
en-es test2008
uedin33.12 7.6568 1.002   submissions/uedin/test2008.en-es.6.sgm
cmu-smt33.10 7.6306 1.008   submissions/cmu-smt/CMU-SMT.test2008-src-en.sgm
limsi32.82 7.6479 1.002   submissions/limsi/LIMSI.test2008.en-es.es.sgm
uw32.72 7.5865 1.014   submissions/uw/test2008.UW.en-es.1best.output.es.sgm
uw-contrast32.62 7.5865 1.012   submissions/uw/test2008.UW.en-es.boost.output.es.sgm
saar32.46 7.5289 1.009   submissions/saar/hybrid-tuned-test2008.en-es.es.sgm
saar-contrast31.49 7.4137 1.017   submissions/saar/hybrid-untuned-test2008.en-es.es.sgm
upc31.31 7.7285 0.939   submissions/upc/europarltest2008.spa.xml
ucl26.08 7.0809 0.901   submissions/ucl/test2008-out.en-es.sgm
rbmt421.83 6.2081 1.001   submissions/rbmt4/test2008.en-es.sgm
rbmt321.05 6.1208 0.987   submissions/rbmt3/test2008.en-es.sgm
rbmt520.76 5.9611 1.027   submissions/rbmt5/test2008.en-es.sgm
rbmt619.33 5.8588 1.007   submissions/rbmt6/test2008.en-es.sgm
rbmt118.71 5.8214 0.969   submissions/rbmt1/test2008.en-es.sgm
en-fr newstest2008
lium-systran21.43 6.2227 1.016   submissions/lium-systran/lium-systran_newstest2008_enfr_primary.sgm
lium-systran-contrast21.12 6.1787 1.027   submissions/lium-systran/lium-systran_newstest2008_enfr_contrastive.sgm
limsi20.95 6.0894 1.022   submissions/limsi/LIMSI.newstest2008.en-fr.fr.sgm
rbmt419.80 5.9526 1.012   submissions/rbmt4/newstest2008.en-fr.sgm
rbmt518.50 5.7862 1.022   submissions/rbmt5/newstest2008.en-fr.sgm
saar-contrast18.35 5.6933 1.001  Beats primary by more than 1 BLEU. submissions/saar/hybrid-untuned-newstest2008.en-fr.fr.sgm
uedin18.25 5.8786 0.981   submissions/uedin/newstest2008.en-fr.6.sgm
rbmt317.80 5.7294 0.990   submissions/rbmt3/newstest2008.en-fr.sgm
saar17.22 5.4345 1.035   submissions/saar/hybrid-tuned-newstest2008.en-fr.fr.sgm
rbmt616.69 5.6434 0.995   submissions/rbmt6/newstest2008.en-fr.sgm
xerox-contrast14.66 5.4428 0.975  Only .06 better than primary. Out. submissions/xerox/newstest2008-src.en.matrax_coupling.en_fr.sgm
xerox14.60 5.4517 0.971   submissions/xerox/newstest2008-src.en.matrax.en_fr.sgm
rbmt114.25 5.1876 0.900   submissions/rbmt1/newstest2008.en-fr.sgm
en-fr test2008
lium-systran32.91 7.6558 0.998   submissions/lium-systran/lium-systran_test2008_enfr_primary.sgm
limsi32.35 7.5642 1.000   submissions/limsi/LIMSI.test2008.en-fr.fr.sgm
lium-systran-contrast32.18 7.5728 0.999   submissions/lium-systran/lium-systran_test2008_enfr_contrastive.sgm
uedin31.11 7.3951 1.008   submissions/uedin/test2008.en-fr.6.sgm
saar-contrast28.24 6.8519 1.050  Beats primary by more tha 1 BLEU. submissions/saar/hybrid-untuned-test2008.en-fr.fr.sgm
saar26.18 6.4252 1.121   submissions/saar/hybrid-tuned-test2008.en-fr.fr.sgm
ucl24.36 6.8260 0.899   submissions/ucl/test2008-out.en-fr.sgm
rbmt421.33 6.1364 1.003   submissions/rbmt4/test2008.en-fr.sgm
rbmt319.53 5.8488 0.994   submissions/rbmt3/test2008.en-fr.sgm
rbmt618.13 5.7032 1.001   submissions/rbmt6/test2008.en-fr.sgm
rbmt116.61 5.2670 0.865   submissions/rbmt1/test2008.en-fr.sgm
rbmt512.47 3.6694 1.002   submissions/rbmt5/test2008.en-fr.sgm
en-hu newstest2008
uedin6.45 3.7142 1.021   submissions/uedin/newstest2008.en-hu.4.sgm
es-de newstest2008
uedin11.97 4.9028 1.014   submissions/uedin/newstest2008.es-de.5.sgm
es-de test2008
uedin20.96 6.1110 0.993   submissions/uedin/test2008.es-de.5.sgm
es-en newstest2008
cued-contrast22.88 6.5230 1.011  Highest score, but the filename says 'unconstrained'... submissions/cued/es-en-newstest2008.CUED_unconstrained.en.sgm
saar22.08 6.4399 1.011   submissions/saar/hybrid-tuned-newstest2008.es-en.en.sgm
saar-contrast21.51 6.2375 1.056   submissions/saar/hybrid-untuned-newstest2008.es-en.en.sgm
limsi21.32 6.6539 0.944   submissions/limsi/LIMSI.newstest2008.es-en.en.sgm
cued20.99 6.3083 1.006   submissions/cued/es-en-newstest2008.CUED_constrained.en.sgm
rbmt520.45 5.9668 1.101   submissions/rbmt5/newstest2008.es-en.sgm
ucb20.17 6.3973 0.969   submissions/ucb/ucb_newstest2008.es2en.sgm
uedin20.07 6.1142 1.022   submissions/uedin/newstest2008.es-en.7.sgm
upc19.61 6.4170 0.943   submissions/upc/newstest2008.eng.xml
rbmt419.08 5.7899 1.117   submissions/rbmt4/newstest2008.es-en.sgm
rbmt318.97 5.8045 1.104   submissions/rbmt3/newstest2008.es-en.sgm
cmu-smt18.95 6.3173 0.934   submissions/cmu-smt/CMU-SMT.newstest2008-src-es.sgm
rbmt618.60 5.7951 1.103   submissions/rbmt6/newstest2008.es-en.sgm
es-en test2008
limsi33.75 7.9063 0.994   submissions/limsi/LIMSI.test2008.es-en.en.sgm
cmu-smt33.62 7.8993 0.990   submissions/cmu-smt/CMU-SMT.test2008-src-es.sgm
uedin33.58 7.8278 1.000   submissions/uedin/test2008.es-en.7.sgm
cued-contrast33.55 7.9133 0.990  Higher score than primary, but by less than 1 BLEU. submissions/cued/es-en-test2008.CUED_unconstrained.en.sgm
saar33.16 7.8661 0.982   submissions/saar/hybrid-tuned-test2008.es-en.en.sgm
cued33.11 7.8590 0.993   submissions/cued/es-en-test2008.CUED_constrained.en.sgm
dcu32.85 7.7691 0.998   submissions/dcu/es-en.DCU.sgm
upc32.80 7.7406 1.007   submissions/upc/europarltest2008.eng.xml
saar-contrast31.07 7.3685 1.051   submissions/saar/hybrid-untuned-test2008.es-en.en.sgm
ucl26.13 7.2109 0.901   submissions/ucl/test2008-out.es-en.sgm
rbmt319.34 5.7517 1.079   submissions/rbmt3/test2008.es-en.sgm
rbmt419.04 5.6830 1.093   submissions/rbmt4/test2008.es-en.sgm
rbmt518.51 5.6620 1.078   submissions/rbmt5/test2008.es-en.sgm
rbmt618.03 5.6085 1.083   submissions/rbmt6/test2008.es-en.sgm
fr-en newstest2008
lium-systran-contrast21.91 6.6296 0.975  Winner contrast. Not by much, though.... submissions/lium-systran/lium-systran_newstest2008_fren_contrastive.sgm
lium-systran21.82 6.6325 0.972   submissions/lium-systran/lium-systran_newstest2008_fren_primary.sgm
cued-contrast21.28 6.3187 1.032  Beats primary by more than 1 BLEU. Unconstrained. submissions/cued/fr-en-newstest2008.CUED_unconstrained.en.sgm
limsi20.98 6.5551 0.969   submissions/limsi/LIMSI.newstest2008.fr-en.en.sgm
saar-contrast20.25 6.0649 1.065  Beats primary by more than 1 BLEU. submissions/saar/hybrid-untuned-newstest2008.fr-en.en.sgm
cued19.58 6.1082 1.033   submissions/cued/fr-en-newstest2008.CUED_constrained.en.sgm
uedin19.16 5.9742 1.046   submissions/uedin/newstest2008.fr-en.7.sgm
rbmt518.55 5.8103 1.098   submissions/rbmt5/newstest2008.fr-en.sgm
saar17.76 6.1512 0.916   submissions/saar/hybrid-tuned-newstest2008.fr-en.en.sgm
rbmt417.53 5.6776 1.073   submissions/rbmt4/newstest2008.fr-en.sgm
rbmt317.07 5.4593 1.131   submissions/rbmt3/newstest2008.fr-en.sgm
rbmt616.78 5.5123 1.110   submissions/rbmt6/newstest2008.fr-en.sgm
cmu-statxfer15.08 5.2398 1.121   submissions/cmu-statxfer/newstest2008.en.cmu-statxfer-fren-1.sgm
cmu-statxfer-contrast14.66 5.1660 1.122   submissions/cmu-statxfer/newstest2008.en.cmu-statxfer-fren-2.sgm
fr-en test2008
lium-systran33.94 7.9220 0.997   submissions/lium-systran/lium-systran_test2008_fren_primary.sgm
lium-systran-contrast33.50 7.8558 1.001   submissions/lium-systran/lium-systran_test2008_fren_contrastive.sgm
uedin33.46 7.8159 1.003   submissions/uedin/test2008.fr-en.7.sgm
limsi33.39 7.8781 0.994   submissions/limsi/LIMSI.test2008.fr-en.en.sgm
cued-contrast33.11 7.8766 0.991  Doesn't beat primary by more than 1 BLEU. submissions/cued/fr-en-test2008.CUED_unconstrained.en.sgm
cued32.83 7.7988 1.000   submissions/cued/fr-en-test2008.CUED_constrained.en.sgm
dcu31.74 7.6994 0.991   submissions/dcu/fr-en.DCU.sgm
saar-contrast30.79 7.3282 1.055  Beat primary by more than 1 BLEU. submissions/saar/hybrid-untuned-test2008.fr-en.en.sgm
saar28.20 7.2215 0.850   submissions/saar/hybrid-tuned-test2008.fr-en.en.sgm
systran27.55 7.0163 1.004   submissions/systran/test2008_fren_systransub.sgm
ucl26.52 7.2263 0.926   submissions/ucl/test2008-out.fr-en.sgm
rbmt521.41 6.1650 1.071   submissions/rbmt5/test2008.fr-en.sgm
cmu-statxfer-contrast21.04 6.2139 1.061  Doesn't beat primary by more than 1 BLEU. submissions/cmu-statxfer/test2008.en.cmu-statxfer-fren-2.sgm
cmu-statxfer20.50 6.1032 1.070   submissions/cmu-statxfer/test2008.en.cmu-statxfer-fren-1.sgm
rbmt419.04 5.8271 1.045   submissions/rbmt4/test2008.fr-en.sgm
rbmt318.24 5.5841 1.096   submissions/rbmt3/test2008.fr-en.sgm
rbmt617.20 5.4800 1.077   submissions/rbmt6/test2008.fr-en.sgm