Wednesday, December 6, 2023 | |
| 8:45–9:00 | Opening Remarks |
| 9:00-10:30 | Session 1: Shared Task Overview Papers I |
| 9:00–9:30 | Findings of the 2023 Conference on Machine Translation (WMT23): LLMs Are Here but Not Quite There Yet Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Philipp Koehn, Benjamin Marie, Christof Monz, Makoto Morishita, Kenton Murray, Makoto Nagata, Toshiaki Nakazawa, Martin Popel, Maja Popović and Mariya Shmatova |
| 9:30–9:45 | Findings of the WMT 2023 Biomedical Translation Shared Task: Evaluation of ChatGPT 3.5 as a Comparison System Mariana Neves, Antonio Jimeno Yepes, Aurélie Névéol, Rachel Bawden, Giorgio Maria Di Nunzio, Roland Roller, Philippe Thomas, Federica Vezzani, Maika Vicente Navarro, Lana Yeganova, Dina Wiemann and Cristian Grozea |
| 9:45–10:00 | Findings of the WMT 2023 Shared Task on Discourse-Level Literary Translation: A Fresh Orb in the Cosmos of LLMs Longyue Wang, Zhaopeng Tu, Yan Gu, Siyou Liu, Dian Yu, Qingsong Ma, Chenyang Lyu, Liting Zhou, Chao-Hong Liu, Yufeng Ma, Weiyu Chen, Yvette Graham, Bonnie Webber, Philipp Koehn, Andy Way, Yulin Yuan and Shuming Shi |
| 10:00–10:15 | Findings of the Second WMT Shared Task on Sign Language Translation (WMT-SLT23) Mathias Müller, Malihe Alikhani, Eleftherios Avramidis, Richard Bowden, Annelies Braffort, Necati Cihan Camgöz, Sarah Ebling, Cristina España-Bonet, Anne Göhring, Roman Grundkiewicz, Mert Inan, Zifan Jiang, Oscar Koller, Amit Moryossef, Annette Rios, Dimitar Shterionov, Sandra Sidler-Miserez, Katja Tissi and Davy Van Landuyt |
| 10:15–10:30 | Findings of the WMT 2023 Shared Task on Parallel Data Curation Steve Sloto, Brian Thompson, Huda Khayrallah, Tobias Domhan, Thamme Gowda and Philipp Koehn |
| 10:30–11:00 | Coffee Break |
| 11:00–12:30 | Session 2: Shared Task System Description Posters I |
| General Translation Task | |
| 11:00–12:30 | Samsung R&D Institute Philippines at WMT 2023 Jan Christian Blaise Cruz |
| 11:00–12:30 | NAIST-NICT WMT’23 General MT Task Submission Hiroyuki Deguchi, Kenji Imamura, Yuto Nishida, Yusuke Sakai, Justin Vasselli and Taro Watanabe |
| 11:00–12:30 | CUNI at WMT23 General Translation Task: MT and a Genetic Algorithm Josef Jon, Martin Popel and Ondřej Bojar |
| 11:00–12:30 | SKIM at WMT 2023 General Translation Task Keito Kudo, Takumi Ito, Makoto Morishita and Jun Suzuki |
| 11:00–12:30 | KYB General Machine Translation Systems for WMT23 Ben LI, Yoko Matsuzaki and Shivam Kalkar |
| 11:00–12:30 | Yishu: Yishu at WMT2023 Translation Task Luo Min, yixin tan and Qiulin Chen |
| 11:00–12:30 | PROMT Systems for WMT23 Shared General Translation Task Alexander Molchanov and Vladislav Kovalenko |
| 11:00–12:30 | AIST AIRC Submissions to the WMT23 Shared Task Matiss Rikters and Makoto Miwa |
| 11:00–12:30 | MUNI-NLP Submission for Czech-Ukrainian Translation Task at WMT23 Pavel Rychly and Yuliia Teslia |
| 11:00–12:30 | Exploring Prompt Engineering with GPT Language Models for Document-Level Machine Translation: Insights and Findings Yangjian Wu and Gang Hu |
| 11:00–12:30 | Treating General MT Shared Task as a Multi-Domain Adaptation Problem: HW-TSC’s Submission to the WMT23 General MT Shared Task Zhanglin Wu, Daimeng Wei, Zongyao Li, Zhengzhe YU, Shaojun Li, Xiaoyu Chen, Hengchao Shang, Jiaxin GUO, Yuhao Xie, Lizhi Lei, Hao Yang and Yanfei Jiang |
| 11:00–12:30 | UvA-MT’s Participation in the WMT 2023 General Translation Shared Task Di Wu, Shaomu Tan, David Stap, Ali Araabi and Christof Monz |
| 11:00–12:30 | Achieving State-of-the-Art Multilingual Translation Model with Minimal Data and Parameters Hui Zeng |
| 11:00–12:30 | IOL Research Machine Translation Systems for WMT23 General Machine Translation Shared Task Wenbo Zhang |
| 11:00–12:30 | GTCOM and DLUT’s Neural Machine Translation Systems for WMT23 Hao Zong |
| Test Suites | |
| 11:00–12:30 | RoCS-MT: Robustness Challenge Set for Machine Translation Rachel Bawden and Benoît Sagot |
| 11:00–12:30 | Multifaceted Challenge Set for Evaluating Machine Translation Performance Xiaoyu Chen, Daimeng Wei, Zhanglin Wu, Ting Zhu, Hengchao Shang, Zongyao Li, Jiaxin GUO, Ning Xie, Lizhi Lei, Hao Yang and Yanfei Jiang |
| 11:00–12:30 | Linguistically Motivated Evaluation of the 2023 State-of-the-art Machine Translation: Can ChatGPT Outperform NMT? Shushen Manakhimova, Eleftherios Avramidis, Vivien Macketanz, Ekaterina Lapshinova-Koltunski, Sergei Bagdasarov and Sebastian Möller |
| 11:00–12:30 | IIIT HYD’s Submission for WMT23 Test-suite Task Ananya Mukherjee and Manish Shrivastava |
| 11:00–12:30 | Test Suites Task: Evaluation of Gender Fairness in MT with MuST-SHE and INES Beatrice Savoldi, Marco Gaido, Matteo Negri and Luisa Bentivogli |
| Biomedical Translation Task | |
| 11:00–12:30 | Biomedical Parallel Sentence Retrieval Using Large Language Models Sheema Firdous and Sadaf Abdul Rauf |
| 11:00–12:30 | The Path to Continuous Domain Adaptation Improvements by HW-TSC for the WMT23 Biomedical Translation Shared Task Zhanglin Wu, Daimeng Wei, Zongyao Li, Zhengzhe YU, Shaojun Li, Xiaoyu Chen, Hengchao Shang, Jiaxin GUO, Yuhao Xie, Lizhi Lei, Hao Yang and Yanfei Jiang |
| 11:00–12:30 | Investigating Techniques for a Deeper Understanding of Neural Machine Translation (NMT) Systems through Data Filtering and Fine-tuning Strategies Lichao Zhu, Maria Zimina, Maud Bénard, Behnoosh Namdar, Nicolas Ballier, Guillaume Wisniewski and Jean-Baptiste Yunès |
| Literary Translation Task | |
| 11:00–12:30 | MAX-ISI System at WMT23 Discourse-Level Literary Translation Task Li An, Linghao Jin and Xuezhe Ma |
| 11:00–12:30 | The MAKE-NMTVIZ System Description for the WMT23 Literary Task Fabien Lopez, Gabriela González, Damien Hansen, Mariam Nakhle, Behnoosh Namdarzadeh, Nicolas Ballier, Marco Dinarelli, Emmanuelle Esperança-Rodier, Sui He, Sadaf Mohseni, Caroline Rossi, Didier Schwab, Jun Yang, Jean-Baptiste Yunès and Lichao Zhu |
| 11:00–12:30 | DUTNLP System for the WMT2023 Discourse-Level Literary Translation Anqi Zhao, Kaiyu Huang, Hao Yu and Degen Huang |
| 11:00–12:30 | HW-TSC’s Submissions to the WMT23 Discourse-Level Literary Translation Shared Task Yuhao Xie, Zongyao Li, Zhanglin Wu, Daimeng Wei, Xiaoyu Chen, Zhiqiang Rao, Shaojun Li, Hengchao Shang, Jiaxin GUO, Lizhi Lei, Hao Yang and Yanfei Jiang |
| 11:00–12:30 | TJUNLP:System Description for the WMT23 Literary Task in Chinese to English Translation Direction Shaolin Zhu and Deyi Xiong |
| African Languages Translation Task | |
| 11:00–12:30 | Machine Translation for Nko: Tools, Corpora, and Baseline Results Moussa Doumbouya, Baba Mamadi Diané, Solo Farabado Cissé, Djibrila Diané, Abdoulaye Sow, Séré Moussa Doumbouya, Daouda Bangoura, Fodé Moriba Bayo, Ibrahima Sory 2. Conde, Kalo Mory Diané, Chris Piech and Christopher Manning |
| Sign Language Translation Task | |
| 11:00–12:30 | TTIC’s Submission to WMT-SLT 23 Marcelo Sandoval-Castaneda, Yanhong Li, Bowen Shi, Diane Brentari, Karen Livescu and Gregory Shakhnarovich |
| 11:00–12:30 | KnowComp Submission for WMT23 Sign Language Translation Task Baixuan Xu, Haochen Shi, Tianshi Zheng, Qing Zong, Weiqi Wang, Zhaowei Wang and Yangqiu Song |
| Parallel Data Curation Task | |
| 11:00–12:30 | A Fast Method to Filter Noisy Parallel Data WMT2023 Shared Task on Parallel Data Curation Nguyen-Hoang Minh-Cong, Nguyen Van Vinh and Nguyen Le-Minh |
| 11:00–12:30 | A Sentence Alignment Approach to Document Alignment and Multi-faceted Filtering for Curating Parallel Sentence Pairs from Web-crawled Data Steinthor Steingrimsson |
| 12:30–14:00 | Lunch Break |
| 14:00–15:30 | Session 3: Research Papers on Document-Level Translation and Use of Large Language Models (Chair: Nikolay Bogoychev) |
| 14:00–14:15 | Document-Level Language Models for Machine Translation Frithjof Petrick, Christian Herold, Pavel Petrushkov, Shahram Khadivi and Hermann Ney |
| 14:15–14:30 | ChatGPT MT: Competitive for High- (but Not Low-) Resource Languages Nathaniel Robinson, Perez Ogayo, David R. Mortensen and Graham Neubig |
| 14:30–14:45 | Large Language Models Effectively Leverage Document-level Context for Literary Translation, but Critical Errors Persist Marzena Karpinska and Mohit Iyyer |
| 14:45–15:00 | Identifying Context-Dependent Translations for Evaluation Set Production Rachel Wicks and Matt Post |
| 15:00–15:15 | Machine Translation with Large Language Models: Prompting, Few-shot Learning, and Fine-tuning with QLoRA Xuan Zhang, Navid Rajabi, Kevin Duh and Philipp Koehn |
| 15:15–15:30 | Towards Effective Disambiguation for Machine Translation with Large Language Models Vivek Iyer, Pinzhen Chen and Alexandra Birch |
| 15:30–16:00 | Coffee Break |
| 16:00–17:30 | Session 4: Research Papers on Translation Modelling (Chair: Huda Khayrallah) |
| 16:00–16:15 | A Closer Look at Transformer Attention for Multilingual Translation Jingyi Zhang, Hongfei Xu, Kehai Chen and Gerard de Melo |
| 16:15–16:30 | Bridging the Gap between Position-Based and Content-Based Self-Attention for Neural Machine Translation Felix Schmidt and Mattia Di Gangi |
| 16:30–16:45 | Visual Prediction Improves Zero-Shot Cross-Modal Machine Translation Tosho Hirasawa, Emanuele Bugliarello, Desmond Elliott and Mamoru Komachi |
| 16:45–17:00 | The Gender-GAP Pipeline: A Gender-Aware Polyglot Pipeline for Gender Characterisation in 55 Languages Benjamin Muller, Belen Alastruey, Prangthip Hansanti, Elahe Kalbassi, Christophe Ropers, Eric Smith, Adina Williams, Luke Zettlemoyer, Pierre Andrews and Marta R. Costa-jussà |
| 17:00–17:15 | Towards Better Evaluation for Formality-Controlled English-Japanese Machine Translation Edison Marrese-Taylor, Pin Chen Wang and Yutaka Matsuo |
| 17:15–17:30 | There’s No Data like Better Data: Using QE Metrics for MT Data Filtering Jan-Thorsten Peter, David Vilar, Daniel Deutsch, Mara Finkelstein, Juraj Juraska and Markus Freitag |
Thursday, December 7, 2023 | |
| 9:00–10:30 | Session 5: Shared Task Overview Papers II |
| 9:00–9:15 | Results of WMT23 Metrics Shared Task: Metrics Might Be Guilty but References Are Not Innocent Markus Freitag, Nitika Mathur, Chi-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frederic Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie and George Foster |
| 9:15–9:30 | Findings of the WMT 2023 Shared Task on Quality Estimation Frederic Blain, Chrysoula Zerva, Ricardo Rei, Nuno M. Guerreiro, Diptesh Kanojia, José G. C. de Souza, Beatriz Silva, Tânia Vaz, Yan Jingxuan, Fatemeh Azadi, Constantin Orasan and André Martins |
| 9:30–9:45 | Findings of the Word-Level AutoCompletion Shared Task in WMT 2023 Lemao Liu, Francisco Casacuberta, George Foster, Guoping Huang, Philipp Koehn, Geza Kovacs, Shuming Shi, Taro Watanabe and Chengqing Zong |
| 9:45–10:00 | Findings of the WMT 2023 Shared Task on Machine Translation with Terminologies Kirill Semenov, Vilém Zouhar, Tom Kocmi, Dongdong Zhang, Wangchunshu Zhou and Yuchen Eleanor Jiang |
| 10:00–10:15 | Findings of the WMT 2023 Shared Task on Automatic Post-Editing Pushpak Bhattacharyya, Rajen Chatterjee, Markus Freitag, Diptesh Kanojia, Matteo Negri and Marco Turchi |
| 10:15–10:30 | Findings of the WMT 2023 Shared Task on Low-Resource Indic Language Translation Santanu Pal, Partha Pakray, Sahinur Rahman Laskar, Lenin Laitonjam, Vanlalmuansangi Khenglawt, Sunita Warjri, Pankaj Kundan Dadure and Sandeep Kumar Dash |
| 10:30–11:00 | Coffee Break |
| 11:00–12:30 | Session 6: Shared Task System Description Posters II |
| Metrics Task | |
| 11:00–12:30 | ACES: Translation Accuracy Challenge Sets at WMT 2023 Chantal Amrhein, Nikita Moghe and Liane Guillou |
| 11:00–12:30 | Challenging the State-of-the-art Machine Translation Metrics from a Linguistic Perspective Eleftherios Avramidis, Shushen Manakhimova, Vivien Macketanz and Sebastian Möller |
| 11:00–12:30 | Tokengram_F, a Fast and Accurate Token-based chrF++ Derivative Sören DREANO, Derek Molloy and Noel Murphy |
| 11:00–12:30 | Embed_Llama: Using LLM Embeddings for the Metrics Shared Task Sören DREANO, Derek Molloy and Noel Murphy |
| 11:00–12:30 | eBLEU: Unexpectedly Good Machine Translation Evaluation Using Simple Word Embeddings Muhammad ElNokrashy and Tom Kocmi |
| 11:00–12:30 | Cometoid: Distilling Strong Reference-based Machine Translation Metrics into Even Stronger Quality Estimation Metrics Thamme Gowda, Tom Kocmi and Marcin Junczys-Dowmunt |
| 11:00–12:30 | MetricX-23: The Google Submission to the WMT 2023 Metrics Shared Task Juraj Juraska, Mara Finkelstein, Daniel Deutsch, Aditya Siddhant, Mehdi Mirzazadeh and Markus Freitag |
| 11:00–12:30 | GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4 Tom Kocmi and Christian Federmann |
| 11:00–12:30 | Metric Score Landscape Challenge (MSLC23): Understanding Metrics’ Performance on a Wider Landscape of Translation Quality Chi-kiu Lo, Samuel Larkin and Rebecca Knowles |
| 11:00–12:30 | MEE4 and XLsim : IIIT HYD’s Submissions’ for WMT23 Metrics Shared Task Ananya Mukherjee and Manish Shrivastava |
| 11:00–12:30 | Quality Estimation Using Minimum Bayes Risk Subhajit Naskar, Daniel Deutsch and Markus Freitag |
| 11:00–12:30 | Evaluating Metrics for Document-context Evaluation in Machine Translation Vikas Raunak, Tom Kocmi and Matt Post |
| 11:00–12:30 | Semantically-Informed Regressive Encoder Score Vasiliy Viskov, George Kokush, Daniil Larionov, Steffen Eger and Alexander Panchenko |
| 11:00–12:30 | Empowering a Metric with LLM-assisted Named Entity Annotation: HW-TSC’s Submission to the WMT23 Metrics Shared Task Zhanglin Wu, Yilun Liu, Min Zhang, Xiaofeng Zhao, Junhao Zhu, Ming Zhu, Xiaosong Qiao, Jingfei Zhang, Ma Miaomiao, Zhao Yanqing, Song Peng, shimin tao, Hao Yang and Yanfei Jiang |
| Quality Estimation Task | |
| 11:00–12:30 | Unify Word-level and Span-level Tasks: NJUNLP’s Participation for the WMT2023 Quality Estimation Shared Task Xiang Geng, Zhejian Lai, Yu Zhang, shimin tao, Hao Yang, Jiajun CHEN and Shujian Huang |
| 11:00–12:30 | HW-TSC 2023 Submission for the Quality Estimation Shared Task Yuang Li, Chang Su, Ming Zhu, Mengyao Piao, Xinglin Lyu, Min Zhang and Hao Yang |
| 11:00–12:30 | Scaling up CometKiwi: Unbabel-IST 2023 Submission for the Quality Estimation Shared Task Ricardo Rei, Nuno M. Guerreiro, José Pombal, Daan van Stigt, Marcos Treviso, Luisa Coheur, José G. C. de Souza and André Martins |
| 11:00–12:30 | SurreyAI 2023 Submission for the Quality Estimation Shared Task Archchana Sindhujan, Diptesh Kanojia, Constantin Orasan and Tharindu Ranasinghe |
| 11:00–12:30 | MMT’s Submission for the WMT 2023 Quality Estimation Shared Task Yulong Wu, Viktor Schlegel, Daniel Beck and Riza Batista-Navarro |
| 11:00–12:30 | IOL Research’s Submission for WMT 2023 Quality Estimation Shared Task ZEYU YAN |
| Word-Level Autocompletion Task | |
| 11:00–12:30 | SJTU-MTLAB’s Submission to the WMT23 Word-Level Auto Completion Task Xingyu Chen and Rui Wang |
| 11:00–12:30 | PRHLT’s Submission to WLAC 2023 Angel Navarro, Miguel Domingo and Francisco Casacuberta |
| 11:00–12:30 | KnowComp Submission for WMT23 Word-Level AutoCompletion Task Yi Wu, Haochen Shi, Weiqi Wang and Yangqiu Song |
| Termninology Translation Task | |
| 11:00–12:30 | Terminology-Aware Translation with Constrained Decoding and Large Language Model Prompting Nikolay Bogoychev and Pinzhen Chen |
| 11:00–12:30 | Lingua Custodia’s Participation at the WMT 2023 Terminology Shared Task Jingshu Liu, Mariam Nakhlé, Gaëtan Caillout and Raheel Qadar |
| 11:00–12:30 | Domain Terminology Integration into Machine Translation: Leveraging Large Language Models Yasmin Moslem, Gianfranco Romani, Mahdi Molaei, John D. Kelleher, Rejwanul Haque and Andy Way |
| 11:00–12:30 | OPUS-CAT Terminology Systems for the WMT23 Terminology Shared Task Tommi Nieminen |
| 11:00–12:30 | VARCO-MT: NCSOFT’s WMT’23 Terminology Shared Task Submission Geon Woo Park, Junghwa Lee, Meiying Ren, Allison Shindell and Yeonsoo Lee |
| Automatic Postediting Task | |
| 11:00–12:30 | HW-TSC’s Participation in the WMT 2023 Automatic Post Editing Shared Task Jiawei Yu, Min Zhang, Zhao Yanqing, Xiaofeng Zhao, Yuang Li, Su Chang, Yinglu Li, Ma Miaomiao, shimin tao and Hao Yang |
| Indic Languages Translation Task | |
| 11:00–12:30 | Neural Machine Translation for English - Manipuri and English - Assamese Goutam Agrawal, Rituraj Das, Anupam Biswas and Dalton Meitei Thounaojam |
| 11:00–12:30 | GUIT-NLP’s Submission to Shared Task: Low Resource Indic Language Translation Mazida Ahmed, Kuwali Talukdar, Parvez Boruah, Prof. Shikhar Kumar Sarma and Kishore Kashyap |
| 11:00–12:30 | NICT-AI4B’s Submission to the Indic MT Shared Task in WMT 2023 Raj Dabre, Jay Gala and Pranjal Chitale |
| 11:00–12:30 | Machine Translation Advancements for Low-Resource Indian Languages in WMT23: CFILT-IITB’s Effort for Bridging the Gap Pranav Gaikwad, Meet Doshi, Sourabh Deoghare and Pushpak Bhattacharyya |
| 11:00–12:30 | Low-Resource Machine Translation Systems for Indic Languages Ivana Kvapilíková and Ondřej Bojar |
| 11:00–12:30 | MUNI-NLP Systems for Low-resource Indic Machine Translation Edoardo Signoroni and Pavel Rychly |
| 11:00–12:30 | NITS-CNLP Low-Resource Neural Machine Translation Systems of English-Manipuri Language Pair Kshetrimayum Boynao Singh, Avichandra Singh Ningthoujam, Loitongbam Sanayai Meetei, Sivaji Bandyopadhyay and Thoudam Doren Singh |
| 11:00–12:30 | IACS-LRILT: Machine Translation for Low-Resource Indic Languages Dhairya Suman, Atanu Mandal, Santanu Pal and Sudip Naskar |
| 11:00–12:30 | IOL Research Machine Translation Systems for WMT23 Low-Resource Indic Language Translation Shared Task Wenbo Zhang |
| 14:00–15:30 | Session 7: Panel on Large Language Models and Machine Translation |
| Eleftheria Briakou (University of Maryland), Arul Menezes (Microsoft), Jose de Souza (Unbabel), moderated by Philipp Koehn (Johns Hopkins University) | |
| 16:00–17:30 | Session 8: Research Papers on Evaluation (Chair: Kenton Murray) |
| 16:00–16:15 | Trained MT Metrics Learn to Cope with Machine-translated References Jannis Vamvas, Tobias Domhan, Sony Trenous, Rico Sennrich and Eva Hasler |
| 16:15–16:30 | Training and Meta-Evaluating Machine Translation Evaluation Metrics at the Paragraph Level Daniel Deutsch, Juraj Juraska, Mara Finkelstein and Markus Freitag |
| 16:30–16:45 | Automating Behavioral Testing in Machine Translation Javier Ferrando, Matthias Sperber, Hendra Setiawan, Dominic Telaar and Saša Hasan |
| 16:45–17:00 | One Wide Feedforward Is All You Need Telmo Pires, António Vilarinho Lopes, Yannick Assogba and Hendra Setiawan |
| 17:00–17:15 | A Benchmark for Evaluating Machine Translation Metrics on Dialects without Standard Orthography Noëmi Aepli, Chantal Amrhein, Florian Schottmann and Rico Sennrich |
| 17:15–17:30 | The Devil Is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, André Martins, Graham Neubig, Ankush Garg, Jonathan Clark, Markus Freitag and Orhan Firat |