SyncAI.news, a Varaisys broadcasting
Last Translation Benchmark
VZ

Vil\'em Zouhar, Niyati Bafna, Mukund Choudhary, Maike Z\"ufle, Sara Rajaee, Pinzhen Chen, Jannis Vamvas, Sara Papi, Ona de Gibert, Bhavitvya Malik, Eliya Habba, Orfeas Menis Mastromichalakis, Patr\'icia Schmidtov\'a, Michelle Wastl, Sheriff Issaka, Leshem Choshen, Stella Biderman, Antonis Anastasopoulos, Jan Niehues, Rico Sennrich, Mrinmaya Sachan, Ond\v{r}ej Bojar, Kenton Murray, J\"org Tiedemann, Alham Fikri Aji, Philipp Koehn, Christof Monz, Alexandra Birch, Sowmya Vajjala, Chalamalasetti Kranti, Cristina Espa\~na-Bonet, Nobin Sarwar, David Kacz\'er, Sourajit Saha, Jonathan Tonglet, Shunta Asano, Malik Marmonier, Daban Q. Jaff, Vaisakhi Mishra, Hend Al-Khalifa, Gabriele Sarti, Nils Rehlinger, Juan Daniel Cuervo Villa, Dominik Mach\'a\v{c}ek, Saugata Purkayastha, Jagannathan Ramanujam, Shubhashis Roy Dipta, Aviral Nigam, Shuaib Shuaib Yusuf, Heejin Do, Jonathan Yahav, Johannes-Rudolf David, Maria Carmen Staiano, Zuzana Nadova, Fred Philippy, Ron Keinan, Maria Lymperaiou, Silvia Casola, Fabian Retkowski, Andr\'es Jerez, Hanna Yukhy- menko, Sangwon Ryu, Avantica Vempati, Sukannya Purkayastha, Adrian Cosma, Erivan Inan, Vitalii Babenko, Wafa Aissa, Valentin Scourneau, Fatima Haouari, Venkata Prasanth Kumar Gummadi, Mehdi Jafarzadeh, Manon Reusens, Kaiser Sun, Lukas Edman, Shaomu Tan, Giuseppe Gallipoli, Pawan Sasanka Ammanamanchi, Manar Ali, Mohammad Sadegh Gholizadeh, Dipankar Srirag, Marek \v{S}uppa, Javier Garc\'ia Gilabert, Ruta Binkyte, Ana-Maria Bucur, Sabry E. Farrag, Youssef Saber, Yihong Liu, Theresia Veronika Rampisela, Christian Hoang, Nicoleta Cojocaru, Jan Koco\'n, Jean Maillard, Xiaochuang Yuan, Sina Ahmadi, Daryna Dementieva, Philipp Mondorf, Kaustubh Dhole, Valmik Nahata, Roman Wixinger, Amir Hossein Yari, Shenbin Qian, Manuel Tuor, Fida Mohammad Thoker, Sergey Troshin, Lance Calvin Lim Gamboa, Amir Arsalan Rezapour, K\"atriin Kukk, Koel Dutta Chowdhury, Shaswati Saha, Seth Aycock, Bo Chen, Linh Vu, Vatsal Venkatkrishna, Shayan Bali, Arafat Ahsan, Luan Thanh Nguyen, Hassan Soliman, Ngoc Quynh Tram Do, Azmine Toushik Wasi, R. Damanhuri, Marius Huber, Kazuki Egashira, Jimson Paulo Layacan, David Africa, Vladislav Poritski, Mike Zhang, Deep Shah, Abdulaziz Nura Kani, Luis Frentzen Salim, Paul Gavrikov, Bello Umar Bello, Ayush Sunil Munot, Anumit Garg, Yolanda Xavier, Qiaoyuan Zheng, Kawsar Ahmed, Debanshu Das, Zimu Wang, Gengyu Rao, Kamile Dementaviciute, Farhan Farsi, L D M S Sai Teja, Dawei Zhu, Yi Fan, Wei Liu, Marco Gaido, Elias Herranen, Sankalan Pal Chowdhury, Karen Sanchez, Guy Kaplan, Farzad Shami, Ashok Urlana, Amir Hossein Kargaran, Sofie Goethals, Priyaranjan Pattnayak, Oksana Volchek, Marii Ojastu, Hongbin Na, Emilian Radoi, Chenyi Zhao, Carlos Hinojosa, Andrei Niculae, Andrea Gregor de Varda, Zaid Alyafeai, Tomasz Limisiewicz, Reem Alzahrani, Pouya Sadeghi, Nehal Kathrotia, Mateusz Lango, Enzo Doyen, Alex Fl\"uckiger, Ulysses Sekai Tully Carr, Samuel Simko, Ritwik Tiwari, Rishit Dagli, Isaac R Caswell, Bowen Yi, Aicha Chorana, Selja Ker\"anen, Sadiksha Chitrakar, Muhammad Ravi Shulthan Habibi, Joy Olusanya, Bishal Shrestha, Zhengxiang Wang, Vivek Harsha Lakkamaneni, Sophia Conrad, Panayiotis Panayiotou, Nazia Tasnim, Marta Punsola Mun\'arriz, Marko Culjak, Luis Lara, Jenny Chim, Jannatul Nayem, Fidel Rodr\'iguez Vel\'asquez, Eran Yahav, Blanka K\"ov\'er, Beatrice Savoldi, Anmol Goel, Aishik Mandal, Tosin Adewumi, Raoyuan Zhao, Mykola Haltiuk, Antonia Karamolegkou, Yuxing Lu, Thura Aung, Naser Almousa, Tommaso Cerruti, Raia Abu Ahmad, Beni Egressy, Alireza Pakniat, St\'ephane J. P. S. Thunus, Rachel Bawden, Lena Libon, Samridhya Biswas, Prakhar Gupta, Nusrat Jahan Lia, Nguyen Tai, Natchapon Jongwiriyanurak, Minh Ngoc Do, Ivan Bara\'c, Dzmitry Kuzmin, Badal Nyalang, Antoine Taroni, Andy Catruna, Rushikesh Zawar, Roland Aydin, Pavel Stepachev, Ilai Yaron Levy, Andreas Simons, Rayyan Merchant, Ziyi Yang, Samuel Frontull, Kenneth Enevoldsen, Harris Abdul Majid, Tim Graf, Tatiana Bielakova, Sharifa Djurabaeva, Shaoxiong Ji, Jirui Qi, Ayla Rigouts Terryn, Yurii Paniv, Xiyan Fu, Sunisth Kumar, Shree Harsha Bokkahalli Satish, Papa Abdou Karim Karou Diallo, Mengyu Ye, Maximilian Vieweg, Matija Akrap, Krist\'yna Onderkov\'a, Joseph Attieh, Ivan Yuri De Leon, Ibrahim Baroud, Esrael Teferi Tensay, Elisabeth Fittschen, David Duki\'c, Beno\^it Sagot, Jingwei Ni, Yu Fan, Juri Opitz

· 1 min read

ResearcharXiv cs.CL

Last Translation Benchmark

arXiv:2609.04173v2 Announce Type: replace Abstract: For scientific progress, we need benchmarks that test the limits of state-of-the-art models, and evaluation methods that inform us about failure cases. As models get stronger, standard benchmarks for machine translation are approaching saturation. Further, automatic translation metrics are unreliable, opaque, and vulnerable to reward-hacking. Even gold human evaluation is not problem-free, because it often lacks reproducibility, objectivity, and scalability. Overall, this prevents us from tracking progress in the field and identifying pathways for improvement. We introduce the Last Translation Benchmark, a collection of human-authored and peer-reviewed examples (texts, images, audio, videos) that break leading machine translation models. We also present a new evaluation approach: each example comes with handcrafted verification rules describing concrete failure cases on that example, therefore allowing reliable and actionable future evaluation. The Last Translation Benchmark is a live dataset that accepts ongoing contributions. The latest version is LTBv1, containing accepted contributions prior to September 1st 2026, with future releases planned as new data is continuously collected.

Original source

This story was published by arXiv cs.CL and written by Vil\'em Zouhar, Niyati Bafna, Mukund Choudhary, Maike Z\"ufle, Sara Rajaee, Pinzhen Chen, Jannis Vamvas, Sara Papi, Ona de Gibert, Bhavitvya Malik, Eliya Habba, Orfeas Menis Mastromichalakis, Patr\'icia Schmidtov\'a, Michelle Wastl, Sheriff Issaka, Leshem Choshen, Stella Biderman, Antonis Anastasopoulos, Jan Niehues, Rico Sennrich, Mrinmaya Sachan, Ond\v{r}ej Bojar, Kenton Murray, J\"org Tiedemann, Alham Fikri Aji, Philipp Koehn, Christof Monz, Alexandra Birch, Sowmya Vajjala, Chalamalasetti Kranti, Cristina Espa\~na-Bonet, Nobin Sarwar, David Kacz\'er, Sourajit Saha, Jonathan Tonglet, Shunta Asano, Malik Marmonier, Daban Q. Jaff, Vaisakhi Mishra, Hend Al-Khalifa, Gabriele Sarti, Nils Rehlinger, Juan Daniel Cuervo Villa, Dominik Mach\'a\v{c}ek, Saugata Purkayastha, Jagannathan Ramanujam, Shubhashis Roy Dipta, Aviral Nigam, Shuaib Shuaib Yusuf, Heejin Do, Jonathan Yahav, Johannes-Rudolf David, Maria Carmen Staiano, Zuzana Nadova, Fred Philippy, Ron Keinan, Maria Lymperaiou, Silvia Casola, Fabian Retkowski, Andr\'es Jerez, Hanna Yukhy- menko, Sangwon Ryu, Avantica Vempati, Sukannya Purkayastha, Adrian Cosma, Erivan Inan, Vitalii Babenko, Wafa Aissa, Valentin Scourneau, Fatima Haouari, Venkata Prasanth Kumar Gummadi, Mehdi Jafarzadeh, Manon Reusens, Kaiser Sun, Lukas Edman, Shaomu Tan, Giuseppe Gallipoli, Pawan Sasanka Ammanamanchi, Manar Ali, Mohammad Sadegh Gholizadeh, Dipankar Srirag, Marek \v{S}uppa, Javier Garc\'ia Gilabert, Ruta Binkyte, Ana-Maria Bucur, Sabry E. Farrag, Youssef Saber, Yihong Liu, Theresia Veronika Rampisela, Christian Hoang, Nicoleta Cojocaru, Jan Koco\'n, Jean Maillard, Xiaochuang Yuan, Sina Ahmadi, Daryna Dementieva, Philipp Mondorf, Kaustubh Dhole, Valmik Nahata, Roman Wixinger, Amir Hossein Yari, Shenbin Qian, Manuel Tuor, Fida Mohammad Thoker, Sergey Troshin, Lance Calvin Lim Gamboa, Amir Arsalan Rezapour, K\"atriin Kukk, Koel Dutta Chowdhury, Shaswati Saha, Seth Aycock, Bo Chen, Linh Vu, Vatsal Venkatkrishna, Shayan Bali, Arafat Ahsan, Luan Thanh Nguyen, Hassan Soliman, Ngoc Quynh Tram Do, Azmine Toushik Wasi, R. Damanhuri, Marius Huber, Kazuki Egashira, Jimson Paulo Layacan, David Africa, Vladislav Poritski, Mike Zhang, Deep Shah, Abdulaziz Nura Kani, Luis Frentzen Salim, Paul Gavrikov, Bello Umar Bello, Ayush Sunil Munot, Anumit Garg, Yolanda Xavier, Qiaoyuan Zheng, Kawsar Ahmed, Debanshu Das, Zimu Wang, Gengyu Rao, Kamile Dementaviciute, Farhan Farsi, L D M S Sai Teja, Dawei Zhu, Yi Fan, Wei Liu, Marco Gaido, Elias Herranen, Sankalan Pal Chowdhury, Karen Sanchez, Guy Kaplan, Farzad Shami, Ashok Urlana, Amir Hossein Kargaran, Sofie Goethals, Priyaranjan Pattnayak, Oksana Volchek, Marii Ojastu, Hongbin Na, Emilian Radoi, Chenyi Zhao, Carlos Hinojosa, Andrei Niculae, Andrea Gregor de Varda, Zaid Alyafeai, Tomasz Limisiewicz, Reem Alzahrani, Pouya Sadeghi, Nehal Kathrotia, Mateusz Lango, Enzo Doyen, Alex Fl\"uckiger, Ulysses Sekai Tully Carr, Samuel Simko, Ritwik Tiwari, Rishit Dagli, Isaac R Caswell, Bowen Yi, Aicha Chorana, Selja Ker\"anen, Sadiksha Chitrakar, Muhammad Ravi Shulthan Habibi, Joy Olusanya, Bishal Shrestha, Zhengxiang Wang, Vivek Harsha Lakkamaneni, Sophia Conrad, Panayiotis Panayiotou, Nazia Tasnim, Marta Punsola Mun\'arriz, Marko Culjak, Luis Lara, Jenny Chim, Jannatul Nayem, Fidel Rodr\'iguez Vel\'asquez, Eran Yahav, Blanka K\"ov\'er, Beatrice Savoldi, Anmol Goel, Aishik Mandal, Tosin Adewumi, Raoyuan Zhao, Mykola Haltiuk, Antonia Karamolegkou, Yuxing Lu, Thura Aung, Naser Almousa, Tommaso Cerruti, Raia Abu Ahmad, Beni Egressy, Alireza Pakniat, St\'ephane J. P. S. Thunus, Rachel Bawden, Lena Libon, Samridhya Biswas, Prakhar Gupta, Nusrat Jahan Lia, Nguyen Tai, Natchapon Jongwiriyanurak, Minh Ngoc Do, Ivan Bara\'c, Dzmitry Kuzmin, Badal Nyalang, Antoine Taroni, Andy Catruna, Rushikesh Zawar, Roland Aydin, Pavel Stepachev, Ilai Yaron Levy, Andreas Simons, Rayyan Merchant, Ziyi Yang, Samuel Frontull, Kenneth Enevoldsen, Harris Abdul Majid, Tim Graf, Tatiana Bielakova, Sharifa Djurabaeva, Shaoxiong Ji, Jirui Qi, Ayla Rigouts Terryn, Yurii Paniv, Xiyan Fu, Sunisth Kumar, Shree Harsha Bokkahalli Satish, Papa Abdou Karim Karou Diallo, Mengyu Ye, Maximilian Vieweg, Matija Akrap, Krist\'yna Onderkov\'a, Joseph Attieh, Ivan Yuri De Leon, Ibrahim Baroud, Esrael Teferi Tensay, Elisabeth Fittschen, David Duki\'c, Beno\^it Sagot, Jingwei Ni, Yu Fan, Juri Opitz. SyncAI.news shows a preview; the complete article is on the publisher's site.

Read the full story on arxiv.org

Similar News