ADAPT researcher Keren Artiaga from Munster Technological University (MTU) recently had a paper accepted to EMNLP Findings. The study, titled “Rethinking Sign Language Translation: The Impact of Signer Dependence on Model Evaluation”, will be presented at the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), taking place from 4th to 9th November in Suzhou, China.
The research highlights that performance reported in existing works does not reliably indicate how well sign language translation models truly generalise. By applying signer-fold cross-validation on benchmark datasets with state-of-the-art gloss-free models, the study showed that results drop significantly under signer-independant evaluation.
The paper recommends adopting signer-independent testing protocols, restructuring datasets with explicit signer-independent, sentence-disjoint splits, and reporting overlap statistics to ensure a more reliable assessment of translation systems.
Co-authored by Sabyasachi Kamila (Manipal Institute of Technology Bengaluru, MAHE, Manipal, India), Haithem Afli (ADAPT MTU), Conor Lynch (Nimbus Research Centre, MTU – Cork), and Mohammed Hasanuzzaman (ADAPT MTU & Queen’s University Belfast).
