Cover not available
Part of
Eye Tracking and Multidisciplinary Studies on Translation
Edited by Callum Walker and Federico M. Federici
[Benjamins Translation Library 143] 2018
► pp. 55–69

Chapter 4
Eye tracking as a measure of cognitive effort for post-editing of machine translation

ORCID logo

The three measurements for post-editing effort as proposed by Krings (2001) have been adopted by many researchers in subsequent studies and publications. These measurements comprise temporal effort (the speed or productivity rate of post-editing, often measured in words per second at the segment level), technical effort (the number of actual edits performed by the post-editor, sometimes approximated using the Translation Edit Rate metric (Snover et al. 2006), again usually at the segment level), and cognitive effort. Cognitive effort has been measured using think-aloud protocols, pause measurement, and, increasingly, eye-tracking. This chapter provides a review of studies of post-editing effort using eye-tracking, noting the influence of publications by Danks et al. (1997), and O’Brien (2006, 2008), before describing a single study in detail. The detailed study examines whether predicted affort indicators affect post-editing effort and results were previously published as Moorkens et al. (2015). This chapter focuses instead on methodology and the logistics of running an eye-tracking study recording over 70 sessions. Most of the eye-tracking data analysed were unused in the previous publication, and the small amount presented was not described in detail due to space constraints. In this study average fixation count per segment correlates very strongly with temporal effort, and average fixation duration correlates strongly with technical effort, a result that we compare with other studies of post-editing effort.

Article outline
  • 1.Introduction
  • 2.The task of post-editing
  • 3.Eye-tracking measures used in studies of post-editing
  • 4.Post-editing study
    • 4.1Methodology
    • 4.2Results: Stage 1
    • 4.3Results: Stage 2
    • 4.4Results: Stage 3
  • 5.Conclusion
  • References
Published online: 16 October 2018
DOI logo
https://doi.org/10.1075/btl.143.04moo
References
Allen, Jeffrey
2003 “Post-editing.” In Computers and Translation, ed. by Harold Somers, 297–318. Amsterdam: John Benjamins.DOI logoGoogle Scholar
ALPAC
1966 Languages and machines: computers in translation and linguistics. A report by the Automatic Language Processing Advisory Committee. Washington, DC.Google Scholar
Alves, Fabio, Arlene Koglin, Bartolomé Mesa-Lao, Mercedes García Martínez, Norma B. de Lima Fonseca, Arthur de Melo Sá, José Luiz Gonçalves, Karina Sarto Szpak, Kyoko Sekino, and Marceli Aquino
2016 “Analysing the Impact of Interactive Machine Translation on Post-editing Effort.” In New Directions in Empirical Translation Process Research, ed. by Michael Carl, Srinivas Bangalore, and Moritz Schaeffer, 77–94. Heidelberg: Springer.DOI logoGoogle Scholar
Bentivogli, Luisa, Arianna Bisazza, Mauro Cettolo, and Marcello Federico
2016 “Neural versus Phrase-Based Machine Translation Quality: a Case Study.” In Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP). Austin, Texas.DOI logoGoogle Scholar
Beyer, Robert T.
1965 “Hurdling the Language Barrier.” Physics Today 18 (1): 46–52.DOI logoGoogle Scholar
Bruckner, Christine, and Mirko Plitt
2001 “Evaluating the Operational Benefit of Using Machine Translation Output as Translation Memory Input.” In Proceedings of MT Summit VIII, 61–65.Google Scholar
Caffrey, Colm
2009 Relevant Abuse? Investigating the Effects of an Abusive Subtitling Procedure on the Perception of TV Anime Using Eye Tracker and Questionnaire. PhD dissertation, Dublin City University.Google Scholar
Carl, Michael, Barbara Dragsted, Jakob Elming, Daniel Hardt, and Arnt Lykke Jakobsen
2011 “The Process of Post-Editing: a Pilot Study.” In Proceedings of NLPSC 2011.Google Scholar
Carl, Michael, Silke Gutermuth, and Silvia Hansen-Schirra
2015 “Post-editing machine translation: Efficiency, strategies, and revision processes in professional translation settings. Psycholinguistic and Cognitive Inquiries into Translation and Interpreting.” In Psycholinguistic and Cognitive Inquiries into Translation and Interpreting, ed. by Aline Ferreira, and John W. Schwieter, 145–174. Amsterdam: John Benjamins.Google Scholar
Castilho, Sheila, Joss Moorkens, Federico Gaspari, Iacer Calixto, John Tinsley, and Andy Way
2017 “Is Neural Machine Translation the New State of the Art?” Prague Bulletin of Mathematical Linguistics 108 (1): 109–120.DOI logoGoogle Scholar
Chang, Vincent Chieh-Ying
2009 Testing Applicability of Eye-tracking and fMRI to Translation and Interpreting Studies: An Investigation into Directionality. PhD dissertation. Imperial College, London.Google Scholar
Daems, Joke, Sonia Vandepitte, Robert J. Hartsuiker, and Lieve Macken
2017 “Identifying the Machine Translation Error Types with the Greatest Impact on Post-Editing Effort.” Frontiers in Psychology 8: 1282.DOI logoGoogle Scholar
Danks, Joseph H., Gregory M. Shreve, Stephen B. Fountain, and Michael K. McBeath
1997 Cognitive Processes in Translation and Interpreting. Thousand Oaks, California: Sage Publications.Google Scholar
Denkowski, Michael, and Alon Lavie
2014 “Meteor Universal: Language Specific Translation Evaluation for Any Target Language.” In Proceedings of the EACL 2014 Workshop on Statistical Machine Translation.Google Scholar
Dragsted, Barbara
2010 “Coordination of Reading and Writing Processes in Translation: An Eye on Uncharted Territory.” In Translation and Cognition, ed. by Gregory M. Schreve, and Erik M. Angelone, 41–62. Amsterdam: John Benjamins.DOI logoGoogle Scholar
Duchowski, Andrew
2003 Eye Tracking Methodology: Theory and Practice. Heidelberg: Springer.DOI logoGoogle Scholar
García, Ignacio
2011 “Translating by Post-editing: Is It the Way Forward?” Machine Translation 25: 217–237.DOI logoGoogle Scholar
Gaspari, Federico, Antonio Toral, Sudip Kumar Naskar, Declan Groves, Andy Way
2014 “Perception vs Reality: Measuring Machine Translation Post-Editing Productivity.” In Proceedings of AMTA 2014 Workshop on Post-editing Technology and Practice, Vancouver, 60–72.Google Scholar
Gonçalves, José Luiz
2016 “Investigating Saccades as an Index of Cognitive Effort in Post-editing and Translation.” In Proceedings of EST Congress 2016, Aarhus, Denmark.Google Scholar
Hansen-Schirra, Silvia
2017 “EEG and Universal Language Processing in Translation.” In The Handbook of Translation and Cognition, ed. by John W. Schwieter, and Aline Ferreira, 232–247. Hoboken: John Wiley & Sons.DOI logoGoogle Scholar
Hokamp, Chris, and Qun Liu
2015 “HandyCAT.” In Proceedings of European Association for Machine Translation (EAMT) 2015, Antalya, 216.Google Scholar
Hutchins, William John
1992 “Météo.” In An Introduction to Machine Translation, ed. by W. John Hutchins, and Harold L. Somers, 207–220. London: Academic Press.Google Scholar
Hvelplund, Kristian T.
2011 Allocation of Cognitive Resources in Translation: An Eye-tracking and Key-logging Study. PhD dissertation, Copenhagen Business School.Google Scholar
Jakobsen, Arnt Lykke
1999 “Logging Target Text Production with Translog.” Copenhagen Studies in Language 24: 9–20.Google Scholar
Koglin, Arlene
2015 “An Empirical Investigation of Cognitive Effort Required to Post-edit Machine Translated Metaphors Compared to the Translation of Metaphors.” Translation and Interpreting 7 (1): 126–141.Google Scholar
Krings, Hans P.
2001 Repairing Texts: Empirical Investigations of Machine Translation Post-editing Processes. Translated by G. S. Koby. Kent, OH: The Kent State University Press.Google Scholar
Lacruz, Isabel, and Gregory M. Shreve
2014 “Pauses and Cognitive Effort in Post-editing.” In Post-editing of Machine Translation: Processes and Applications, ed. by Sharon O’Brien, Laura Winther Balling, Michael Carl, Michel Simard, and Lucia Specia, 246–272. Newcastle-Upon-Tyne: Cambridge Scholars.Google Scholar
Läubli, Samuel, Mark Fishel, Gary Massey, Maureen Ehrensberger-Dow, and Martin Volk
2013 “Assessing Post-Editing Efficiency in a Realistic Translation Environment.” In Proceedings of MT Summit XIV Workshop on Post-editing Technology and Practice, Nice, 83–91.Google Scholar
Läubli, Samuel, and Ulrich Germann
2016 “Statistical Modelling and Automatic Tagging of Human Translation Processes.” In New Directions in Empirical Translation Process Research, ed. by Michael Carl, Srinivas Bangalore, and Moritz Schaeffer, 77–94. Heidelberg: Springer.DOI logoGoogle Scholar
Lommel, Arle R., and Donald A. DePalma
2016 Europe’s Leading Role in Machine Translation: How Europe Is Driving the Shift to MT. Boston: Common Sense Advisory Report.Google Scholar
Moorkens, Joss, Sharon O’Brien, Igor Antonio Lourenco Silva, Norma Fonseca, and Fabio Alves
2015 “Correlations of Perceived Post-editing Effort with Measurements of Actual Effort.” Machine Translation 29 (3–4): 267–284.DOI logoGoogle Scholar
Moorkens, Joss
2017 “Under Pressure: Translation in Times of Austerity.” Perspectives: Studies in Translation Theory and Practice, 25 (3): 464–477.DOI logoGoogle Scholar
Moorkens, Joss, and Sharon O’Brien
2015 “Post-editing Evaluations: Trade-offs Between Novice and Professional Participants.” In Proceedings of European Association for Machine Translation Conference (EAMT) 2015, Antalya, Turkey, 75–81.Google Scholar
Mossop, Brian
2007 Revising and Editing for Translators. Manchester: St. Jerome.Google Scholar
Nitzke, Jean, Katharina Oster
2016 “Comparing Translation and Post-editing: An Annotation Schema for Activity Units.” In New Directions in Empirical Translation Process Research, ed. by Michael Carl, Srinivas Bangalore, and Moritz Schaeffer, 77–94. Heidelberg: Springer.DOI logoGoogle Scholar
Nunes Vieira, Lucas
2015 Cognitive Effort in Post-Editing of Machine Translation: Evidence from Eye Movements, Subjective Ratings, and Think-Aloud Protocols. PhD dissertation. Newcastle University.Google Scholar
O’Brien, Sharon
2005 “Methodologies for Measuring the Correlations Between Post-editing Effort and Machine Translatability.” Machine Translation 19 (1): 37–58.DOI logoGoogle Scholar
2006 “Eye Tracking and Translation Memory Matches.” Perspectives: Studies in Translatology 14 (3): 185–205.Google Scholar
2008 “Processing Fuzzy Matches in Translation Memory Tools: An Eye-tracking Analysis.” In Looking at Eyes. Eye Tracking Studies of Reading and Translation Processing, ed. by Susanne Göpferich, Arnt Lykke Jakobsen and Inger Mees, 79–102. Frederiksberg: Samfundslitteratur.Google Scholar
2010 “Introduction to Post-editing: Who, What, How and Where to Next?” In Proceedings of AMTA 2010, the 9th Conference of the Association for Machine Translation in the Americas, Denver, USA.Google Scholar
2011 “Towards Predicting Post-Editing Productivity”. Machine Translation 25 (3): 197.DOI logoGoogle Scholar
Plitt, Mirko, and François Masselot
2010 “A Productivity Test of Statistical Machine Translation Post-editing in a Typical Localization Context.” Prague Bulletin of Mathematical Linguistics 93: 7–16.DOI logoGoogle Scholar
Saldanha, Gabriela, and Sharon O’Brien
2014 Research Methodologies in Translation Studies. London: Routledge.DOI logoGoogle Scholar
Schmidtke, Dag
2016 “Large Scale Machine Translation Publishing, With Acceptable Quality, for Microsoft Support Content.” In Proceedings of AMTA 2016 Workshop on Interacting with Machine Translation (iMT 2016). Austin, Texas.Google Scholar
Senez, Dorothy
1998 “Post-editing Service for Machine Translation Users at the European Commission.” Translating and the Computer 20. Proceedings of the Twentieth International Conference. 12–13 November 1998.Google Scholar
Shreve, Gregory M., and Bruce Diamond
1997 “Cognitive Processes in Translation and Interpreting: Critical Issues.” In Cognitive Processes in Translation and Interpreting, ed. by Joseph H. Danks, Gregory M. Shreve, Stephen B. Fountain, and Michael K. McBeath, 233–251. Thousand Oaks, California: Sage Publications.Google Scholar
Snover, Matthew, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul
2006 “A Study of Translation Edit Rate with Targeted Human Annotation.” In Proceedings of Association for Machine Translation in the Americas, 223–231 8–12 August 2006, Cambridge, Massachusetts, USA.Google Scholar
Specia, Lucia, Nicola Cancedda, Marc Dymetman, Marco Turchi, and Nello Cristianini
2009 “Estimating the Sentence-level Quality of Machine Translation Systems.” In Proceedings of the 13th Annual Conference of the EAMT, Barcelona, 28–35.Google Scholar
Specia, Lucia
2011 “Exploiting Objective Annotations for Measuring Translation Post-editing Effort.” In Proceedings of the 15th conference of EAMT, Leuven, 73–80.Google Scholar
Su, Keh-Yih, Ming-Wen Wu, and Jing-Shin Chang
1992 “A New Quantitative Quality Measure for Machine Translation Systems.” In Proceedings of COLING-92, Nantes, 433–439.Google Scholar
Teixeira, Carlos S. C.
2014 “Perceived vs. Measured Performance in the Post-editing of Suggestions from Machine Translation and Translation Memories.” In Proceedings of AMTA 2014 Workshop on Post-editing Technology and Practice, Vancouver, 45–59.Google Scholar
Turian, Joseph P., Luke Shen, and I. Dan Melamed
2003 “Evaluation of Machine Translation and Its Evaluation.” In Proceedings of MT Summit 2003, 386–393. New Orleans, Louisiana.Google Scholar
Vasconcellos, Muriel, and Marjorie León
1985 “SPANAM and ENGSPAN: Machine Translation at the Pan American Health Organization.” Computational Linguistics 11 (2–3): 122–136.Google Scholar
Wagner, Emma
1985 “Post-editing Systran: A Challenge for Commission Translators.” Terminologie et Traduction 3: 1–7.Google Scholar
Way, Andy
2013 “Traditional and Emerging Use-cases for Machine Translation.” In Proceedings of Translating and the Computer 35.Google Scholar
Wilms, Franz-Josef M.
1981 “Von SUSY zu SUSY-BSA: Forderungen and ein anwenderbezogenes MU-System.” Sprache und Datenverarbeitung 5: 38–43.Google Scholar
Cited by

Cited by 10 other publications

Cui, Ying, Xiao Liu & Yuqin Cheng
2023. A Comparative Study on the Effort of Human Translation and Post-Editing in Relation to Text Types: An Eye-Tracking and Key-Logging Experiment. SAGE Open 13:1 ► pp. 215824402311558 ff. DOI logo
Cumbreño, Cristina & Nora Aranberri
2021. What Do You Say? Comparison of Metrics for Post-editing Effort. In Explorations in Empirical Translation Process Research [Machine Translation: Technologies and Applications, 3], ► pp. 57 ff. DOI logo
Jiménez-Crespo, Miguel A.
2020. The “technological turn” in translation studies. Translation Spaces 9:2 ► pp. 314 ff. DOI logo
Kasperė, Ramunė, Jurgita Motiejūnienė, Irena Patasienė, Martynas Patašius & Jolita Horbačauskienė
2023. Is machine translation a dim technology for its users? An eye tracking study. Frontiers in Psychology 14 DOI logo
Kruger, Jan-Louis
2021. Eye tracking. In Handbook of Translation Studies [Handbook of Translation Studies, 5], ► pp. 80 ff. DOI logo
Sánchez-Gijón, Pilar, Joss Moorkens & Andy Way
2019. Post-editing neural machine translation versus translation memory segments. Machine Translation 33:1-2 ► pp. 31 ff. DOI logo
Tardel, Anke
2021. Measuring Effort in Subprocesses of Subtitling. In Explorations in Empirical Translation Process Research [Machine Translation: Technologies and Applications, 3], ► pp. 81 ff. DOI logo
Tosun, Sümeyra
2024. Machine translation: Turkish–English bilingual speakers’ accuracy detection of evidentiality and preference of MT. Cognitive Research: Principles and Implications 9:1 DOI logo
Walker, Callum
2021. The Cognitive Paradigm in Translation Studies. In An Eye-Tracking Study of Equivalent Effect in Translation, ► pp. 13 ff. DOI logo
Zou, Longhui, Michael Carl & Devin Gilbert
2023. Integrating Trados-Qualitivity Data to the CRITT TPR-DB: Measuring Post-editing Process Data in an Ecologically Valid Setting. In Corpora and Translation Education [New Frontiers in Translation Studies, ], ► pp. 63 ff. DOI logo

This list is based on CrossRef data as of 22 april 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.