ADAPT's Dr Joos Moorkens received the best paper award at the Computer 38 (#TC38) in London recently. Dr Moorkens presentation, titled A Crowd-sourced Comparative Evaluation of Phrase-Based SMT and Neural Machine Translation, was highly praised during the event and beat off competition to take the top prize.
The use of machine translation (MT) has become widespread since statistical machine translation (SMT) became the dominant paradigm. However, there is growing interest in the research community in the possibilities of neural machine translation (NMT) based largely on impressive results in automatic evaluation. There has to date been no published large-scale human evaluations of NMT output. This presentation reported on a comparative human evaluation of phrase-based SMT and NMT in four language pairs, using a crowdsourcing platform to compare output from both systems using a variety of metrics. These metrics comprise automatic evaluation, human rankings of adequacy and fluency, error-type markup, and post-editing effort (technical and temporal effort). This evaluation is part of the work of the TraMOOC project, which aims to create a replicable semi-automated methodology for high-quality MT of educational data. While the primary intention for this evaluation is to identify the best MT paradigm for the proposed methodology for TraMOOC, it is believed that the evaluation results will be of interest to the wider research community and to those in the translation industry interested in the deployment of cutting-edge MT systems.