TITANIA Protocol for the Evaluation of Automated Translation

Abstract

Research on automated translation (AT) quality — encompassing Neural Machine Translation (NMT) and Large Language Model (LLM)-based systems — suffers from recurring methodological weaknesses that undermine the validity of its findings: evaluation corpora built from isolated, decontextualized sentences rather than full texts; reference translations produced by students or crowdsourced workers rather than qualified professionals; conflation of the roles of reference translator and error annotator in a single informant; misalignment between the quality dimensions chosen for evaluation and the conditions under which translations were produced; and statistical robustness tests applied to fundamentally biased datasets. Taken together, these practices generate results that may appear scientifically sound but are, in fact, methodologically indefensible. The TITANIA Protocol for the Evaluation of Automated Translation, developed within the TITANIA research line at the Universidad Pablo de Olavide (PI: Elisa Alonso Jiménez), addresses each of these problems through a set of evidence-based, operationalizable requirements. Grounded in functionalist translation theory — which treats translation as a purposeful communicative act inseparable from its target audience, genre, and social function — the protocol establishes standards across eight areas: the qualification of reference translators (ISO 17100; UNE-ISO 18587:2020); the mandatory specification of a full translation brief prior to any translation activity; corpus composition (full, real texts with explicit metadata); traceability and open data publication; error annotation procedures and annotator qualification; the alignment of evaluation dimensions with research design; the strict separation of evaluator roles; and the appropriate application of statistical analysis. The protocol also provides an eleven-step evaluation workflow and a reporting checklist to ensure reproducibility. It is applied within TITANIA's study of NMT and generative AI in two genres — corporate websites and news — where failures in persuasive function and framing have measurable real-world consequences that automatic metrics consistently fail to detect. The protocol is intended as a shared resource for the translation studies and NLP research communities.