BLEU vs COMET for Regulated Translation: Which Metric to Use
Table of Contents
Direct answer
BLEU and COMET measure different things and should be reported together. BLEU counts n-gram overlap with a human reference. COMET is a trained neural model that estimates quality from the source, the translation and the reference. For regulated content in the United States, use COMET for semantic comparison, BLEU or chrF as a lexical guardrail, and human review for critical errors. No single score is an acceptance rule.
Definition and scope
What BLEU measures
BLEU stands for Bilingual Evaluation Understudy. The original paper proposed it as an automatic method for evaluating machine translation. It computes modified n-gram precision between a candidate translation and one or more human reference translations. It applies a brevity penalty to short output.
A higher score means more surface-form overlap with the reference. It does not mean the meaning is correct. Many tools display BLEU on a 0 to 100 scale, while Google Cloud describes it as closer to 1 when the candidate is closer to the reference.
What COMET measures
COMET is a neural framework for training MT evaluation models. It uses multilingual pretrained representations and human judgment data. It can take the source, the candidate and the reference, and predict a quality score or rank competing translations.
The authors note that modern neural translation can legitimately deviate from a reference’s wording. Overlap metrics such as BLEU can misrepresent quality in that case. COMET was designed to address this gap.
COMET is a family of models. The score range and meaning depend on the model and the setup. Google Cloud describes reference-based COMET on a 0 to 1 scale, where 1 is a perfect translation. Other implementations expose other ranges.
Scope of this reference
This text covers evaluating machine translation output for regulated documents. Examples are clinical documentation, medical device instructions for use, legal contracts, financial disclosures and defense technical manuals. It applies to buyers and operators in the United States. It does not state any regulatory threshold, because no regulator named in this text sets a BLEU or COMET value for acceptance.
Terminology note: MT and NMT systems are not the same as AI translation. BLEU and COMET evaluate MT output. They are not the quality process itself.
Side-by-side comparison
Criterion | BLEU | COMET |
Method | Direct n-gram calculation | Trained neural regression or ranking model |
Inputs | Candidate and reference | Source, candidate and (reference-based) reference |
Strength | Interpretable, traceable to specific n-grams | Better aligned with human judgments overall |
Weakness | Penalizes valid paraphrase and word order changes | Model dependent, can miss number and entity errors |
Cost | Low | Higher compute, needs model and language coverage |
Reproducibility control | Tokenization and reference set | Model name, version and inputs |
Portability of score | Not portable across corpora or languages | Not portable across models or setups |
The BLEU Meets COMET study by Glushkova, Zerva and Martins supports the combined view. COMET aligns better with human judgments overall. BLEU and chrF stay valuable because their lexical evidence is more interpretable and can expose certain critical deviations.
Steps to evaluate a translation system with both metrics
Each step lists an owner and a control.
Build the test set. Owner: localization manager with a subject-matter expert. Control: a held-out set that is representative of the real document type and never used for training or tuning. Record the language pair, domain and reference construction.
Fix the references. Owner: linguist lead. Control: version-controlled reference translations with access control, so that no one edits them after scoring begins.
Compute BLEU reproducibly. Owner: MT engineer. Control: use SacreBLEU, which produces shareable and comparable scores, and store the signature string with every result.
Compute COMET with a documented model. Owner: MT engineer. Control: log the exact model, version, inputs, language coverage and whether the run is reference-based or reference-free.
Test significance. Owner: MT engineer. Control: use paired bootstrap resampling or paired approximate randomization. The SacreBLEU documentation lists default values of 1,000 resamples for bootstrap confidence and paired bootstrap, and 10,000 for paired approximate randomization.
Compare against a validated baseline. Owner: localization manager. Control: judge the delta against the incumbent system on the same test set, never against a generic number.
Add targeted checks. Owner: QA lead. Control: automated checks for numbers, units, product names, dosage terms and safety terminology.
Run human review. Owner: subject-matter expert reviewer. Control: an error-classification review, such as an MQM-style scheme, with sign-off recorded in an audit trail.
Archive the evidence. Owner: compliance lead. Control: store metric outputs, signatures, model versions and review records with retention rules, so an auditor can reproduce the decision.
Decision criteria
When to weight BLEU
Use BLEU as a regression guardrail when a system is retrained or a prompt, glossary or translation memory changes. A drop in BLEU or chrF on a fixed test set flags a lexical change that can be traced to specific n-grams. This is useful when terminology must remain stable.
Phrase’s documentation on MT metrics shows one practical rule set: a BLEU improvement of at least 5 absolute points, or a chrF improvement of at least 4, with no significant decrease in COMET. This is one vendor’s configuration. It is not a regulatory standard, and it should be validated on your own data before use.
When to weight COMET
Use COMET when comparing systems that produce different but valid wordings. This includes comparing two engines, two vendors, or a fine-tuned model against a general one. COMET is more resilient to synonyms and paraphrase, where BLEU may score low on acceptable output.
When neither is enough
Automatic metrics are proxies. The Cambridge review of machine translation evaluation states that human translators remain the most appropriate judges of translation quality. A survey of evaluation metrics reaches a similar position: automatic metrics are cost-effective and easy to use for comparing systems, but their quality is not as good as human evaluation.
For a release or compliance decision, a score supports the decision. It does not make it.
Decision matrix
Situation | Primary metric | Required addition |
Comparing two engines for clinical documents | COMET | BLEU delta, human review of critical errors |
Checking a glossary update for a device manual | BLEU or chrF | Terminology check, COMET non-regression |
Vendor selection for contracts | COMET and BLEU | MQM-style review by legal reviewers |
Marketing or literary content | COMET | Document-level evaluation and human review |
Safety-critical text with doses and units | Neither alone | Automated number checks, SME sign-off |
Examples
The examples below are illustrative. The figures are invented to show reasoning, not reported results.
Example 1: paraphrase scored low by BLEU
The reference reads “Store the device at room temperature.” The candidate reads “Keep the device at ambient temperature.” The meaning is the same. BLEU gives a low score because few n-grams match. COMET, which compares meaning representations, would be expected to score it higher. If a team used BLEU alone, it might reject a correct translation.
Example 2: number error missed by a neural metric
The reference says “Administer 15 mg.” The candidate says “Administer 50 mg.” The sentence is otherwise fluent. Research summarized in the BLEU Meets COMET paper reports that neural metrics can miss deviations in numbers, named entities, deletions and insertions. A fluent, high-scoring sentence can still carry a dangerous error. An automated number check and SME review catch it.
Example 3: scores from a shared task
The WMT 2024 literary translation findings report BLEU, chrF2, COMET, TER and document-level d-BLEU in separate columns. A Google baseline row shows BLEU 37.4 and COMET 80.50. These values use different scales and different numbers of references, since COMET was computed with one reference and the other metrics with two. The official ranking was based on human judgments. The numbers are task-specific and are not thresholds.
Failure modes and mitigations
Failure mode | Cause | Mitigation |
Generic threshold used as acceptance rule | Belief that a BLEU of 30 or 50 is universally good | Google Cloud warns that scores should not be compared across corpora and languages. Set thresholds per language pair, domain and baseline |
Irreproducible BLEU | Different tokenizers or implementations | Use SacreBLEU and store signatures |
Unlogged COMET model | Model updated silently | Pin model name and version, store in the audit trail |
Unsupported language | COMET model lacks coverage | Verify language coverage before scoring, use human review otherwise |
Sentence scores hide discourse problems | Sentence-level metrics ignore context | Add document-level evaluation, as WMT 2024 did with d-BLEU |
Small differences treated as real | No significance testing | Use paired bootstrap or approximate randomization |
Test set contamination | Test sentences used in training or glossary | Hold out data under access control |
Score replaces review | Metric treated as quality assurance | Require SME sign-off for regulated content |
Data handling is a separate failure mode. Scoring regulated documents often means moving source text and references into evaluation tools. Access control, hosting location and vendor terms must be checked before documents are sent to any service. AD VERBUM discusses these exposures in its article on machine translation risks, security and compliance.
Our perspective
We see the BLEU versus COMET question as a governance question. A localization team in a regulated industry is not choosing a metric. It is deciding what evidence will support a release, and who can reproduce that evidence later.
In our work, scores are inputs. We treat BLEU and COMET as screening and monitoring tools. They show whether a change moved output in a measurable direction. They do not show whether a dosage statement, a contractual obligation or a defense specification is correct.
For that reason our process is AI+HUMAN hybrid. Subject-matter experts review the output, including medical professionals, engineers and legal scholars from our network of 3,500+ linguists. Terminology comes from the client’s own term bases and translation memories. Quality controls follow ISO 17100 and ISO 18587, and security controls follow ISO 27001 and ISO 42001.
We recommend that buyers ask any vendor for three records: the metric signatures and model versions used, the test set description, and the human review log. A vendor that can produce all three has a process that can be audited.
— Eric Brown
Where AD VERBUM fits
AD VERBUM is a language service provider with 25+ years of experience. It operates a proprietary LangOps System hosted on EU servers and delivers AI+HUMAN hybrid translation in 150+ languages, including regional variants. Subject-matter experts review output for regulated sectors such as life sciences, legal, finance, defense and manufacturing. Certifications include ISO 9001, ISO 17100, ISO 18587, ISO 13485, ISO 27001, ISO 42001 and AQAP2110, audited by Bureau Veritas. Pricing is by quote. See the translation, localization and related services.
Key points
BLEU measures n-gram overlap with a reference. COMET is a trained model that estimates quality.
COMET aligns better with human judgments overall. BLEU and chrF are more interpretable.
Neither metric is portable across language pairs, corpora, models or setups.
Use SacreBLEU signatures and log the exact COMET model and version.
Test significance with paired bootstrap or approximate randomization.
Neural metrics can miss numbers, entities, deletions and insertions. Add explicit checks.
Regulated content needs human SME review and an audit trail, not a score alone.
Do not apply a generic score threshold as an acceptance rule.
FAQ
What is a good BLEU score?
No universal value exists. Google Cloud states that BLEU should not be compared across different corpora and languages. A useful target is set on the same language pair, domain, reference set and tokenizer, against a validated baseline.
Is COMET better than BLEU?
COMET correlates better with human judgments in most comparisons, and it handles paraphrase better. BLEU is cheaper, easier to trace and useful as a regression check. Research on combining them shows each catches problems the other can miss.
Can BLEU or COMET approve a translation for regulatory use?
No. Both are automatic proxies. For high-stakes content, a qualified human reviewer should judge critical errors, terminology and regulatory meaning, and the review should be recorded. Ask your regulatory affairs or quality lead which documentation your submission requires.
Do BLEU and COMET need a reference translation?
BLEU always needs one or more references. COMET can run reference-based or reference-free, depending on the model. Record which mode was used, since the scores are not comparable between modes.
Are BLEU and COMET suitable for LLM-based translation?
Both can score LLM output, but they remain proxies. Document-level quality, consistency and style may need document-level metrics and human review. Validate any metric on your own language pairs and content before relying on it.
Sources
BLEU: a Method for Automatic Evaluation of Machine Translation
BLEU Meets COMET: Combining Lexical and Neural Metrics Towards Robust Machine Translation Evaluation
Findings of the WMT 2024 Shared Task on Discourse-Level Literary Translation
How to evaluate machine translation: A review of automated and human metrics
