A new study challenges the idea that prompt engineering can reliably boost AI translation performance in large language models (LLMs).
Researchers from Charles University, Johns Hopkins, LMU Munich, ETH Zurich, University of Amsterdam, and Poznan University of Technology argue that if a model cannot generalize — apply what it learned during training to new, unseen inputs — no amount of carefully worded prompting will improve the output.
In their July 13, 2025 paper, they tested six LLMs, including GPT-4o-mini, Gemini-2.0-flash, Llama-3.1, Qwen2.5, EuroLLM, and TowerInstruct, across three language pairs: Czech–Ukrainian, German–English, and English–Chinese.
The researchers systematically introduced different types of “noise” into prompts, from typos and phonetic spellings to informal register shifts and simplifications, to measure how errors affect AI translation and evaluation.
Noise Affects Instruction Following More Than Translation
They found that “prompt quality strongly affects the translation performance,” but not always in predictable ways — the effect depends heavily on the type of error. Random character-level noise and phonetic errors had the strongest negative impact, while phrasal simplifications sometimes even improved performance by making instructions clearer.
If a model has not learned to translate a given language pair or style, no amount of carefully worded prompting will make it perform better.
Lower-quality prompts mainly reduced models’ ability to follow instructions, often leading to redundant phrases such as “Here is your translation”, rather than directly affecting translation quality itself.
That means the actual translations may still be retrievable and of appropriate quality, though this would fail in automated pipelines.
High prompt noise also increased the rate of off-target language outputs, with models often producing translations in the wrong language, particularly in less well-supported pairs such as Czech–Ukrainian.
Despite heavy prompt distortion, models were consistently able to identify the task and deliver translations far beyond what the input text appeared to allow. “LLMs are capable of providing translations even when the prompt is illegible to humans,” they said.
Limits of Prompt Engineering
But the researchers emphasize a bigger point: prompt engineering has limits. They stress that the real bottleneck is whether an LLM can generalize. If a model has not learned to translate a given language pair or style, no amount of carefully worded prompting will make it perform better.
For the language industry, the implications are clear. LLMs can handle imperfect English prompts, but prompt engineering should not be expected to deliver sustained improvements in such cases. When LLMs fail, it is not because the prompt was poorly written, it is because the model itself lacks the necessary competence.
That means the focus should shift from endless prompt tweaking to the fundamentals of model capability and training.
Authors: Patrícia Schmidtová, Niyati Bafna, Seth Aycock, Gianluca Vico, Wiktor Kamzela, Katharina Hämmerl, and Vilém Zouhar
