Researchers evaluated 12 open-weight models on two Ukrainian tasks: news classification and legal case outcome prediction. While some models improved significantly, others degraded when few-shot prompting was applied.
On the news classification task, models gained an average of 24 percentage points in accuracy. In contrast, performance gains on the legal text task were minimal at 3.4 percentage points, with two models showing degradation.
Previous research measured hidden state shifts between zero-shot and few-shot modes to understand this phenomenon. However, the study notes that few-shot prompts are longer than zero-shot prompts, and this length difference alone alters model representations.
To address this confounding factor, the authors proposed using length-matched random text as a control. By subtracting the representation shift caused by prompt length from the total shift, they created a content delta metric.
This metric revealed that raw shift does not predict whether few-shot prompting helps or hurts (correlation coefficient of 0.20). Conversely, content delta showed a strong positive correlation (rho = +0.65) with performance changes. Causal masking experiments on Llama 3.3 70B confirmed these findings.
Source: https://arxiv.org/abs/2609.15990
