Merging a LoRA adapter into a base model typically removes runtime overhead and produces a standalone checkpoint. For native 4-bit microscaling checkpoints, this process involves re-deriving the checkpoint's code plane through a quantizer, which can cause accuracy loss and coupling to a quantization convention.
Scale-QLoRA addresses this by adapting only the native per-block scale field, training these scales on the deployment grid, and freezing the code plane. This approach maintains a bit-exact, code-invariant artifact within a fixed native format, block layout, and code plane.
Across multiple models and tasks, Scale-QLoRA achieves accuracy-lossless merging, comparable to merge-aware QAT-LoRA, but with structural differences. It simplifies lifecycle management, enables exact rollback, code-plane deduplication, and significantly faster scale-only task swaps.
This method reduces training overhead, dropping the straight-through estimator, and enhances deployment stability and speed.
Source: https://arxiv.org/abs/2609.04526