| Uniform HETLoRA | Weighted HETLoRA | FLoRA | RB-LoRA | |||||
|---|---|---|---|---|---|---|---|---|
| Dataset | 1-Shot | 3-Shot | 1-Shot | 3-Shot | 1-Shot | 3-Shot | 1-Shot | 3-Shot |
| Dolly | 0.53 | 0.51 | 0.54 | 0.52 | 0.26 | 0.26 | 0.57 | 0.57 |
| Alpaca | 0.52 | 0.52 | 0.51 | 0.52 | 0.31 | 0.31 | 0.54 | 0.54 |
Table 1: MMLU accuracy evaluated on a 1,444-question subset under 1- and 3-shot communication settings on LLaMa3-8b.
Federated fine‑tuning of foundation models is impeded by the need to communicate billions of parameters. Low‑rank adaptation (LoRA) alleviates this by updating only compact adapter matrices. However, varying client device capabilities lead to different adapter ranks, causing rank heterogeneity that undermines aggregation, and existing reconciliation methods still incur bias or inefficiency. To address this challenge, we propose \emph{RB-LoRA}, a principled rank‑balanced aggregation framework that decomposes each update into rank‑wise components and aligns them using analytically derived weights. Experiments on both language and vision models demonstrate consistent improvements under one and three rounds of communication in federated learning.
In federated LoRA, clients fine-tune and transmit only their adapter matrices, preserving privacy and reducing communication costs. However, clients in practice often adopt different adapter ranks according to their computational capacities, leading to rank heterogeneity where client updates reside in distinct low-dimensional subspaces.
Existing methods attempt to heuristically mitigate rank heterogeneity through zero-padding, replication, and stacking. However, these heuristics only try to make updates with different ranks look the same without any analytical basis. As a result, they may unintentionally prioritize either low-rank or high-rank clients, leading to degraded performance. Furthermore, existing studies do not account for data heterogeneity, where client contributions should be weighted by local dataset size according to FedAvg principles.
RB-LoRA addresses these limitations by formulating rank-balanced aggregation as a weighted-alignment optimization, decomposing adapter updates rank-wise and deriving weights analytically from data volume and rank rarity.
Rank-Wise Decomposition: RB-LoRA begins with a rank-wise decomposition of each client's LoRA update, expressing it as a sum of rank-one matrices formed by outer products of basis vectors.
Generalized Aggregation Representation: We align heterogeneous LoRA updates to a common rank via zero-padding and formulate the aggregated update as a weighted alignment optimization. This unified framework subsumes prior methods (zero-padding, replication, stacking, sketching) as specific choices of the weighting matrix.
Analytically Derived Weights: Our factorized weighting scheme combines two factors: (1) data proportionality (αk), ensuring client contributions scale with local dataset size; (2) rank rarity (βr), compensating for the imbalance where higher-rank components contribute more directions. The weights are derived as γr(k) = αk · βr, providing a principled approach beyond heuristics.
Projection: After aggregation, the global adapter is projected to each client's local rank using SVD-based rank reduction, enabling seamless continuation in subsequent rounds.
We evaluated RB-LoRA on federated fine-tuning of language models (LLaMA2/3) and vision transformers. The experiments were conducted on Alpaca and Dolly datasets distributed across 10 simulated clients with non-IID splits. As shown in Table 1 (above), RB-LoRA consistently outperforms all baselines—Uniform HETLoRA, Weighted HETLoRA, and FLoRA—in both 1-shot and 3-shot communication settings.
RB-LoRA achieves improvements in MMLU accuracy while maintaining comparable perplexity on WikiText-2 and PTB. The method demonstrates stronger generalization on zero-shot reasoning benchmarks, achieving consistently higher accuracy than other HETLoRA-based methods across both common and advanced task groups.
On vision transformers, RB-LoRA also shows consistent improvements, demonstrating its modality-agnostic robustness. The method accelerates convergence and improves final top-1 accuracy on the Food-101 dataset while maintaining linear communication and computation scaling.
| Method | #Params/round | Complexity |
|---|---|---|
| Uniform HETLoRA | 1.00× | O(d²KR) |
| Weighted HETLoRA | 1.00× | O(d²KR) |
| FLoRA | 1.57× | O(dK²R²) |
| RB-LoRA | 1.00× | O(d²KR) |
RB-LoRA maintains the same parameter transmission overhead as Uniform/Weighted HETLoRA (1.00Ă—) while achieving superior accuracy. In contrast, FLoRA uses about 1.57Ă— more global parameters and incurs quadratic aggregation complexity as the number of clients increases. RB-LoRA achieves a better trade-off between accuracy and efficiency, with linear scaling in both communication and computation.
Our factorized weighting scheme uses only O(K + R) parameters, compared to O((KR)²) entries in a full aggregation matrix. Consequently, the aggregation complexity is reduced from O(d(KR)²) to O(d²KR), enabling efficient federated fine-tuning even with a large number of clients.
@inproceedings{ha2026rb,
title={RB-LoRA: Rank-Balanced Aggregation for Low-Rank Adaptation with Federated Fine-Tuning},
author={Ha, Sihyeon and Oh, Yongjeong and Jeon, Yo-Seb},
booktitle={Findings of the Association for Computational Linguistics: EACL 2026},
pages={1737--1746},
year={2026}
}