MixLoRA-DSI: Dynamically Expandable Mixture-of-LoRA Experts for Rehearsal-Free Generative Retrieval over Dynamic Corpora
Researchers have developed a new framework called MixLoRA-DSI to improve generative retrieval over dynamic corpora. This is a challenge because updating models with new documents can be computationally expensive and impractical under resource constraints. The proposed framework combines an expandable mixture of Low-Rank Adaptation experts with a layer-wise out-of-distribution-driven expansion strategy, which enables sublinear parameter growth by selectively introducing new ex
Researchers have developed a new framework called MixLoRA-DSI to improve generative retrieval over dynamic corpora. This is a challenge because updating models with new documents can be computationally expensive and impractical under resource constraints. The proposed framework combines an expandable mixture of Low-Rank Adaptation experts with a layer-wise out-of-distribution-driven expansion strategy, which enables sublinear parameter growth by selectively introducing new experts only when significant numbers of out-of-distribution documents are detected. Experiments show that MixLoRA-DSI outperforms full-model update baselines with minimal parameter overhead and lower training costs.
---
Why it matters: This matters to researchers in AI because it addresses a common challenge in generative retrieval, where models need to adapt to new data without requiring significant computational resources or retraining from scratch. The proposed framework's ability to selectively introduce new experts when necessary can lead to more efficient and scalable model updates.
Source: https://arxiv.org/abs/2507.09924
This article was originally published at: https://arxiv.org/abs/2507.09924