Text-to-LoRA
Text-to-LoRA
Text-to-LoRA (T2L) is a hypernetwork from Sakana AI that generates LoRA adapters for large language models from a natural language task description, in a single forward pass. Published in 2025, T2L eliminates the traditional fine-tuning loop — instead of curating a dataset and training a LoRA adapter for each task, users describe what they want and receive an instant adapter.
How It Works
- A text encoder processes the task description (e.g., “answer multiple-choice science questions”)
- The hypernetwork (a series of MLP layers) generates the low-rank A and B matrices for all layers and module types in the target LLM
- The generated LoRA weights are applied to the base model, specializing it for the described task
Three architecture variants explore different capacity tradeoffs: L (outputs both A and B), M (shared output layer for A and B), S (outputs only one rank). The M variant offers the best balance of performance and efficiency.
Key Results
- Compression matches task-specific training. T2L trained to reconstruct 9 benchmark-specific LoRAs achieved 73.4% average accuracy — matching the 73.0% of the individual LoRAs.
- Zero-shot generalization. Trained with SFT on 479 tasks, T2L generated useful LoRA adapters for entirely unseen benchmarks, outperforming multi-task LoRA baselines.
- Semantic LoRA clusters. Generated LoRAs form meaningful clusters in a reduced representation space — similar tasks produce similar adapters, suggesting the hypernetwork learns a structured LoRA manifold.
- Works across model families. Evaluated on LLaMA and Gemma base models.