76th Deep Learning Meetup: MatryoshkaLoRA / Agents of Chaos

Hi Deep Learners,

We are happy to announce our upcoming Vienna Deep Learning Meetup on October 21 at FH Technikum Wien. Our Agenda:

*** Agenda:

• 18:15 Arrival • 18:30 Welcome by the meetup organizers • Introduction by the hosts • 18:45 Talk 1: MatryoshkaLoRA: Learning Accurate Hierarchical Low-Rank Representations for LLM Fine-Tuning by Ionut-Vlad Modoranu *(ISTA)* • 19:30 Announcements • Networking Break • 20:00 Talk 2: Agents of Chaos by Jacob Palecek *(aots.io)* • 20:30 Networking • \~22:00 Wrap up & End

***

Talk Details: -------------- Talk 1: MatryoshkaLoRA: Learning Accurate Hierarchical Low-Rank Representations for LLM Fine-Tuning With the rise in scale for deep learning models to billions of parameters, the computational cost of fine-tuning remains a significant barrier to deployment. While Low-Rank Adaptation (LoRA) has become the standard for parameter-efficient fine-tuning, the need to set a predefined, static rank r requires exhaustive grid searches to balance efficiency and performance. Existing rank-adaptive solutions such as DyLoRA mitigate this by sampling ranks during the training from a predefined distribution. However, they often yield sub-optimal results at higher ranks due to lack of consistent gradient signals across the full hierarchy of ranks, thus making these methods data-inefficient.

In this talk, we propose MatryoshkaLoRA, a general, Matryoshka-inspired training framework for LoRA that learns accurate hierarchical low-rank representations by inserting a fixed, carefully crafted diagonal matrix P between the existing LoRA adapters to scale their sub-ranks accordingly. By introducing this simple modification, our general framework recovers LoRA and DyLoRA only by changing P and ensures all sub-ranks embed the available gradient information efficiently.

Our MatryoshkaLoRA supports dynamic rank selection with minimal degradation in accuracy. We further propose Area Under the Rank Accuracy Curve (AURAC), a metric that consistently evaluates the performance of hierarchical low-rank adapters. Our results demonstrate that MatryoshkaLoRA learns more accurate hierarchical low-rank representations than prior rank-adaptive approaches and achieves superior accuracy performance trade-offs across ranks on the evaluated datasets

About the speaker: Ionut-Vlad Modoranu is a PhD student at the Institute of Science and Technology Austria (ISTA), specializing in efficient optimization for deep learning. His research focuses on reducing the memory usage and computational cost while maintaining the performance, including the development of practical optimizers for large-scale models, with publications at top tier international conferences.

Talk 2: Agents of Chaos by Jacob Palecek *(aots.io)* (details will follow)

We are looking forward to welcoming you at this meetup! Your VDLM organizer team