Research
Efficient models, from parameters to pipelines
My research focuses on making large language models less expensive to train, adapt, and deploy. I study optimization methods that exploit sparsity, low precision, second-order information, and distributed model structure. Alongside theoretical work, I build practical implementations and open-source tools.
Overview
Compression and efficient optimization
Large models are costly not only because they contain many parameters, but because training and deployment repeatedly move, store, and update those parameters. My work looks for useful structure in that cost: which weights can be removed, which need to change during adaptation, how much numerical precision is necessary, and how optimization behaves when a model is split across pipeline stages.
01 · Sparsity
LLM pruning
Pruning removes model weights that contribute less to a chosen objective. The difficult part is not producing zeros; it is deciding what to remove and compensating for the combined effect. My work studies layer-wise and block-wise methods that use second-order information for this decision.
02 · Adaptation
Sparse fine-tuning
Sparse fine-tuning updates a selected subset of model parameters while keeping the rest fixed. This turns parameter selection into part of the optimization problem and can reduce the amount of trainable state needed to adapt a model.
03 · Low precision
Quantization
Quantization represents model values with fewer bits. Lower precision can reduce storage and memory traffic, but it also introduces approximation error. My research in this area examines optimization methods that control that trade-off without assuming the straight-through estimator is the only available approach.
04 · Distributed training
Pipeline parallelism
Pipeline parallelism divides a model into stages that can work on different microbatches concurrently. The resulting delays make its optimization dynamics different from ordinary synchronous training. I study theoretical models that make these delayed updates explicit and allow convergence to be analyzed.