My complete publication list, newest first.
Analyzes PipeDream, a way to train models by splitting their layers across multiple devices. It proves when this pipeline approach converges and shows how outdated updates worsen as more stages are added, clarifying when pipeline parallelism will scale well.
Adapts Muon-style momentum, a promising neural-network optimizer, to distributed systems where workers run at different speeds. By discarding updates that arrive too late, Ringmaster LMO avoids waiting for stragglers while retaining convergence guarantees and strong training performance.
Repurposes activation-aware pruning scores to choose a small, fixed set of LLM weights for fine-tuning. A hybrid version adds LoRA without increasing the trainable-parameter budget, offering a memory-efficient adaptation strategy that performed strongly on arithmetic tasks.
Shows how to use PyTorch's automatic differentiation to compute selected pieces of an LLM's Hessian—the matrix describing local curvature—and estimate its full diagonal. The open-source implementation makes second-order analysis practical without storing the impossibly large full matrix.
Shows how errors from compressing each LLM layer increase perplexity, a measure of prediction quality. This result enables HIGGS, a quantization method needing no calibration data, and an optimal way to distribute bit widths across layers under a fixed budget.
Prunes LLM weights in blocks, using second-order information about how weights interact to compensate for removals and adaptive masks to revise earlier choices. It supports flexible and hardware-friendly sparsity patterns, reducing model size while preserving accuracy.
Studies how to fine-tune LLMs compressed to only one or two bits per parameter. PV-Tuning moves beyond the commonly used straight-through estimator through a more general optimization framework, improving compressed Llama and Mistral models for memory-limited devices.
Designs an asynchronous training method for systems whose workers differ in both computation and communication speed. Shadowheart compresses updates and avoids waiting for slow workers, with proofs that no comparable centralized compressed method can achieve a better overall training time.
Monitors changing network bandwidth during distributed training and adjusts each worker's overall gradient-compression budget, then distributes that budget across neural-network layers to minimize compression error. This reduces communication time while retaining the convergence behavior of fixed-ratio compression.