New blog posts, publications, and projects from Ivan Ilin.https://ivanilin.org/https://ivanilin.org/blog/phd-proposal-defense/https://ivanilin.org/blog/phd-proposal-defense/My proposal connects LLM pruning, sparse fine-tuning, and pipeline-parallel optimization into one research program.Wed, 22 Jul 2026 00:00:00 GMTenBlog postResearchMilestoneKAUSThttps://ivanilin.org/blog/trace-strg-award/https://ivanilin.org/blog/trace-strg-award/A USD 100,000 Small Translational Research Grant will support TRACE, a project for turning textbooks into verified, interactive courses.Mon, 01 Jun 2026 00:00:00 GMTenBlog postProjectsGrantKAUSThttps://ivanilin.org/publications/#demystifying-pipeline-parallelismhttps://ivanilin.org/publications/#demystifying-pipeline-parallelismPublication — Analyzes PipeDream, a way to train models by splitting their layers across multiple devices. It proves when this pipeline approach converges and shows how outdated updates worsen as more stages are added, clarifying when pipeline parallelism will scale well.enPublicationPreprintPipeline parallelismDistributed optimizationDelayed updateshttps://ivanilin.org/projects/pipeline-parallelism-theory/https://ivanilin.org/projects/pipeline-parallelism-theory/Project — Randomized PipeDream captures PipeDream's stale-weight behavior in an analyzable block-SGD model, revealing how pipeline depth affects convergence.enProjectResearch / Distributed OptimizationPipeline parallelismDistributed optimizationDelayed updateshttps://ivanilin.org/publications/#ringmaster-lmohttps://ivanilin.org/publications/#ringmaster-lmoPublication — Adapts Muon-style momentum, a promising neural-network optimizer, to distributed systems where workers run at different speeds. By discarding updates that arrive too late, Ringmaster LMO avoids waiting for stragglers while retaining convergence guarantees and strong training performance.enPublicationPreprintAsynchronous optimizationMomentumLinear minimization oraclehttps://ivanilin.org/projects/sparse-fine-tuning/https://ivanilin.org/projects/sparse-fine-tuning/Project — We introduce Super, which selects a sparse trainable support using activation-aware pruning scores, and Supra, a matched-budget sparse-plus-LoRA adapter.enProjectResearch / Efficient AdaptationSparse fine-tuningEfficient adaptationLLM pruninghttps://ivanilin.org/publications/#super-tuninghttps://ivanilin.org/publications/#super-tuningPublication — Repurposes activation-aware pruning scores to choose a small, fixed set of LLM weights for fine-tuning. A hybrid version adds LoRA without increasing the trainable-parameter budget, offering a memory-efficient adaptation strategy that performed strongly on arithmetic tasks.enPublicationPreprintSparse fine-tuningLLM pruningEfficient adaptationhttps://ivanilin.org/blog/teaching-cs-260/https://ivanilin.org/blog/teaching-cs-260/This fall I am supporting Design and Analysis of Algorithms through office hours, assignments, grading, and exams.Sun, 14 Sep 2025 00:00:00 GMTenBlog postTeachingAlgorithmsKAUSThttps://ivanilin.org/blog/kaust-graduate-seminar/https://ivanilin.org/blog/kaust-graduate-seminar/A recording of my department-wide Computer Science Graduate Seminar talk on efficient pruning of large language models.Thu, 17 Apr 2025 00:00:00 GMTenBlog postTalksLLM pruningKAUSThttps://ivanilin.org/blog/rising-stars-ai-symposium/https://ivanilin.org/blog/rising-stars-ai-symposium/I presented a poster on block-wise LLM pruning at the 2025 KAUST Rising Stars in AI Symposium.Thu, 10 Apr 2025 00:00:00 GMTenBlog postTalksLLM pruningKAUSThttps://ivanilin.org/publications/#hessian-of-perplexityhttps://ivanilin.org/publications/#hessian-of-perplexityPublication — Shows how to use PyTorch's automatic differentiation to compute selected pieces of an LLM's Hessian—the matrix describing local curvature—and estimate its full diagonal. The open-source implementation makes second-order analysis practical without storing the impossibly large full matrix.enPublicationTechnical reportLLM HessianPyTorch autogradSecond-order methodshttps://ivanilin.org/publications/#higgshttps://ivanilin.org/publications/#higgsPublication — Shows how errors from compressing each LLM layer increase perplexity, a measure of prediction quality. This result enables HIGGS, a quantization method needing no calibration data, and an optimal way to distribute bit widths across layers under a fixed budget.enPublicationConference paperQuantizationLLM compressionLinearity theoremhttps://ivanilin.org/projects/thanos/https://ivanilin.org/projects/thanos/Project — A block-wise algorithm for pruning large language models using second-order information and coordinated weight compensation.enProjectResearch / LLM PruningLLM pruningSecond-order optimizationStructured sparsityhttps://ivanilin.org/publications/#thanoshttps://ivanilin.org/publications/#thanosPublication — Prunes LLM weights in blocks, using second-order information about how weights interact to compensate for removals and adaptive masks to revise earlier choices. It supports flexible and hardware-friendly sparsity patterns, reducing model size while preserving accuracy.enPublicationPreprintLLM pruningSecond-order optimizationStructured sparsityhttps://ivanilin.org/blog/kaust-masters-thesis-and-sands/https://ivanilin.org/blog/kaust-masters-thesis-and-sands/I completed my master’s work on efficient LLM pruning and presented the Thanos project at the KAUST SANDS seminar.Tue, 26 Nov 2024 00:00:00 GMTenBlog postResearchLLM pruningKAUSThttps://ivanilin.org/publications/#pv-tuninghttps://ivanilin.org/publications/#pv-tuningPublication — Studies how to fine-tune LLMs compressed to only one or two bits per parameter. PV-Tuning moves beyond the commonly used straight-through estimator through a more general optimization framework, improving compressed Llama and Mistral models for memory-limited devices.enPublicationConference paperQuantizationLLM compressionOptimizationhttps://ivanilin.org/publications/#shadowheart-sgdhttps://ivanilin.org/publications/#shadowheart-sgdPublication — Designs an asynchronous training method for systems whose workers differ in both computation and communication speed. Shadowheart compresses updates and avoids waiting for slow workers, with proofs that no comparable centralized compressed method can achieve a better overall training time.enPublicationConference paperDistributed optimizationAsynchronous SGDHeterogeneityhttps://ivanilin.org/publications/#kimadhttps://ivanilin.org/publications/#kimadPublication — Monitors changing network bandwidth during distributed training and adjusts each worker's overall gradient-compression budget, then distributes that budget across neural-network layers to minimize compression error. This reduces communication time while retaining the convergence behavior of fixed-ratio compression.enPublicationWorkshop paperGradient compressionDistributed optimizationBandwidth awarenesshttps://ivanilin.org/projects/3d-engine/https://ivanilin.org/projects/3d-engine/Project — A custom CPU-rendering experiment that grew from a C++ course project into a lightweight game engine and a downloadable online FPS.enProjectEngineering / Computer GraphicsComputer graphicsGame engineC++Multiplayerhttps://ivanilin.org/projects/vectozavr-academy/https://ivanilin.org/projects/vectozavr-academy/Project — A visual mathematics school that became my first successful business—and then a long-running experiment in AI-assisted education.enProjectEducation / AIOnline educationAI-assisted learningCourse authoringProduct developmenthttps://ivanilin.org/projects/arduino-gameboy/https://ivanilin.org/projects/arduino-gameboy/Project — A pocket console with one joystick, a 128×64 OLED, a homemade PCB, and three games—including a tiny ray-cast 3D engine.Thu, 19 Nov 2020 00:00:00 GMTenProjectHardwareArduinoEmbedded systemsGame developmentPCB fabricationhttps://ivanilin.org/projects/pseudo-3d-shooter/https://ivanilin.org/projects/pseudo-3d-shooter/Project — A custom C++ ray-casting engine that turned a 2D map into a textured online FPS—with recursive mirrors, reflected gunfire, OBJ maps, and genetic-algorithm bots.Mon, 24 Feb 2020 00:00:00 GMTenProjectGame Engine / Computer GraphicsC++Ray castingGame developmentMultiplayerhttps://ivanilin.org/projects/four-bit-adder/https://ivanilin.org/projects/four-bit-adder/Project — A hand-soldered binary calculator built from discrete transistors—from Boolean logic and a faulty one-bit prototype to a working four-bit PCB.Wed, 14 Aug 2019 00:00:00 GMTenProjectHardwareDigital logicElectronicsPCB designTransistorshttps://ivanilin.org/projects/vectozavr-youtube/https://ivanilin.org/projects/vectozavr-youtube/Project — How a student science blog grew into a Russian-language channel built around one rule: make the project first, then explain what it taught me.enProjectScience communicationScience communicationProject-based learningMathematicsProgrammingElectronicshttps://ivanilin.org/projects/ilinblog/https://ivanilin.org/projects/ilinblog/Project — A hand-built 2017 PHP and MySQL science blog that became a workshop for physics, programming, electronics, and learning in public—now preserved as a static archive.enProjectEducation / Web DevelopmentWeb developmentScience communicationEducational publishinghttps://ivanilin.org/projects/voicecut/https://ivanilin.org/projects/voicecut/Project — An open-source, local-first tool that turns retake-heavy narration into a clean audio or video edit while preserving the speaker's real voice.enProjectOpen Source / Audio ProcessingAudio processingSpeech editingOpen source