Experience
Research Scientist Intern
I worked on machine learning systems for large language model training, from distributed reinforcement learning infrastructure to efficient multi-LoRA training and GPU kernels.
Optimized Raskel, an internal multi-LoRA training framework, by tuning batching and parallelization strategies to improve model FLOPs utilization by 5%. Optimized its Triton multi-LoRA GEMM kernel to increase H20 throughput by up to 20% across different workloads.
Added LoRA support for vision-language model training in VeOmni, along with model FLOPs utilization calculations for performance analysis and optimization.
Built an open-source Tinker-compatible API server on verl, enabling fine-tuning with VeOmni/FSDP actors, vLLM/VeXact rollout, reference engines, and dynamic model offloading. The system supported reinforcement learning on 120B models across 64 GPUs.
Implemented importance-sampling and distributionally robust optimization losses in verl. Added streaming synchronization for GPT-OSS mixture-of-experts weights, handling gate and up projections interleaved across experts and reducing peak memory for the affected block to 1–3% of the original.
Investigated how gradient-matrix singular values correlate with downstream loss curves across LoRA rank, learning rate, alpha, and matrix initialization. Used these relationships to predict stable, near-optimal configurations under practical search constraints, reducing hyperparameter-search time by 80% compared with exhaustive sweeps.