Supercharging LLMs: Scalable RL with torchforge and Weaver
Scaling PyTorch-native RL with torchforge on 512 GB200, collaboration with PyTorch and CoreWeave
Here is a collection of text I enjoyed and inspire me.
I write about ML + Systems, philosophy of religion, ideas in classical text, cold-war history.
Note: Some blogs are still being migrated.
Scaling PyTorch-native RL with torchforge on 512 GB200, collaboration with PyTorch and CoreWeave
A 1-year retrospective on KernelBench / progress towards automated GPU Kernel Generations
Amdahl’s Law, Strong Scaling, Asynchronous Execution, Blackwell, Hopper, Ampere, Turing, Volta, TMA | with SemiAnalysis
How do state-of-the-art LLMs like Gemini 1.5 and Claude 3 scale to long context windows beyond 1M tokens? Well, Ring Attention presents a way to split attention calculation across GPUs while hiding the communication overhead in a ring,enabling zero overhead scaling.
Blog I helped write at Anyscale. The FAX system at Cohere enables distributed training of Large Language Model using JAX's pjit tensor parallelism, TPUv4s' pods of accelerators with high-speed interconnect, and Ray's powerful yet simple worker orchestration.
Blog I helped write at Anyscale. This explains how ByteDances uses Ray Dataset and Train to build their NLP pipeline for products such as TikTok.
In Hegel’s “The Tübingen Essay”, Hegel proposes the importance of folk religion, which itself is constituted “with respect to ceremonies”. However, this seems to deviate from German Protestant (particularly Luther’s) thoughts’ de-emphasis on rituals, which is often deemed as too Jewish or Catholic. Thus, does Hegel deviate from Luther’s “sola fide” (by faith alone)?
In Prefaces to the Old Testament, Luther explains the causes for the extraneous laws in the Old Testament, showing the impossibility to obey laws leads to the yearning for God’s Grace. Thus, for Christians, faith matters more than following laws. However, Luther in Temporal Authority points out that “True Christians”, although do not need laws, should follow the extensive set of temporal laws. There seems to be a contradiction: if only “faith and love” matters, why shall Christians still follow temporal laws?
Serve in heaven or reign in hell? Which is it to be? Roots of Romantic Satanism in Ridley Scott’s Prometheus (2012) and its sequel Alien: Covenant (2017).
How does love appear as a potent and potentially dangerous emotion and force in Virgil's Aeneid and Lucretius' The Way Things Are?
What I saw at Consumer Electronics Show 2018
My Experience at TechCrunch Shenzhen 2017 Hackathon, Conference, and Innovation Tour
“Our policy is directed not against any country or doctrine but against hunger, poverty, desperation, and chaos.”, said General George C. Marshall.. but is it really just that?
Mach 2, 50,000 Feet.. Why did the superior Canadian homegrown fighter jet got cancelled?