How Together AI Uses NVIDIA's Full Stack to Deliver AI Responses in Under 100ms

Together AI's Dan Fu, Vice President of Kernels, explains how Together AI leverages NVIDIA GPUs to deliver AI responses in under 100 milliseconds while achieving some of the industry's lowest token costs. From developing a megakernel that fits an entire model into a single CUDA kernel to powering real-time voice agents that return their first 64 words in under 100 milliseconds, Together AI is focused on minimizing latency at every layer of the AI stack.

Learn how Together AI uses NVIDIA CUDA, TensorRT-LLM, Together ATLAS, and NVIDIA Blackwell to power real-time inference at scale. Fu also shares how Together AI helped Cursor accelerate the path from model optimization to production-ready inference endpoints using NVIDIA TensorRT-LLM on Blackwell, enabling a fast, responsive coding experience.

He also highlights ATLAS (AdapTive-LeArning Speculator System), which dynamically adapts models to changing traffic and workload patterns, and explains why NVIDIA's software stack—including CUDA, CUTLASS, Dynamo, and TensorRT-LLM—is foundational to Together AI's infrastructure. Looking ahead, Fu discusses the team's excitement for NVIDIA Vera Rubin as the platform for the next generation of long-context, ultra-low-latency AI applications.

Read more in the blog