A Field Guide to the NVIDIA Stack
Understanding the hardware and software stack that turns GPUs into usable AI infrastructure.
Premise
This note is different from my previous ones, and the premise is quite simple. I struggled to find balanced and straightforward material that clearly explains (A) the products Nvidia develops, (B) how these products are organized, and (C) the storyline behind their existence. So, I decided to explore these questions on my own. I later thought it would be beneficial to organize my findings and share them in accessible language, as they might help others better understand Nvidia’s ecosystem.
In this note, I connect the dots and share fundamental information that will enhance many people’s understanding of Nvidia’s market position today. It’s important to note that Nvidia is not just about chips; it’s about the entire stack, and the connections between its various layers provide insight into the recent years of AI infrastructure. This note focus on the last 5 to 6 years of the company and a glimpse of what’s ahead in light of the AI boom.
Intro
Ask someone outside the tech industry what NVIDIA makes, and you will usually hear one of two answers: graphics cards or “the AI chip company”. Ask someone who follows markets, and you may hear back “GPUs for AI datacenters” and get a stock chart instead of a proper product explanation. Both answers work at the dinner table. Neither explains the company very well.
NVIDIA started in 1993 by building hardware for video game graphics. Its technology now sits underneath many of the large language models, drug discovery pipelines, and autonomous vehicle prototypes announced this year. A faster chip is only part of how that happened.
NVIDIA sells a stack: hardware and software layers released on a schedule the company controls. Each layer handles work that the one above it would otherwise need to solve again. The chip matters, but a chip alone trains nothing. NVIDIA also ships the systems that hold the silicon, the network that joins those systems, the programming layer that software uses to control them, and platforms for turning trained models into deployable products.
Without those layers, NVIDIA looks like a hardware company that caught the right moment. With them, it looks more like a hardware vendor combined with an operating system for a new kind of computing.
This note follows the stack one layer at a time and all the content is sourced only via NVIDIA’s technical documentation. Given its importance for the ecosystem I dedicate more time on NeMo, the platform for adapting a general-purpose model, evaluating it, adding safeguards, and preparing it for real users.
Layer 1: accelerated computing
A CPU handles many different kinds of work, usually as a sequence of instructions in whatever order a program requires. A GPU was originally designed for the matrix and vector arithmetic used to render a 3D scene across millions of pixels at once.
Neural networks rely on a similar operation: multiplying and summing large matrices of numbers. That work can be divided into many calculations that run in parallel. GPUs were already good at the arithmetic AI needed, which is why they became the default hardware for training and running models.
NVIDIA releases a new GPU architecture about every two years: Ampere in 2020, Hopper in 2022, and Blackwell in 2024. Rubin, the successor to Blackwell, was previewed for 2026. The changes are not limited to raw speed. NVIDIA redesigns each generation around the workloads developers are running.
Blackwell provides a documented example. NVIDIA says the GPU uses two dies connected by a custom interconnect and contains about 208 billion transistors, roughly 2.6 times the number in Hopper. Programmers can still address it as one accelerator. Blackwell also introduced lower-precision number formats, including NVFP4, which exchange some numerical precision for higher speed and better memory efficiency.
That trade can work for AI because a neural network is a statistical system. It usually does not need every individual number to be exact. It needs the overall computation to converge on a useful result. A spreadsheet or a bank ledger would be far less tolerant of the same approximation.
Layer 2: host compute platforms
A fast GPU is useful only when data reaches it quickly enough to keep it busy. That becomes a bottleneck as models grow.
NVIDIA responded by building Grace, its own Arm-based CPU, instead of relying entirely on standard server processors. Grace is designed around the memory bandwidth and coherency requirements of GPU-heavy workloads. It is not a general enterprise processor later adapted for AI.
Grace connects to a Hopper or Blackwell GPU through NVLink-C2C, a dedicated chip-to-chip interconnect. NVIDIA calls the resulting combinations “superchips”, including Grace Hopper and Grace Blackwell, with a successor generation previewed - Vera CPU, which will become available soon.
The tight connection lets the CPU and GPU share a pool of memory instead of repeatedly moving data over a general-purpose bus. Once a model no longer fits comfortably in one GPU’s memory, that bus can limit performance more than the GPU’s compute speed.
Layer 3: AI platform
NVIDIA sells complete machines as well as chips. A DGX platform / system combines eight GPUs with networking, storage, and a tuned software stack in one integrated unit. NVIDIA’s product material calls it a “turnkey AI supercomputer.”
DGX SuperPOD extends the same approach to a cluster. It is a reference architecture for connecting many DGX systems across a data center. NVIDIA uses SuperPOD for its own AI and high performance computing research. Its documentation presents the product as a complete system, including the management software required to operate it at scale, rather than a collection of hardware that customers must integrate themselves.
Jetson applies the same hardware and software foundation at the other end of the size range. These small, low-power modules run CUDA-based AI inside robots, drones, and cameras instead of data center racks. Within the platform’s limits, code tested on a data center GPU belongs to the same software family as code deployed to a physical device at the edge.
Layer 4: networking
Ten thousand GPUs are not automatically ten thousand times faster than one. Training a large model requires the GPUs to synchronize partial results constantly, so the slowest connection can constrain the entire cluster.
NVIDIA uses different networking technologies at different physical scales. NVLink connects nearby GPUs at high bandwidth, allowing dozens of GPUs in a rack to behave in many respects like parts of one larger unit. Quantum InfiniBand and Spectrum-X Ethernet connect racks across a data center.
BlueField data processing units move networking, storage, and security work away from the GPU. The GPU can then spend more of its cycles on computation. NVIDIA describes the Blackwell-based GB200 NVL72 in these terms: 36 Grace CPUs and 72 Blackwell GPUs connected by NVLink and NVSwitch as one large addressable unit, rather than 72 separate computers sharing a rack.
Layer 5: programming foundation
This layer is easy to miss because it is the least visible layer of the stack, but arguably the most important. Parallel arithmetic on a chip is not useful without a practical way to program it. CUDA, released in 2007, provides that programming environment.
NVIDIA’s documentation describes CUDA as a way for C and C++ developers to build, optimize, and deploy GPU-accelerated applications. The same environment covers embedded devices, desktop workstations, cloud platforms, and supercomputers. It also supports computations distributed across several GPUs.
The release date matters. CUDA had more than a decade to develop before transformer-based models made GPUs central to AI research. When demand arrived, developers already had tools, libraries, examples, and experience to build on.
That head start may be harder to reproduce than any single chip generation. With enough capital and manufacturing access, a competitor can develop comparable hardware. Rebuilding years of tooling and institutional knowledge takes longer.
CUDA-X adds libraries for work that developers would otherwise need to implement with low-level GPU code. cuDNN handles core neural network operations. TensorRT optimizes trained models for inference. RAPIDS provides GPU-accelerated data analytics. These libraries are what make a fast chip usable by an ordinary engineering team.
Layer 6: microservices platform
The layers so far explain how teams train large models. They explain much less about what happens next: adapting a general-purpose model to a team’s data, measuring it against that team’s standards, protecting it, and running it in front of users.
NeMo addresses that part of the process. It also shows how NVIDIA’s software offering has moved beyond libraries that accelerate training and toward an application platform.
According to NVIDIA’s documentation, NeMo supports the development and deployment of specialized AI agents on open-source models. Its services cover synthetic data generation, fine-tuning, evaluation, security testing, and safety controls during inference. Teams reach those services through a common set of APIs, with access control and observability included for production use.
NeMo can run locally in Docker for experiments or on Kubernetes for production. It organizes resources around workspaces, projects, and entities. A workspace creates an authorization boundary between teams, clients, or environments. A project groups related work inside that workspace, such as a fine-tuning run or an evaluation campaign. Entities are the shared models, datasets, jobs, and configurations that NeMo services use.
Six base microservices divide the work (note that recent NVIDIA's Nemo Platform1 continues to expand the services list):
Data Designer generates synthetic training and evaluation data. Teams can control variation through seeding when labeled data is scarce, sensitive, or missing a case they need to test.
Customizer runs fine-tuning from a base checkpoint, a formatted dataset, and a selected job type. It supports LoRA, full supervised fine-tuning, and direct preference optimization, and records the training as a trackable job.
Evaluator scores model output with LLM-as-judge methods and metrics for agentic behavior and retrieval-augmented generation, in addition to generic benchmarks.
Guardrails applies content safety checks, topic restrictions, and prompt injection detection while a model handles a live conversation.
Safe Synthesizer creates synthetic data with privacy controls, including replacement of personally identifiable information and support for differential privacy.
Auditor tests a deployed model or agent for vulnerabilities with configurable probes. It handles security testing, while Guardrails protects live inference.
NVIDIA’s documentation (as of July, 2026) lists support in Customizer for base models from the Llama, Llama Nemotron, Phi, Qwen, and Mistral families, along with embedding models and GPT-OSS models. Larger checkpoints correspond to specific GPU memory tiers. NVIDIA lists A100 80GB, H100, and B200 for models requiring 80GB of GPU memory. That detail ties the platform back to the rest of the stack. NeMo’s software requirements are expressed in terms of NVIDIA hardware.
When a base or fine-tuned model is ready to serve requests, NIM packages it as a production inference endpoint. Dynamo, their framework for serving generative AI models in distributed environments, can distribute larger volumes of inference traffic across a GPU fleet.
NeMo does not compete with CUDA or replace the GPUs below it. It builds on both. The platform takes a base model and a GPU cluster and gives a team a path toward a customized, evaluated model with safeguards that it can serve to users. For most organizations, that is a more immediate problem than training a model from scratch.
Beyond text: simulation and robotics
The stack also supports simulation and robotics. The arrangement remains familiar: hardware, a programming layer, and a platform for a particular workload.
Omniverse is NVIDIA’s real-time 3D simulation platform. It uses Pixar’s OpenUSD format to build digital twins of factories, products, and cities. Isaac provides robotics software that works with Jetson hardware, allowing a policy trained in simulation to run on a physical machine.
Cosmos is NVIDIA’s newer (initially launched in January 2025) world foundation model platform for generating and predicting physical, video-based scenarios. In practice, it can produce training data before a robot interacts with the physical world. This part of NVIDIA’s business is newer and smaller than its language model serving work, but it uses the same stack.
In closing, why does the stack matter?
These products are designed to work together. CUDA makes a Blackwell GPU programmable. Libraries and platforms such as NeMo turn CUDA into something an engineering team can use for a specific job. DGX turns chips into a computer. DGX SuperPOD turns computers into a cluster.
NVIDIA coordinates the release schedule across these layers. That gives the company a different competitive position from a vendor trying to sell the fastest chip in a given year. NVIDIA also influences the standard against which other parts of the industry build.
A coordinated platform makes AI infrastructure easier to use. It also places a large share of that infrastructure behind one company’s roadmap.
The first effect is practical. A small team can now fine-tune, evaluate, and deploy a specialized model with less infrastructure expertise than a comparable project required five years ago. Platforms such as NeMo absorb work that once needed a dedicated infrastructure team.
The same coordination also routes much of the world’s AI infrastructure through NVIDIA’s release cadence and its definitions of “customized”, “evaluated”, and “safe.”
The stack does not tell us whether that concentration is temporary, while credible alternatives2 mature3, or a lasting feature of large-scale AI infrastructure. As with everything in the world of technology, people building on it still need to ask the question, assess the alternatives, and weigh the pros and cons that will help drive the decisions for the challenges or ideas at play.
NVIDIA NeMo microservices and the NeMo platform represent the individual modular building blocks versus the complete, integrated enterprise deployment of those components. The microservices are individual REST API services (like NeMo Customizer or Evaluator), whereas the platform is the collective assembly of these services working together on a Kubernetes cluster.
E.g., AMD, Intel, Huawei, Qualcomm, Custom Silicon (Google, AWS, Meta, Microsoft, Broadcom, Marvell, Micron), AI startups specialized chipmakers (Etched, Cerebras, Tenstorrent).
Check Artificial Analysis Benchmarks for (a) AI Hardware - AA-SLT - Simple throughput, output speed, TTFT on synthetic inference workloads and (b) Hardware for the Agent Era - AA-AgentPerf - Agents per MW and other metrics via real coding agent inference workloads.
If you found this useful, please cite this guide as:
Müller, Lucas. (Aug 2026). A Field Guide to the NVIDIA Stack. lucasmuller.com. https://notes.lucasmuller.com/p/a-field-guide-to-the-nvidia-stack
or
@article{lucasmuller2026default,
title = {A Field Guide to the NVIDIA Stack},
author = {Müller, Lucas},
journal = {lucasmuller.com},
year = {2026},
month = {Jul},
url = {https://notes.lucasmuller.com/p/a-field-guide-to-the-nvidia-stack}
}