Rust at the Metal: The GPU Layer Driving Modern AI

In this issue, we explore how Rust is moving closer to the GPU layer, the foundation where modern AI performance is forged. From portable technologies like Vulkan and Metal to low-level CUDA acceleration, Rust reaches the hardware depths that power today's most demanding workloads.

This issue delves deep into the technologies that are making Rust's GPU integration possible, and how they are shaping the next generation of AI and machine learning.

Let's dive into the current issue.

GPUs have evolved from fixed-function graphics accelerators into massively parallel compute engines that now underpin much of modern AI and scientific computing. Unlike CPUs, which optimize for sequential logic and branch-heavy workloads, GPUs are designed to execute thousands of lightweight threads in parallel, making them ideal for matrix multiplication, convolution, and other high-throughput numerical operations. Major vendors such as NVIDIA, AMD, Intel, and Apple each maintain their own hardware ecosystems and software stacks. NVIDIA dominates AI workloads through its CUDA platform, while AMD’s ROCm and Apple’s Metal Performance Shaders (MPS) are closing the gap with increasingly mature driver and compiler support.

At the software level, frameworks like PyTorch and TensorFlow abstract this complexity by dispatching tensor operations to backend kernels implemented in CUDA, HIP, or MPS. These kernels are compiled into low-level GPU code and executed by vendor drivers on the hardware’s compute cores. Although GPUs share architectural roots with the graphics pipeline, AI frameworks do not rely on vertex or fragment shaders; instead, they use compute kernels that directly exploit the GPU’s general-purpose SIMD (Single Instruction, Multiple Data) capabilities. This decoupling from the graphics stack has allowed modern GPUs to transcend their gaming origins, becoming the de facto engines for large-scale AI and data-parallel computation.

In reality, most Rust developers working in AI and machine learning will not need to delve into the intricate details of GPU acceleration. The frameworks and libraries they use for training, inference, or data processing manage GPU acceleration behind the scenes. However, it is important to recognize that Rust is making progress in this area. A more robust and unified GPU foundation allows higher-level AI frameworks to be developed entirely in Rust, without the need for C++ bindings or vendor-specific glue code.

Rust’s emergence in GPU computing isn’t about replacing CUDA or rewriting established APIs. On one side, some projects aim to unify all API’s, such as wgpu, Rust’s implementation of the WebGPU standard, unifies access to Vulkan, Metal, DirectX, and WebAssembly. It gives developers a portable foundation for GPU compute that runs across platforms and architectures. Another project Rust-GPU aims to make Rust as a first-class language and ecosystem for GPU graphics & compute shaders

On the other hand, Cudarc and Rust-CUDA extend Rust’s reach directly into NVIDIA’s ecosystem, as well as ZLUDA which allow running unmodified CUDA applications using non-NVIDIA GPUs. There’s also Vulkano, a safe and rich wrapper for the Vulkan API.

Bridging these layers are new compute-language projects like CubeCL and Krnl, which allow developers to write GPU kernels directly in Rust syntax.

While it's still early in the process, the direction for Rust is becoming clear. The Rust ecosystem is developing in a way that goes beyond simply wrapping around existing GPU APIs. This represents a subtle yet significant shift, positioning Rust not just as a newcomer to GPU programming, but as a language poised to redefine how AI workloads interact with hardware.

Each week, we spotlight a project pushing Rust's AI frontier forward.

If Rust had a direct path to NVIDIA GPUs, it would be Rust CUDA.

Recently rebooted as part of the broader Rust-GPU initiative, Rust-CUDA is an ongoing effort to bring native CUDA kernel development into the Rust ecosystem, letting developers write, compile, and launch GPU kernels directly in Rust without relying on C++ bindings.

Its design allows Rust code to integrate more naturally with the CUDA toolchain while exploring how Rust's modern tooling can improve GPU programming workflows.

Although still in early development, with active refactoring, and incomplete features, the reboot marks an essential step toward unifying Rust's GPU ecosystem.

Its long-term vision is to align with the Rust-GPU project, potentially allowing the same Rust codebase to target multiple GPU architectures, from Vulkan to CUDA.

That vision was illustrated earlier this year in the blog post "Rust on Every GPU", which demonstrated a single Rust codebase running seamlessly across CUDA, SPIR-V, Metal, and DirectX backends. The accompanying demo project, rust-gpu-chimera, showcased this cross-GPU capability in action, a glimpse of a future where developers write GPU code once in Rust and deploy it anywhere.

🔗 Explore ➡ Rust-CUDA  

Subscribe to receive the latest insights, and updates delivered directly to your inbox and if you know someone who’d enjoy this, feel free to forward it their way!

If you have any feedback or ideas you’d like to see in future issues, we would love to hear from you, drop us a line at [email protected]