Why bring data science to the GPU?

But how does a GPU actually help you accelerate your code?

A CPU and a GPU are built for different jobs

Source: NVIDIA CUDA C++ Programming Guide

A few specialists vs many simple workers

CPU

a few powerful cores

GPU

many simple cores

• CPU cores are few but powerful, great at complex, branchy logic
• GPU cores are many but simple, built to all do the same operation on different data, in parallel
• The more your work splits into independent pieces, the more using the GPU pays off

Two machines, two separate memories

• The GPU is a separate device with its own memory: it cannot read data sitting in system RAM
• Before the GPU can work on your data, that data must be copied into the GPU's memory

So step one of "run it on the GPU" is always: get the data there.

Parallel work: threads, blocks, grids

The GPU runs one operation across thousands of elements at once, one thread per item.

Source: NVIDIA CUDA C++ Programming Guide

• Threads are grouped into blocks; blocks together form a grid
• You describe the work once; the GPU runs it across the whole grid in parallel
• Perfect for independent work, not optimal for sequential, step-by-step logic

Why GPU is better for parallel than sequential

Sequential

one after another → time

Parallel

all at once → done sooner

• When work items are independent, their order doesn't matter i.e item B doesn't need item A's result
• A GPU runs a huge number of independent items at the same time
• Step-by-step work that must run in order can't use that; this isn't where a GPU shines

SMs and warps: where the work runs

• A GPU is made of Streaming Multiprocessors (SMs), its parallel engines
• Your blocks are distributed across the SMs; more SMs → more work at once, automatically
• Inside an SM, threads run in lock-step groups of 32, called a warp (SIMT: Single Instruction, Multiple Threads)

Source: NVIDIA CUDA C++ Programming Guide

Compute-bound vs IO-bound

Adding compute units only helps when computation is the bottleneck. If you're waiting on data, more cores don't make the wait shorter.

So what actually fits a GPU?

Takeaway: a GPU shines on big, math-heavy, parallel work, so the goal is to match your workload to its strengths.

A CUDA kernel is just C, with a twist

// __global__ marks a function that runs ON the GPU, once per thread
__global__ void add(float *a, float *b, float *c) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;  // which element am I?
    c[i] = a[i] + b[i];                             // one thread, one element
}

// The host launches it across a grid:  <<< blocks, threads-per-block >>>
add<<<blocks, threads>>>(a, b, c);   // every element added in parallel

The __global__ keyword and the <<< >>> launch syntax are the C extensions CUDA adds. A sum, a sort, a join all become kernels like this, but you'll rarely write one: the layers above ship them ready-made.

The CUDA Toolkit: the foundation libraries

Everything above the driver is built on the Toolkit. It provides:

These hand-tuned libraries are the ready-made kernels: the sorts, joins, and math behind your data work come straight from here.

CUDA Python: pick your entrypoint

You don't have to write C++ to use CUDA.

All three reach the GPU underneath.

CUDA-X & RAPIDS: kernels you don't have to write

RAPIDS packages thousands of optimized GPU kernels behind APIs you already know:

You stand on thousands of tuned kernels

• Each tile is a GPU kernel that experts wrote and tuned over years
• RAPIDS bundles them behind the APIs you already know
• You write familiar Python; RAPIDS runs the CUDA for you

How much faster on real data? We'll explore that in this tutorial

The mental model for today

• A GPU is many simple cores + its own fast memory → built for parallel, math-heavy work
• Getting data onto the GPU is the slow part → keep it there
• CUDA (C-based) exposes that power, but you rarely touch it
• CUDA Python and the CUDA Toolkit are the building blocks
• RAPIDS / CUDA-X hand you ready-made kernels, so familiar Python code just runs on the GPU