Use the GPUs on Hydra

Use the GPUs on Hydra#

Colette Kelly · August 12, 2026 · how-to, computing, gpu

This resource covers what a GPU actually is, when and why you should use one, how to access a GPU node on Hydra, and how to build a conda environment with nvForest so you can run a trained scikit-learn random forest on the GPU.

If you are new to high-performing computing or have never used the Poseidon or Hydra computer clusters before, I recommend checking out my other tutorials on high-performance computing:

🔗 HPC basics

🔗 Getting started with Hydra

Here are the third party references I used when figuring this out:

🔗 nvForest docs

🔗 RAPIDS install guide

🔗 RAPIDS platform support

Overview#

Why use a GPU?#

I have several Random Forest models that predict nitrous oxide (N₂O) from things like temperature, salinity, oxygen, and nitrate. I tried using these models to predict N₂O on a large feature dataset, using traditional CPU resources on Hydra. The result: disaster. Well not quite… but even with the backend parallel processing built into Scikit-Learn’s Random Forest Inference Regressor running on 16 parallel cores, it took four minutes for one model (out of four) to do the prediction on a single month of data (out of a 384-month dataset). Why? Each random forest model has 600 decision trees, and each month of data includes 20 million rows. Here’s that example:

import numpy as np
import pandas as pd
import xarray as xr
import gsw
from joblib import load

TPATH = "/proj/boom/data/argo_rfromv22_temp"
SPATH = "/proj/boom/data/argo_rfromv22_sal"
O2PATH = "/proj/boom/data/gobai-highres-o2"
NO3PATH = "/proj/boom/data/gobai-highres-no3"

FEATURE_DIR = "/proj/boom/data/highres_features"
PRED_DIR = "/proj/boom/data/highres_n2o_pred/etphighres"
MODEL_DIR = "/user/colette.kelly/ml-argo-n2o"

# lat-lon slices for the eastern tropical Pacific
LAT = slice(-30,25)
LON = slice(360 - 180, 360 - 70)

# pressure window for prediction - set to something small for testing
PMIN, PMAX = 0.0, 5000.0

# model features
# needs to match the feature order each RF was trained on
MODEL_FEATURES = {
    1: ["AOU", "CT", "SA", "no3"],
    2: ["o2", "CT", "SA", "no3"],
    3: ["AOU", "CT", "SA"],
    4: ["o2", "CT", "SA"]}

F32 = np.float32

year = 2024
month = 1

tab = pd.read_parquet(f"{FEATURE_DIR}/features_{year}_{month:02d}.parquet")
sub = tab[(tab["pres"] > PMIN) & (tab["pres"] < PMAX)]

model_id = 1
cols = MODEL_FEATURES[model_id]

# set up Numpy array to do predictions on
# use "sub" -> so the N2O predictions will only be made for the selected pressure range
X = np.ascontiguousarray(sub[cols].to_numpy(dtype=F32))
X.shape

# load RF models
models = {mid: load(f"/user/colette.kelly/ml-argo-n2o/model{mid}_rf_full.joblib") for mid in (1,2,3,4)}

model = models[model_id]
model.n_jobs = -1
print(f"[backend] model {model_id}: sklearn ({type(model).__name__})") # expect RandomForestRegressor

%timeit y = np.asarray(model.predict(X), dtype=F32)

6min 15s ± 2.6 s per loop (mean ± std. dev. of 7 runs, 1 loop each)

What is a GPU?#

So, time to turn to a GPU. WHOI’s new computer cluster Hydra has some amazing resources, including a GPU node. Here’s that actually means:

The traditional compute nodes on Hydra have 48 powerful CPUs (central processing units) per node that are great at sequential logic: “do this, then check that, then decide what’s next.”” The GPU node on Hydra has 4 NVDIA GPUs (graphics processing units), each of which has thousands of small, simple cores that are great at doing the same operation across a huge amount of data at once.

A commonly-used analogy: a CPU is a kitchen full of expert chefs each cooking a multi-course meal from start to finish. A GPU is a stadium-full (actually though) of line cooks all chopping onions at the same time. So is your problem, cook this fancy multi-course meal? Or chop 10 million onions the same way? If you’re doing something highly parallelizable, the GPU is the correct path.

The one catch: a GPU has its own memory, separate from the computer’s main (host) memory. Your data has to be copied from host memory into GPU (device) memory and the results copied back. So you should avoid shuttling data across that boundary, and only use a GPU if the computational savings from parallelizing on a GPU are greater than the cost of copying data back and forth.

When (and when not) to use a GPU#

Good fits for a GPU:

  • Training or running machine-learning models on lots of data

  • Large array / linear-algebra math (big matrix ops, FFTs)

  • DataFrame operations on large tables via RAPIDS (cuDF / CuPy)

  • Anything “embarrassingly parallel” — the same simple operation over millions of elements

My use case: applying trained random forest models to ~20 million rows of gridded ocean data to predict N₂O. On a CPU this was painfully slow and kept running out of memory; on the GPU the same inference finishes in seconds.

Skip the GPU for:

  • Small data, because the host↔device copy dominates and you gain nothing

  • Serial, branchy logic that can’t be parallelized

  • I/O-bound work: reading/writing files, downloading data, conda installs

  • Plain pandas on a modest table

A GPU can actually be slower once you count the transfer overhead.

GPU etiquette#

GPU nodes are a scarce, shared resource. Only request one for work that actually uses the GPU. Do your feature engineering, file wrangling, and environment installs on a regular compute node and save the gpu partition for the inference itself.

Start an interactive session on a GPU node#

Similar to starting an interactive session on a traditional compute node, but ask SLURM for the gpu partition and one GPU with --gpus-per-node:

[colette.kelly@hydra-l2 ~]$ srun --partition=gpu --gpus-per-node=1 --mem=32GB --time=00:30:00 --pty bash

Your prompt updates to the GPU node you were assigned:

[colette.kelly@gpu003 ~]$

A note on --mem: that’s host RAM, which is separate from the GPU’s own memory. Loading a single large random forest into host memory can take ~14 GB on its own, so make sure you’ve requested enough memory or the session will time out. You don’t need to size it to the GPU’s memory.

Check the GPU you got#

It’s worth checking the GPU hardware to make sure you’re installing versions of libraries that are compatible with the hardware. This is a bit more involved on GPU than with a traditional compute node. Here’s what you need to do.

Before installing anything, see what hardware you landed on:

[colette.kelly@gpu003 ~]$ nvidia-smi --query-gpu=name,compute_cap,driver_version --format=csv
name, compute_cap, driver_version
NVIDIA H200, 9.0, 610.43.02

  • name → NVIDIA H200. A current-generation (Hopper) datacenter GPU with ~141 GB of on-board memory.

  • compute_cap → 9.0. The GPU’s compute capability — its architecture / feature level. The RAPIDS ecosystem, which includes the nvforest library for RF inference, requires 7.0 or higher (Volta and newer).

  • driver_version → 610.43.02. The NVIDIA driver installed on the node. The driver determines the newest CUDA version the node can run. RAPIDS 26.06 needs driver 525.60.13+ for its CUDA 12 build or 580.65.06+ for CUDA 13. RAPIDS ships separate builds for CUDA 12 and CUDA 13, and you have to install one your driver supports. We have 610 so we can use either.

Set up a conda environment with nvForest#

nvForest is the RAPIDS library for fast decision-tree and forest inference on the GPU (it’s the successor to the now-deprecated cuml.fil). It imports an already-trained scikit-learn / XGBoost / LightGBM forest and runs its predictions on the GPU, so you don’t retrain anything.

Step 1 — see what’s actually available#

Install everything on a regular compute node, not a GPU.

[colette.kelly@hydra-l1 ~]$ srun -N 1 -n 1 --mem=50GB --time=01:00:00 --partition=compute --pty bash
[colette.kelly@cn059 ~]$ module load miniconda/25.9
[colette.kelly@cn059 ~]$ . $CONDA_PREFIX/etc/profile.d/conda.sh
[colette.kelly@cn059 ~]$ # See what channels conda already has configured — look for `defaults`
conda config --show channels
channels:
  - defaults
[colette.kelly@cn059 ~]$ conda search -c rapidsai -c conda-forge nvforest
2 channel Terms of Service accepted
Loading channels: done
# Name                       Version           Build  Channel             
nvforest                    26.04.00 cuda12_cp311_abi3_260408_e77480fb  rapidsai            
nvforest                    26.04.00 cuda13_cp311_abi3_260408_e77480fb  rapidsai            
nvforest                    26.06.00 cuda12_cp311_abi3_260604_a9216c61  rapidsai            
nvforest                    26.06.00 cuda13_cp311_abi3_260604_a9216c61  rapidsai

  • nvforest is on the rapidsai channel, with 26.06.00 available

  • cuda12 = the CUDA 12 build; cuda13 = the CUDA 13 build

  • cp311 = built against CPython 3.11.

  • abi3 is the key part — these are built against Python’s stable ABI, so a single build is forward-compatible across later 3.x. The giveaway is that there’s only one build per (version, CUDA) instead of separate cp311/cp312/cp313 rows. That means the package declares roughly python >=3.11, and 26.06 supports Python 3.11 through 3.14, so python=3.12 resolves to that same cuda12_cp311_abi3 build. You’re not locked to 3.11.

Step 2 — create the environment#

[colette.kelly@cn059 ~]$ CONDA_OVERRIDE_CUDA=12.9 conda create -n nvforest \
  --override-channels -c rapidsai -c conda-forge \
  --solver=libmamba \
  nvforest=26.06.* python=3.12 'cuda-version>=12.2,<=12.9'

  • The CONDA_OVERRIDE_CUDA=12.9 matters because we’re solving on cn059, a regular compute node with no GPU driver visible — without it, conda can’t detect a CUDA driver and the solve gets confused.

  • --override-channels is needed because Hydra’s base conda has only the defaults (Anaconda) channel configured and RAPIDS packages are not built for defaults.

  • -c rapidsai -c conda-forge -c nodefaults are the channels that RAPIDS needs.

  • --solver=libmamba is a fast solver that handles big RAPIDS environments better than the classic solver and gives clearer errors.

  • Pin nvforest=26.06.00 for reproducibility, so you get the build you just inspected rather than whatever’s latest later.

  • python=3.12 — if python=3.12 ever throws a solver complaint, drop to python=3.11 to match the build label.

  • The ‘cuda-version>=12.2,<=12.9’ pin selects the cuda12 build (the cuda12_cp311_abi3 row) over the cuda13 one. Hydra’s H200 + driver 610 would run either, but cuda12 keeps the surrounding conda-forge ecosystem on the better-trodden path.

Step 3 - add other packages#

[colette.kelly@cn059 ~]$ conda activate nvforest 
(nvforest) [colette.kelly@cn059 ~]$ conda install xarray pandas pyarrow scikit-learn netcdf4 matplotlib dask

Add cuDF and CuPy (optional: data on the GPU)#

If you want to read your feature files straight into GPU memory instead of going through pandas on the host, add cuDF. CuPy almost certainly came in already as an nvForest dependency, so check first:

(nvforest) [colette.kelly@cn059 ~]$ conda list | grep -E 'cudf|cupy'
cupy                             14.1.1   ...   conda-forge
cupy-core                        14.1.1   ...   conda-forge

If only cupy shows up, add cudf:

(nvforest) [colette.kelly@cn058 ~]$ CONDA_OVERRIDE_CUDA=12.9 conda install -n nvforest \
  --override-channels -c rapidsai -c conda-forge \
  --solver=libmamba \
  'cudf=26.06.*'

Warnings:

  • Don’t pip install cupy or cudf into a conda RAPIDS environment. Mixing Pip’s CUDA wheels and conda’s CUDA packages breaks things in confusing ways.

  • cuDF pins pandas to a supported range (e.g. pandas < 2.4), so installing it may downgrade pandas. Ordinary feature-engineering code (building DataFrames, to_parquet) uses only stable pandas API and is unaffected.

Verify#

Activate the environment and confirm everything imports together:

(nvforest) [colette.kelly@cn059 ~]$ python -c "import nvforest, cudf, cupy, sklearn; \
print('nvforest', '+ cudf', cudf.__version__, '+ cupy', cupy.__version__, '— all import OK')"

A tiny example: scikit-learn → GPUForestInferenceRegressor#

Exit the interactive session on the compute node:

(nvforest) [colette.kelly@cn059 ~]$ exit
exit

Now I start an interactive session on the GPU node and load an RF model:

[colette.kelly@hydra-l2 ~]$ srun --partition=gpu --gpus-per-node=1 --time=00:30:00 --mem=100GB --pty bash
[colette.kelly@gpu003 ~]$ module load miniconda/25.9
[colette.kelly@gpu003 ~]$ . $CONDA_PREFIX/etc/profile.d/conda.sh
[colette.kelly@gpu003 ~]$ conda activate nvforest
(nvforest) [colette.kelly@gpu003 ~]$ ipython

import joblib, nvforest

skl = joblib.load("/user/colette.kelly/ml-argo-n2o/model1_rf_full.joblib")

fm = nvforest.load_from_sklearn(skl, device = "gpu")

print(type(fm).__name__)
GPUForestInferenceRegressor

import numpy as np

X = np.random.rand(5, skl.n_features_in_).astype("float32")
print(fm.predict(X))
[[328.24543802]
 [328.62658223]
 [333.47100661]
 [328.17146862]
 [328.19903352]]

the CPU/GPU boundary#

The tiny example handed nvForest a NumPy array sitting in host memory. That works, but it copies the data across the host→device boundary on every call. But we want the data to land in GPU memory in the first place. That’s what cuDF is for. It’s pandas’ API on the GPU, so the code barely changes:

import cudf, cupy as cp

cols = ["AOU", "CT", "SA", "no3"]        # the features this forest was trained on
FEATURE_DIR = "/proj/boom/data/highres_features"
year = 2024
month = 1
path = f"{FEATURE_DIR}/features_{year}_{month:02d}.parquet"
# read the parquet straight into GPU memory, only the columns we need
gdf = cudf.read_parquet(path, columns=cols + ["pres", "flat_index"])
gdf = gdf[(gdf["pres"] > 0) & (gdf["pres"] < 5000)]     # filter runs on the GPU

# feature matrix: contiguous float32, still on the GPU
Xg = cp.ascontiguousarray(gdf[cols].astype("float32").to_cupy())

 %timeit yg = fm.predict(Xg) # inference on the GPU; yg is a CuPy array
 344 ms ± 118 μs per loop (mean ± std. dev. of 7 runs, 1 loop each)

⚠️ nvForest always returns a device array that lives on the GPU. You need to copy it back explicitly before you can write it:

yg = fm.predict(Xg)
y = cp.asnumpy(yg).astype(np.float32).reshape(-1)

cp.asnumpy is also a synchronization point between where GPU does the math and CPU writes the file. For a fast pipeline: only copy back the small things you’ll actually write. Here, we do inference on 20 million rows on the GPU but only bring home the 1-D prediction vector.

Finishing up:

flat = gdf["flat_index"].to_numpy() # move the cudf series to host numpy

TPATH = "/proj/boom/data/argo_rfromv22_temp"

import numpy as np, xarray as xr

# scatter + NetCDF write are host-side, since netCDF is a CPU library
LAT = slice(-30,25)
LON = slice(360 - 180, 360 - 70)
src = xr.open_dataset(
    f"{TPATH}/RFROMV22_TEMP_STABLE_{year}_{month:02d}.nc"
).sel(latitude = LAT, longitude = LON)["ocean_temperature"]

out = np.full(src.size, np.nan, dtype = np.float32)
out[flat] = y

da = xr.DataArray(out.reshape(src.shape), dims = src.dims, coords = src.coords, name = f"pN2O_model1")
out = f"n2o_model1_{year}_{month:02d}.nc"
da.to_netcdf(out)
print(f"[predict {out} predicted={len(y):,}")

Scale to all four GPUs#

One month of inference takes seconds. I have 384 of them (monthly, 1993–2024), and Hydra’s GPU node has four H200s. I could have either set up a SLURM job array or submitted one job that fanned tasks across the four GPUs. Before decided what to do, I checked the partition:

[colette.kelly@hydra-l1 ~]$ scontrol show partition gpu | grep -i oversubscribe
   ... OverSubscribe=NO

OverSubscribe=NO means that SLURM will never hand two jobs to the same core. The GPU node is still shareable, each job getting its own slice of cores, memory, and GPUs.

🔬 Sydney gave me a way to check this. I opened one interactive GPU session, then requested a second from another terminal, which was assigned resources on the same node immediately, while the first was still running.

So a job array (--gpus-per-node=1 per task, throttled with --array=0-383%4) would run four single-GPU tasks side by side here and work fine. I still chose a single job that used all four GPUs and fanned the work out over one worker per GPU. The advantage of this is that a worker that holds a GPU for the whole run loads the four forests only once and reuses them, instead of reloading and re-converting them on every one-month task.

CUDA_VISIBLE_DEVICES#

Before submitting a job, you can ask CUDA which of the GPUs it can reach:

python -c "import cupy; print(cupy.cuda.runtime.getDeviceCount())"

SLURM BATCH SCRIPT#

Here’s a simplified version of the SLURM batch script I used to set up this job:

#SBATCH --partition=gpu
#SBATCH --nodes=1
#SBATCH --gpus-per-node=4
#SBATCH --cpus-per-task=96          # the whole node — see the thread note below
#SBATCH --time=01:00:00

NWORKERS=4
THREADS=$(( SLURM_CPUS_PER_TASK / NWORKERS ))   # 96 / 4 = 24 cores each

pids=()
for w in 0 1 2 3; do
    CUDA_VISIBLE_DEVICES=${w} \
    OMP_NUM_THREADS=${THREADS} OPENBLAS_NUM_THREADS=${THREADS} MKL_NUM_THREADS=${THREADS} \
    python3 run_worker.py --worker ${w} --nworkers ${NWORKERS} \
        > worker_${SLURM_JOB_ID}_gpu${w}.log 2>&1 &
    pids+=($!)
done

fail=0
for pid in "${pids[@]}"; do wait "${pid}" || fail=1; done   # reap all, remember failures
exit ${fail}

Each worker takes a round-robin slice of the months (months[worker::nworkers]), loads the four forests onto its GPU once, and reuses them across all its months:

months = all_months()[args.worker::args.nworkers]             # this GPU's ~96 months
backends = {mid: load_backend(mid) for mid in (1, 2, 3, 4)}   # load once, reuse
for (year, month) in months:
    gdf = read_features(year, month)
    for mid in (1, 2, 3, 4):
        predict_and_write(year, month, gdf, mid, backends[mid])

Don’t let four workers each grab all 96 cores#

The GPUs do the inference, but the parquet decode and NetCDF write still runs on the CPU. We don’t want ~384 threads thrashing over 96 cores. The extra lines OMP_NUM_THREADS / OPENBLAS_NUM_THREADS / MKL_NUM_THREADS gives each CPU worker a clean slice.

The payoff#

When the batch job finishes, seff summarizes what it actually used:

[colette.kelly@hydra-l1 ~]$ seff 118272
State: COMPLETED (exit code 0)
Cores per node: 96
CPU Utilized: 00:16:24
CPU Efficiency: 2.39% of 11:26:24 core-walltime
Job Wall-clock time: 00:07:09
Memory Utilized: 256.00 GB (100.00% of 256.00 GB)

Two things:

  • CPU efficiency of 2.39% is OK for a GPU job. I reserved 96 cores for seven minutes but only spent 16 CPU-minutes, because the GPUs did the work — the cores are just there so host-side I/O (parquet decode, NetCDF writes) doesn’t bottleneck. Don’t reflexively “fix” the low number by shrinking the CPU request. But since the node is shared, not exclusive, good etiquette is to size --cpus-per-task to what you need without bottlenecking your code. This takes trial and error; it’s helpful to play with job requests on the compute nodes to get a sense of how much computing power different tasks need.

  • Memory at exactly 100% is fine here but it does mean that I bumped against my --mem=256G ceiling. The node has a full ~1 TB of memory, so I could have requested more. Just don’t reserve 1 TB “to be safe” on a shared node, but rather request what you’ll actually use. Again, this is usually a matter of trial and error as you get used to high-performance computing.

The payoff is that the same .predict() call, ~20 million rows, model already loaded and data already in memory:

backend

time for one .predict()

scikit-learn, CPU

4 min 19 s

nvForest, one H200

0.34 s

So that’s a roughly 750× speedup on identical work. So for the full job — four forests over 384 months — that’s about 31 billion predictions (20,452,736 grid points × 4 models × 384 months). What extrapolates to days of CPU inference finished in 7 minutes across the four H200s. That’s what makes using GPUs worth it.