11 Performance and memory

Modified

October 24, 2026

11.1 Measure the quantity you mean

Quantity	Definition and exclusions
Warm MHD throughput	`N^3 * iterations / elapsed / 1e6` MCell/s; synchronized timed MHD loop only
Phase-average seconds	Sum of a named synchronized production phase divided by measured cycles
Production phase total	Sum of the reported phases; not all process wall time
End-to-end wall time	Wrapper child-process interval, including startup and output work in that interval
Simulation progress	Redshift/physical time reached per wall time under the exact chosen model and tolerances

A fast bare MHD kernel is not the speed of full chemistry, radiation, gravity, spectral diagnostics, or checkpointing. A large timestep can improve progress while changing the trajectory. Report accuracy controls with performance.

11.2 Reproducible kernel benchmark

julia --project=. -t auto scripts/benchmark.jl --backend=cpu --n=32 \
  --iterations=10 --output=docs/data/benchmark-cpu.toml
julia --project=environments/metal -t auto scripts/benchmark.jl \
  --backend=metal --n=32 --iterations=10 --output=docs/data/benchmark-metal.toml
julia --project=environments/cuda -t auto scripts/benchmark.jl \
  --backend=cuda --n=32 --iterations=10 --output=docs/data/benchmark-cuda.toml

Run only the relevant backend after bootstrap. The benchmark warms the operator, resets its initial field before timing, and synchronizes at completion. CPU uses the reference integrator and GPU uses the cube path; this is not identical machine instruction work. Record integrator, grid, iterations, Julia threads, hardware, version, source hash, and scope with the throughput. ^{Warm compilation}

Comparing GLM and CT throughput

--mhd-solver=glm|ct|both selects the solver (default glm, unchanged). CT always runs the :ref integrator, so both prints one table with GLM on its default integrator, GLM on :ref (the like-for-like integrator for CT), and CT, with the centred divB dx/Brms of every row and the exact face divergence for CT. A case that fails to compile or run is reported NOT_RUN with the reason, not omitted. Outputs default to results/benchmark*.toml, one file per row (...-glm.toml, ...-ct.toml), and carry mhd_solver; none overwrites docs/data/benchmark-*.toml.

julia --project=environments/metal -t 4 scripts/benchmark.jl --backend=metal \
  --n=96 --iterations=60 --mhd-solver=both --output=results/bench-metal-compare.toml

Apple M5 Max, N=96, 60 iterations, warm, Float32 (two repeats, 2026-10-24; the GPU is shared, so repeat before quoting): GLM cube 112.1 and 111.1, CT ref 131.9 and 134.3 MCell/s. CT is the :ref integrator and carries 21 extra arrays per cell, so this is a kernel-only number, not a production-step speed; the CT step is also not GLM-limited (its dt is the plain CFL step, see Comparing CT and GLM runs).

GLM :ref runs on Metal through the tuple-packed kernel. Same machine, N=96, 60 iterations, Float32, 2026-10-24 (two --mhd-solver=both repeats, shared GPU): GLM cube 101.9 and 100.1, GLM ref 150.5 and 160.8, CT ref 113.2 and 135.3 MCell/s. In this kernel-only benchmark the cube is therefore not the fastest GLM kernel on this GPU; the production default is unchanged until a coupled-step comparison says otherwise.

For physical trajectories the production runner must not advance uncounted warmup states at a frozen cosmological time. Keep MHD_WARMUP=0; use MHD_PROFILE_SKIP_CYCLES to omit startup cycles from timing while preserving the evolved physical history.

11.3 Recorded measurements

The following records are loaded from benchmark*.json/benchmark*.toml, including an aggregate benchmarks.json when provided. Missing evidence is not a zero or an implied speed comparison.

benchmark-cpu.toml

Download the original record. This is recorded evidence, not a fresh measurement by the documentation build.

Recorded metadata and measurements

{
  "backend": "cpu",
  "created_utc": "2026-10-20T17:15:34.953",
  "device": "Apple M5 Max",
  "finite": true,
  "integrator": "ref",
  "iterations": 20,
  "julia_threads": 4,
  "julia_version": "1.12.6",
  "machine": "PHY-FD69NP6NG2",
  "mcells_per_second": 3.801557104721807,
  "n": 128,
  "platform": "Darwin/aarch64",
  "positive_density": true,
  "schema_version": 1,
  "scope": "warm MHD kernel only; excludes chemistry, FFTs, gravity, diagnostics, startup",
  "source_fingerprint_sha256": "d3871a0a7f907d9dd7c4f23b59924e0413acd65b3dd744b751f148ec81441f73",
  "status": "PASS",
  "wall_seconds": 11.033121125
}

benchmark-metal.toml

Download the original record. This is recorded evidence, not a fresh measurement by the documentation build.

Recorded metadata and measurements

{
  "backend": "metal",
  "created_utc": "2026-10-20T17:15:46.596",
  "device": "Metal.MTL.MTLDevice (object of type AGXG17CDevice)",
  "finite": true,
  "integrator": "cube",
  "iterations": 100,
  "julia_threads": 4,
  "julia_version": "1.12.6",
  "machine": "PHY-FD69NP6NG2",
  "mcells_per_second": 111.05188196743987,
  "n": 128,
  "platform": "Darwin/aarch64",
  "positive_density": true,
  "schema_version": 1,
  "scope": "warm MHD kernel only; excludes chemistry, FFTs, gravity, diagnostics, startup",
  "source_fingerprint_sha256": "d3871a0a7f907d9dd7c4f23b59924e0413acd65b3dd744b751f148ec81441f73",
  "status": "PASS",
  "wall_seconds": 1.888443458
}

11.4 Production profiles

performance.tsv and the log separate CFL, bare MHD, radiation, GLM projection, drag, chemistry, deposit, density assembly, FFT, gas gravity, and particle push. Radiation predictor/corrector/ledger and optional chemistry subdivisions are nested breakdowns, not extra phases to add again to the total. ^{function _time_phase}

Fix diagnostic cadence and final outputs in comparisons. Portable spectrum fallbacks may transfer full transformed grids to the host, while optimized Metal spectrum binning remains on-device; the same table schema does not imply the same overhead. FFT planning, allocation caches, data transfers, and initial compilation must be distinguished from steady state.

11.5 Memory budgeting

The nine Float32 MHD state arrays alone use 36 bytes/cell, excluding integrator scratch, particles, gravity, chemistry, transforms, and diagnostics. For a cubic grid the base four-channel radiation workspace approaches 32 bytes/cell and four persistent half-complex photon fields add about 16 bytes/cell. Optional three-component magnetic compensation adds 12 bytes/cell. These are component estimates, not a total resident-memory prediction.

Use MHD_MEMPROBE=1 and increasing small grids to measure the actual run. Check both live and peak/cached allocations, OS memory pressure, and checkpoint staging. Cubic scaling makes a factor-two grid refinement roughly eight times as costly in cell storage. Keep substantial headroom; do not infer a safe production grid from nominal unified memory or VRAM alone.

Avoid interpreting historical upstream M5 throughput, generic CUDA estimates, or another package’s bytes-per-cell model as a current PMFGLM measurement.