5 Outputs, analysis, and restart

Modified

October 24, 2026

5.1 Run directory

Every run gets its own explicit --output directory. The facade writes metadata before starting the child and updates completion/failure status when the process exits. ^{function run_simulation}

File	Contents
`metadata.toml`	Input configuration, source fingerprint, Julia/system details, manifest hash, status, wall time
`Project.toml`, `Manifest.toml`	Copies of the active environment declarations/manifest when present
`resolved-environment.toml`	All facade-resolved solver options and output paths
`run.log`	Solver stdout/stderr, stage timings, warnings, diagnostic summaries
`history.tsv`	Small redshift/time histories
`spectra.tsv`	Shell/angle power tables with explicit field labels and mode counts
`pdfs.tsv`	Gas/DM density histograms
`modes.tsv`	Matched-mode Fourier diagnostics where enabled
`performance.tsv`	Run-level phase averages and model/initialization metadata
`checkpoint.jls`	Restart state, only when checkpoint writing is enabled

Files can be absent when their diagnostic is disabled or a run fails before its first output. A path in the resolved configuration does not prove a file was written. Check metadata.toml, the log, and the final achieved redshift.

COMPLETED means the requested trajectory reached its target. A cosmological run that exits successfully at a cycle budget before zend is STOPPED_AT_LIMIT, not completed science. INCOMPLETE denotes a missing final redshift record; FAILED denotes an execution error. Do not treat exit code zero or a final diagnostic file alone as proof the endpoint was reached. ^{function _completion_status}

5.2 Histories and cadence

history.tsv includes cycle, measured, z, a, brms, divBdx_over_brms, delta_b_rms, delta_dm_rms, mean/min/max abundances, xe_total, transverse-mode diagnostics, pressure, internal energy, and temperature statistics, with the exact face divergence divb_face_dx_over_brms appended as the last column (NaN for GLM runs). performance.tsv appends mhd_solver, divb_face_dx_over_brms, and the three ct_initial_* numbers; metadata.toml records mhd_solver (and, for CT, the same initial numbers). brms is in code units, temperature is K, and disabled physical channels may appear as NaN; do not substitute zero. ^{function _write_history!}

diagnostics.redshifts requests common comparison epochs; diagnostics.every sets a cycle cadence. Adaptive histories can add samples where density/ionization changes. Initial spectra require MHD_INITIAL_PK=1 (enabled by the facade); final diagnostics are enabled by default. Use the actual z column for alignment rather than assuming the same cycle means the same physical epoch.

5.3 Power spectra

The shared TSV schema is

field cycle measured z a kbin mubin k mu_lo mu_hi P Nmodes

Nmodes=0 bins are empty, not zero power measurements. Gas/DM density, magnetic vector, photon density, and magnetic force spectra use one isotropic mu bin. Photon velocity and photon-baryon slip can retain angular bins. Vector power sums components; it is not a single-component spectrum. ^{function _write_pkmu!} ^{power_spectrum_aniso_gpu}

B_pressure_force, B_tension_force, and B_lorentz_force use the Maxwell split in Equation 6.3. Recover their cross spectrum using

2P_{PT}=P_L-P_P-P_T. \tag{5.1}

Changing cancellation between pressure and tension can alter total force power even when each auto-spectrum changes little. Density/particle mesh spectra are estimator outputs: do not presume a shot-noise subtraction or CIC-window deconvolution that is not explicitly performed by the implementation. Fix MHD_PK_KMAX and bin counts across grids for common physical edges.

The numeric k follows MHD_BOXSIZE, which defaults to 1. The physical TOML box does not by itself change that diagnostic option. Convert the axes and volume normalization using notation.

5.4 PDFs and images

The density PDF is per unit log10(rho) with normalization by total grid cells and bin width. Its schema includes field, epoch, bin edges, rho_mid, count, and pdf. Check histogram support before interpreting tails; do not compare a log-density PDF directly to a PDF per unit linear density.

Optional MHD_DENSITY_FRAME_DIR and MHD_DENSITY_FRAME_ZS request density frames. MHD_FINAL_B_DIR writes a final magnetic-field payload. These are separate from the small summary tables and can add substantial transfers and storage. The Python tools under tools/analysis consume the actual TSV schemas; their command-line help and tools/README.md describe supported plots.

uv sync --extra test
# A cosmological run can infer units from its metadata:
uv run pmf-analyze results/cosmology-first
# Explicit physical units and a separate output directory:
uv run pmf-analyze results/first --output results/first-analysis \
  --length-unit box \
  --box-ckpc-h 1 --spectral-boxsize 1 --hubble 0.674

With a run-directory argument, tables are discovered automatically and the default plot directory is RUN_DIR/analysis. Metadata supplies units for a cosmological run when available; explicit flags override inference. The unit flags in the second command explicitly assign a 1 ckpc/h physical box to the otherwise dimensionless smoke case; replace them with the actual experiment’s values. For the cosmological example, use its configured box_ckpc_h, h, and any MHD_BOXSIZE override. The command writes PDF/PNG diagnostics and an input-hash record; it does not convert code magnetic/force amplitudes into cgs silently.

For optional density frames:

uv run python -m tools.analysis.render_density_frames path/to/frames \
  results/frame-plots --box-ckpc-h 1 --hubble 0.674 --log-min -2 --log-max 2

The two panels are a central density slice and a line-of-sight maximum, not an integrated column density. Fixed color limits support epoch-to-epoch comparison; supplied physical box units label axes in comoving kpc without h.

With FFmpeg installed separately, combine the cycle-sorted PNGs into a movie:

ffmpeg -framerate 12 -pattern_type glob -i 'results/frame-plots/density_cycle*.png' \
  -vf 'scale=1440:640:force_original_aspect_ratio=decrease,pad=1440:640:(ow-iw)/2:(oh-ih)/2' \
  -c:v libx264 -pix_fmt yuv420p results/density-evolution.mp4

Keep the same log-density color limits for every frame and comparison run. The fixed output dimensions also avoid small caption/bounding-box differences changing the encoded frame size. Archive frames.json and the original tables with the movie so its epochs and color normalization remain recoverable.

Keep raw tables alongside generated figures. Plot captions should state the field, actual redshift, physical units, band/window, normalization, and whether the plotted quantity is P(k) or dimensionless k^3 P/(2*pi^2).

5.5 Checkpoints and restart

Enable checkpointing in the original configuration:

[run]
checkpoint_every = 100
checkpoint_final = true
resume = false

To continue, change resume=true, preserve all evolution settings, and rerun against the same output directory. Extending cosmology.zend is allowed by the driver; changing its physical history is not generally compatible. The wrapper additionally rejects a changed source fingerprint. It also compares all resolved evolution options with the original run metadata, including the cosmology, physical box/field normalization, chemistry units and helium treatment. Only endpoint/cycle-budget, resume/checkpoint cadence, logging, and the listed output-cadence controls can change. An incompatible physical configuration is rejected before the previous metadata is overwritten. ^{_validate_restart_environment}

julia --project=. -t auto scripts/run_pmf.jl path/to/resume.toml \
  --backend=cpu --output=results/my-run

The checkpoint writer streams fields through host arrays to a temporary file, then renames it. It stores configuration, runtime counters/clocks, MHD fields, particle and chemistry state, and active photon/compensation payloads. Restore checks field ordering, shape, type, and configuration compatibility. ^{function _write_pmf_checkpoint} ^{function _restore_pmf_checkpoint!}

Julia Serialization is a restart mechanism, not a long-term portable archive or an untrusted-file format. Keep the Julia version, manifest, source, and metadata with a checkpoint; never deserialize untrusted files. The code’s legacy/override migration paths are expert operations, not a promise of cross-version or cross-backend portability.

Restart logs/tables can append to earlier outputs. Check for repeated epochs and preserve pre-resume metadata when publishing an audit trail. Validate a continuous versus split/resumed trajectory before relying on a new checkpoint configuration.

5.6 Streaming histories and direct GIF export

Set diagnostics.streaming=true to emit streaming.tsv at regular diagnostic epochs. A nonzero named [streaming] boost also enables it automatically. The vector mean vbc_{x,y,z}_cms, total vbc_rms_cms, and RMS about the vector mean vbc_dispersion_cms are physical peculiar cm/s, sampled at current DM positions by periodic CIC interpolation of gas velocity. The averages use equal DM particle masses. vc_{x,y,z}_cms records the DM bulk motion; vb_sampled_{x,y,z}_cms records the gas mean sampled on those particles. They are not Eulerian volume averages or a relative-velocity power spectrum. Disabled streaming output is absent, not a zero field. pmf-analyze RUN_DIR plots it alongside the existing baryon/DM density growth histories and spectra. ^{sample_baryon_dm_velocity!}

The legacy modes.tsv gas velocities retain their code units; new gas_v_{x,y,z}_cms rows give explicit physical units. dm_v_{x,y,z} are physical cm/s Fourier modes of a CIC mass/momentum velocity estimate on the gas mesh, using current particle positions for either lattice displacements or absolute positions. Empty cells have zero estimated velocity and are counted in dm_velocity_empty_fraction; a large empty fraction limits the interpretation of gas/DM comparisons. dm_displacement_x remains a lattice-label diagnostic. See relative velocity conventions and tests. Mode wavenumbers use the physical box; spectra.tsv still uses the separate coordinate box documented above.

For a small full-physics exploratory run:

julia --project=. -t 4 scripts/run_pmf.jl examples/streaming-planar-smoke.toml \
  --backend=cpu --output=results/streaming-planar --dry-run
julia --project=. -t 4 scripts/run_pmf.jl examples/streaming-planar-smoke.toml \
  --backend=cpu --output=results/streaming-planar
uv run pmf-analyze results/streaming-planar
uv run python -m tools.analysis.render_density_frames \
  results/streaming-planar/frames results/streaming-planar/animation \
  --box-ckpc-h 2 --hubble 0.674 --log-min -2 --log-max 2 \
  --gif results/streaming-planar/animation/gas.gif --fps 8
uv run python -m tools.analysis.render_density_frames \
  results/streaming-planar/frames/dm results/streaming-planar/animation-dm \
  --box-ckpc-h 2 --hubble 0.674 --log-min -2 --log-max 2 \
  --gif results/streaming-planar/animation-dm/dm.gif --fps 8

diagnostics.frame_redshifts resolves the frame directory inside the run. diagnostics.frame_field="both" writes gas under frames/ and DM under frames/dm/; "gas" is the default and "dm" writes only the latter. The existing exact-redshift landing is shared with these requests. The renderer sorts manifest rows by cycle, rejects duplicate cycles, preserves fixed color limits, and records payload hashes and actual epochs in frames.json. GIF playback uses equal frame intervals; those intervals do not imply equal physical time. GIFs use Pillow through the existing Matplotlib installation; MP4 encoding remains available with the FFmpeg recipe above.

--- title: "Outputs, analysis, and restart" --- ## Run directory {#sec-run-directory} Every run gets its own explicit `--output` directory. The facade writes metadata before starting the child and updates completion/failure status when the process exits. {{< src "src/provenance.jl" "function run_simulation" >}} | File | Contents | |---|---| | `metadata.toml` | Input configuration, source fingerprint, Julia/system details, manifest hash, status, wall time | | `Project.toml`, `Manifest.toml` | Copies of the active environment declarations/manifest when present | | `resolved-environment.toml` | All facade-resolved solver options and output paths | | `run.log` | Solver stdout/stderr, stage timings, warnings, diagnostic summaries | | `history.tsv` | Small redshift/time histories | | `spectra.tsv` | Shell/angle power tables with explicit field labels and mode counts | | `pdfs.tsv` | Gas/DM density histograms | | `modes.tsv` | Matched-mode Fourier diagnostics where enabled | | `performance.tsv` | Run-level phase averages and model/initialization metadata | | `checkpoint.jls` | Restart state, only when checkpoint writing is enabled | Files can be absent when their diagnostic is disabled or a run fails before its first output. A path in the resolved configuration does not prove a file was written. Check `metadata.toml`, the log, and the final achieved redshift. `COMPLETED` means the requested trajectory reached its target. A cosmological run that exits successfully at a cycle budget before `zend` is `STOPPED_AT_LIMIT`, not completed science. `INCOMPLETE` denotes a missing final redshift record; `FAILED` denotes an execution error. Do not treat exit code zero or a final diagnostic file alone as proof the endpoint was reached. {{< src "src/provenance.jl" "function _completion_status" >}} ## Histories and cadence {#sec-histories} `history.tsv` includes `cycle`, `measured`, `z`, `a`, `brms`, `divBdx_over_brms`, `delta_b_rms`, `delta_dm_rms`, mean/min/max abundances, `xe_total`, transverse-mode diagnostics, pressure, internal energy, and temperature statistics, with the exact face divergence `divb_face_dx_over_brms` appended as the last column (`NaN` for GLM runs). `performance.tsv` appends `mhd_solver`, `divb_face_dx_over_brms`, and the three `ct_initial_*` numbers; `metadata.toml` records `mhd_solver` (and, for CT, the same initial numbers). `brms` is in **code units**, temperature is K, and disabled physical channels may appear as `NaN`; do not substitute zero. {{< src "src/production/runner.jl" "function _write_history!" >}} `diagnostics.redshifts` requests common comparison epochs; `diagnostics.every` sets a cycle cadence. Adaptive histories can add samples where density/ionization changes. Initial spectra require `MHD_INITIAL_PK=1` (enabled by the facade); final diagnostics are enabled by default. Use the actual `z` column for alignment rather than assuming the same cycle means the same physical epoch. ## Power spectra {#sec-power-spectra} The shared TSV schema is ```text field cycle measured z a kbin mubin k mu_lo mu_hi P Nmodes ``` `Nmodes=0` bins are empty, not zero power measurements. Gas/DM density, magnetic vector, photon density, and magnetic force spectra use one isotropic mu bin. Photon velocity and photon-baryon slip can retain angular bins. Vector power sums components; it is not a single-component spectrum. {{< src "src/production/runner.jl" "function _write_pkmu!" >}} {{< src "lib/PoissonKernels/src/gpu_fft.jl" "power_spectrum_aniso_gpu" >}} `B_pressure_force`, `B_tension_force`, and `B_lorentz_force` use the Maxwell split in @eq-maxwell. Recover their cross spectrum using $$ 2P_{PT}=P_L-P_P-P_T. $$ {#eq-cross-spectrum} Changing cancellation between pressure and tension can alter total force power even when each auto-spectrum changes little. Density/particle mesh spectra are estimator outputs: do not presume a shot-noise subtraction or CIC-window deconvolution that is not explicitly performed by the implementation. Fix `MHD_PK_KMAX` and bin counts across grids for common physical edges. The numeric `k` follows `MHD_BOXSIZE`, which defaults to **1**. The physical TOML box does not by itself change that diagnostic option. Convert the axes and volume normalization using [notation](notation.qmd#sec-wavenumbers). ## PDFs and images {#sec-plots} The density PDF is per unit `log10(rho)` with normalization by total grid cells and bin width. Its schema includes `field`, epoch, bin edges, `rho_mid`, `count`, and `pdf`. Check histogram support before interpreting tails; do not compare a log-density PDF directly to a PDF per unit linear density. Optional `MHD_DENSITY_FRAME_DIR` and `MHD_DENSITY_FRAME_ZS` request density frames. `MHD_FINAL_B_DIR` writes a final magnetic-field payload. These are separate from the small summary tables and can add substantial transfers and storage. The Python tools under `tools/analysis` consume the actual TSV schemas; their command-line help and `tools/README.md` describe supported plots. ```sh uv sync --extra test # A cosmological run can infer units from its metadata: uv run pmf-analyze results/cosmology-first # Explicit physical units and a separate output directory: uv run pmf-analyze results/first --output results/first-analysis \ --length-unit box \ --box-ckpc-h 1 --spectral-boxsize 1 --hubble 0.674 ``` With a run-directory argument, tables are discovered automatically and the default plot directory is `RUN_DIR/analysis`. Metadata supplies units for a cosmological run when available; explicit flags override inference. The unit flags in the second command explicitly assign a 1 ckpc/h physical box to the otherwise dimensionless smoke case; replace them with the actual experiment's values. For the cosmological example, use its configured `box_ckpc_h`, `h`, and any `MHD_BOXSIZE` override. The command writes PDF/PNG diagnostics and an input-hash record; it does not convert code magnetic/force amplitudes into cgs silently. For optional density frames: ```sh uv run python -m tools.analysis.render_density_frames path/to/frames \ results/frame-plots --box-ckpc-h 1 --hubble 0.674 --log-min -2 --log-max 2 ``` The two panels are a central density slice and a line-of-sight **maximum**, not an integrated column density. Fixed color limits support epoch-to-epoch comparison; supplied physical box units label axes in comoving kpc without h. With FFmpeg installed separately, combine the cycle-sorted PNGs into a movie: ```sh ffmpeg -framerate 12 -pattern_type glob -i 'results/frame-plots/density_cycle*.png' \ -vf 'scale=1440:640:force_original_aspect_ratio=decrease,pad=1440:640:(ow-iw)/2:(oh-ih)/2' \ -c:v libx264 -pix_fmt yuv420p results/density-evolution.mp4 ``` Keep the same log-density color limits for every frame and comparison run. The fixed output dimensions also avoid small caption/bounding-box differences changing the encoded frame size. Archive `frames.json` and the original tables with the movie so its epochs and color normalization remain recoverable. Keep raw tables alongside generated figures. Plot captions should state the field, actual redshift, physical units, band/window, normalization, and whether the plotted quantity is `P(k)` or dimensionless `k^3 P/(2*pi^2)`. ## Checkpoints and restart {#sec-restart} Enable checkpointing in the original configuration: ```toml [run] checkpoint_every = 100 checkpoint_final = true resume = false ``` To continue, change `resume=true`, preserve all evolution settings, and rerun against the same output directory. Extending `cosmology.zend` is allowed by the driver; changing its physical history is not generally compatible. The wrapper additionally rejects a changed source fingerprint. It also compares all resolved evolution options with the original run metadata, including the cosmology, physical box/field normalization, chemistry units and helium treatment. Only endpoint/cycle-budget, resume/checkpoint cadence, logging, and the listed output-cadence controls can change. An incompatible physical configuration is rejected before the previous metadata is overwritten. {{< src "src/provenance.jl" "_validate_restart_environment" >}} ```sh julia --project=. -t auto scripts/run_pmf.jl path/to/resume.toml \ --backend=cpu --output=results/my-run ``` The checkpoint writer streams fields through host arrays to a temporary file, then renames it. It stores configuration, runtime counters/clocks, MHD fields, particle and chemistry state, and active photon/compensation payloads. Restore checks field ordering, shape, type, and configuration compatibility. {{< src "src/production/runner.jl" "function _write_pmf_checkpoint" >}} {{< src "src/production/runner.jl" "function _restore_pmf_checkpoint!" >}} Julia Serialization is a restart mechanism, **not** a long-term portable archive or an untrusted-file format. Keep the Julia version, manifest, source, and metadata with a checkpoint; never deserialize untrusted files. The code's legacy/override migration paths are expert operations, not a promise of cross-version or cross-backend portability. Restart logs/tables can append to earlier outputs. Check for repeated epochs and preserve pre-resume metadata when publishing an audit trail. Validate a continuous versus split/resumed trajectory before relying on a new checkpoint configuration. ## Streaming histories and direct GIF export {#sec-streaming-output} Set `diagnostics.streaming=true` to emit `streaming.tsv` at regular diagnostic epochs. A nonzero named `[streaming]` boost also enables it automatically. The vector mean `vbc_{x,y,z}_cms`, total `vbc_rms_cms`, and RMS about the vector mean `vbc_dispersion_cms` are **physical peculiar cm/s**, sampled at current DM positions by periodic CIC interpolation of gas velocity. The averages use equal DM particle masses. `vc_{x,y,z}_cms` records the DM bulk motion; `vb_sampled_{x,y,z}_cms` records the gas mean sampled on those particles. They are not Eulerian volume averages or a relative-velocity power spectrum. Disabled streaming output is absent, not a zero field. `pmf-analyze RUN_DIR` plots it alongside the existing baryon/DM density growth histories and spectra. {{< src "lib/MHDKernels/src/streaming.jl" "sample_baryon_dm_velocity!" >}} The legacy `modes.tsv` gas velocities retain their code units; new `gas_v_{x,y,z}_cms` rows give explicit physical units. `dm_v_{x,y,z}` are physical cm/s Fourier modes of a CIC mass/momentum velocity estimate on the gas mesh, using current particle positions for either lattice displacements or absolute positions. Empty cells have zero estimated velocity and are counted in `dm_velocity_empty_fraction`; a large empty fraction limits the interpretation of gas/DM comparisons. `dm_displacement_x` remains a lattice-label diagnostic. See [relative velocity conventions and tests](relative-velocities.qmd). Mode wavenumbers use the physical box; `spectra.tsv` still uses the separate coordinate box documented above. For a small full-physics exploratory run: ```sh julia --project=. -t 4 scripts/run_pmf.jl examples/streaming-planar-smoke.toml \ --backend=cpu --output=results/streaming-planar --dry-run julia --project=. -t 4 scripts/run_pmf.jl examples/streaming-planar-smoke.toml \ --backend=cpu --output=results/streaming-planar uv run pmf-analyze results/streaming-planar uv run python -m tools.analysis.render_density_frames \ results/streaming-planar/frames results/streaming-planar/animation \ --box-ckpc-h 2 --hubble 0.674 --log-min -2 --log-max 2 \ --gif results/streaming-planar/animation/gas.gif --fps 8 uv run python -m tools.analysis.render_density_frames \ results/streaming-planar/frames/dm results/streaming-planar/animation-dm \ --box-ckpc-h 2 --hubble 0.674 --log-min -2 --log-max 2 \ --gif results/streaming-planar/animation-dm/dm.gif --fps 8 ``` `diagnostics.frame_redshifts` resolves the frame directory inside the run. `diagnostics.frame_field="both"` writes gas under `frames/` and DM under `frames/dm/`; `"gas"` is the default and `"dm"` writes only the latter. The existing exact-redshift landing is shared with these requests. The renderer sorts manifest rows by cycle, rejects duplicate cycles, preserves fixed color limits, and records payload hashes and actual epochs in `frames.json`. GIF playback uses equal frame intervals; those intervals do not imply equal physical time. GIFs use Pillow through the existing Matplotlib installation; MP4 encoding remains available with the FFmpeg recipe above.