Validation-Centric AI-Assisted GPU Porting of a 250,000+ Line Legacy Weather Simulation Code

GPU porting legacy scientific code requires more than raw performance—it demands proof that the science still holds. This paper documents how researchers ported CReSS, a 260,000-line Fortran weather simulation code, to GPU using an AI agent while preserving 28 years of accumulated scientific validity.

The Core Challenge

Legacy scientific applications carry scientific credibility built through decades of observation comparisons and domain studies. CReSS, developed since 1998, models tropical cyclones, heavy rainfall, and convective systems. GPU porting cannot simply regenerate this code from scratch—it must adapt the existing implementation while preserving validated behavior.

Three technical requirements make this difficult:

• Verification: AI-generated code changes must match the existing CPU behavior
• Numerical validation: GPU floating-point differences must be diagnosed as acceptable variation or implementation defects
• Runtime-state reconstruction: Kernel inputs depend on long initialization chains and time-evolving physical processes—synthetic inputs cannot substitute

The Six-Phase Workflow

The team used Claude Code (Opus 4.5–4.6) as the AI agent across six phases:

1. Code meta-review: Annotate 387 OpenMP parallel regions; flag GPU-porting barriers such as thread-local variables, atomic operations, and global writes
2. Profiling: Identify which 162 of 387 regions execute in the target typhoon scenario
3. Kernel extraction and CPU benchmark generation: Dump runtime states from an actual simulation; build standalone benchmarks for each kernel
4. GPU transformation: Apply OpenACC kernels directives with loop independent annotations; validate each kernel against dumped reference data
5. Integration: Insert kernels into the original code using #ifdef conditional compilation for bisection-style debugging
6. Performance validation: Profile with NVIDIA Nsight; revise anomalous kernels locally without changing inter-kernel interfaces

The element-wise error metric for both CPU verification and GPU validation:

e(i,j,k) = |z(i,j,k) - y_ref(i,j,k)| / max(|y_ref(i,j,k)|, 1e-20)

Kernels pass when the maximum error stays below τ = 10⁻⁵.

Results

Running a real September 2022 typhoon simulation on an NVIDIA GH200 node (H100 GPU + 72-core Grace CPU):

• 162 kernels received validated GPU implementations
• 5.1× speedup over the 72-thread CPU baseline
• Application-level validation passed: pressure perturbation errors at 1.0×10⁻⁵ and 5.6×10⁻⁵, both below the 10⁻⁴ threshold
• Five kernels showed single-element discrepancies requiring human inspection

The five detected discrepancies illustrate why kernel-level validation matters. In bruntv.f90, the CPU computed temperature as 233.16002 K while the GPU computed 233.16000 K—a one-ulp difference in single precision. This flipped a branch condition t <= tlow (threshold: 233.16 K), applying a latent-heat correction only on the GPU. Kernel-level validation exposed the responsible kernel, the affected condition, and the numerical mechanism. CReSS developers confirmed all five cases as acceptable numerical variation.

Workflow Cost and Failure Modes

Total development time ran approximately 100 supervised hours over three months on the target HPC system. Phase 3—kernel extraction and CPU benchmark generation—consumed more than half this time.

The Dump Acquisition Bottleneck

Each dump acquisition requires roughly one hour of simulation re-execution. This makes trial-and-error loops extremely expensive. The workflow must batch all 162 kernels’ dump instrumentation, execute once, then verify.

The primary failure mode: conditional allocation hidden from local code inspection. When a CReSS physics option is disabled, a full-domain array argument gets backed by a minimal dummy object. The callee interface looks identical whether the dummy or real array is present. Dump instrumentation that ignores the physics-option guard causes a segmentation fault at dump time—not at the array declaration. In one session, the agent misdiagnosed this as a bug in the dump routine and spent an entire session modifying the dump function before manual termination.

Specification-Based Session Management

Interactive jobs on shared HPC systems impose session limits (two hours on the test system). The team externalized all workflow rules into persistent specification files: variable_list construction order, binary I/O byte-order settings (-Mbyteswapio), recovery policies, and dummy-array handling rules.

Comparing prompt-based vs. specification-based workflows across five runs each:

Specifications stabilize convergence but do not control cost. Adding an explicit batch-checking recovery rule—after one dump failure, inspect all similar variables before rerunning—reduced dump executions to 1–3 across successful runs, down from cases requiring 5–7.

Snapshot Validation Coverage Limits

The workflow dumps each kernel’s state at its last invocation in the 360-step simulation. This misses branches that execute only at intermediate timesteps. During integration, one kernel failed application-level validation because an AI-generated CPU benchmark omitted a branch absent at the dump point but present at earlier timesteps. The fix required returning to CPU benchmark generation.

A minimal dump configuration for one process, one timestep, and 162 kernels produced over 400 GB of data. Dumping all timesteps is impractical. The validated GPU port therefore covers the target typhoon scenario specifically—not all possible CReSS executions.

Implementation Notes

Unified Memory over explicit data directives: The team used -gpu=managed to simplify the initial porting stage. This avoids complex data-movement management during validation but leaves performance on the table. Memory-bandwidth utilization ran 35–60% of H100 peak, reasonable for directive-based OpenACC.

Compiler-guided variable enumeration: Rather than relying on AI static analysis alone, the team applied OpenMP default(none) to kernel copies. Compiler errors then revealed every variable requiring explicit data-sharing attributes. This converts expensive dump-time failures into cheap compile-time feedback.

Conditional compilation for integration debugging: Each kernel integrates via #ifdef guards, enabling selective GPU kernel activation during application-level validation failures.

Key Takeaways for DevOps and HPC Teams

AI agents accelerate GPU porting of large scientific codes, but the workflow design determines whether that acceleration is practical:

1. Dump-based validation is non-negotiable for scientific applications. Synthetic inputs miss physics-dependent states that accumulate over simulation time.
2. Externalize workflow context as specifications, not prompts. Session boundaries on shared HPC systems erase conversational context; persistent specification files preserve procedural requirements.
3. Make recovery cost-aware explicitly. After any dump failure, mandate batch inspection of similar variables before rerunning the simulation. Agents default to local repair and immediate retry.
4. Treat kernel-level and application-level validation as complementary. Kernel validation localizes numerical discrepancies; application validation catches behavior that only appears across the full time evolution.
5. Phase 3 dominates development cost. Runtime-state reconstruction—not GPU transformation—is where porting projects succeed or fail.

The workflow achieved in roughly 100 supervised hours what the team estimated would otherwise require months to years of manual effort. The bottleneck is not code generation—it is the infrastructure for making generated code verifiable.