Setting a user-defined atmosphere iteration number or time on GPUs causes illegal memory crash

[taimoorsohail]

I am trying to run a ClimaOcean.jl simulation by loading a set of u, v, T, S and e fields and launching the simulation at some specified time and iteration number (basically a rudimentary checkpointer implementation). Here is a working example: https://github.com/taimoorsohail/ocean-ensembles/blob/ts/parallel-run/test/synching_clock.jl

Unfortunately, the synching of the simulation and atmosphere clocks causes an error on GPUs (only - CPU works fine) when doing run!(simulation).

Specifically, I get the error:

ERROR: LoadError: CUDA error: an illegal memory access was encountered (code 700, ERROR_ILLEGAL_ADDRESS)

originating from the line: https://github.com/CliMA/Oceananigans.jl/blob/fc8db79871b673811d6cc5df0e7057af869d601f/src/Models/HydrostaticFreeSurfaceModels/SplitExplicitFreeSurfaces/compute_slow_tendencies.jl#L42
in Oceananigans.jl.

I tested some combinations of time and iteration numbers, and the code seems to work when set to any iteration no. but time is set to 0 OR when the atmosphere and simulation times are set to 0 but ocean time is set to a user defined value.

Is there something specific to GPUs I am hitting here?

[navidcy]

I'll make a MWE that reproduces the error without referencing local paths.

[glwagner]

It seems likely that this has to do with the FieldTimeSeries not getting updated properly. I'm not sure how the error could originate from the indicated place but that could be a red herring, eg the error is triggered prior to it

[glwagner]

Possibly the thing to check is whether you can call Oceananigans.TimeSteppers.update_state!(coupled_model) without an error.

[navidcy]

So me and @taimoorsohail managed to boil this down to the following.

It seems that if we try to change the clock to a time beyond what's already in the atmosphere memory backend we get the strange GPU error breakdown.

Here's a mwe_sync_clock.jl script:

using ClimaOcean, Oceananigans, Oceananigans.Units

arch = GPU()
grid = LatitudeLongitudeGrid(arch, size = (60, 30, 10), z = (-6000, 0), latitude = (-75, 75), longitude = (0, 360), halo = (6, 6, 3))
ocean = ocean_simulation(grid; free_surface = SplitExplicitFreeSurface(grid; substeps=10))

time_instances_in_memory = 14
atmosphere = JRA55PrescribedAtmosphere(arch; backend=JRA55NetCDFBackend(time_instances_in_memory))
radiation = Radiation(arch)
coupled_model = OceanSeaIceModel(ocean; atmosphere, radiation)

@info "initialized a coupled_model with:"

@show coupled_model.ocean.model.clock.iteration
@show coupled_model.ocean.model.clock.time
@show coupled_model.atmosphere.clock.iteration
@show coupled_model.atmosphere.clock.time
@show coupled_model.clock.iteration
@show coupled_model.clock.time

@info "and with an atmosphere that has a JRA55NetCDFBackend with $(time_instances_in_memory) instances in memory."

@info "let's timestep coupled_model..."

for _ in 1:2
    Oceananigans.TimeSteppers.update_state!(coupled_model)
    Oceananigans.TimeSteppers.time_step!(coupled_model, 5minutes)
end

# set the time
JRA55_time_inmemory = time_instances_in_memory * 3hours
time = JRA55_time_inmemory + 4hours

# set the iteration
iteration = 242

@info "...timestepping worked!"
@info "let's change the clock time to $(time / hour) hours and timestep again"

coupled_model.ocean.model.clock.iteration = iteration
coupled_model.ocean.model.clock.time = time
coupled_model.atmosphere.clock.iteration = iteration
coupled_model.atmosphere.clock.time = time
coupled_model.clock.iteration = iteration
coupled_model.clock.time = time

for _ in 1:2
    Oceananigans.TimeSteppers.update_state!(coupled_model)
    Oceananigans.TimeSteppers.time_step!(coupled_model, 5minutes)
end

@info "that worked; the clock is now:"

@show coupled_model.ocean.model.clock.iteration
@show coupled_model.ocean.model.clock.time
@show coupled_model.atmosphere.clock.iteration
@show coupled_model.atmosphere.clock.time
@show coupled_model.clock.iteration
@show coupled_model.clock.time

Above, we create an atmosphere with time_instances_in_memory = 14. For a 3-hourly resolution that the JRA55 is, that means atmosphere has in memory time_instances_in_memory * 3hours.

If we try to set the clock to anything up to 44hours things seem OK. When we cross the 14*3+3 mark, things get hairy.

time = JRA55_time_inmemory + 2hours

julia> include("mwe_clock_sync.jl")
[ Info: initialized a coupled_model with:
coupled_model.ocean.model.clock.iteration = 0
coupled_model.ocean.model.clock.time = 0.0
coupled_model.atmosphere.clock.iteration = 0
coupled_model.atmosphere.clock.time = 0.0
coupled_model.clock.iteration = 0
coupled_model.clock.time = 0.0
[ Info: and with an atmosphere that has a JRA55NetCDFBackend with 14 instances in memory.
[ Info: let's timestep coupled_model...
[ Info: ...timestepping worked!
[ Info: let's change the clock time to 44.0 hours and timestep again
[ Info: that worked; the clock is now:
coupled_model.ocean.model.clock.iteration = 244
coupled_model.ocean.model.clock.time = 159000.0
coupled_model.atmosphere.clock.iteration = 244
coupled_model.atmosphere.clock.time = 159000.0
coupled_model.clock.iteration = 244
coupled_model.clock.time = 159000.0
159000.0

time = JRA55_time_inmemory + 4hours

julia> include("mwe_clock_sync.jl")
[ Info: initialized a coupled_model with:
coupled_model.ocean.model.clock.iteration = 0
coupled_model.ocean.model.clock.time = 0.0
coupled_model.atmosphere.clock.iteration = 0
coupled_model.atmosphere.clock.time = 0.0
coupled_model.clock.iteration = 0
coupled_model.clock.time = 0.0
[ Info: and with an atmosphere that has a JRA55NetCDFBackend with 14 instances in memory.
[ Info: let's timestep coupled_model...
[ Info: ...timestepping worked!
[ Info: let's change the clock time to 46.0 hours and timestep again
ERROR: LoadError: CUDA error: an illegal memory access was encountered (code 700, ERROR_ILLEGAL_ADDRESS)
Stacktrace:
  [1] throw_api_error(res::CUDA.cudaError_enum)
    @ CUDA /storage3/navidcy/.julia/packages/CUDA/ja0IX/lib/cudadrv/libcuda.jl:30
  [2] isdone
    @ /storage3/navidcy/.julia/packages/CUDA/ja0IX/lib/cudadrv/stream.jl:134 [inlined]
  [3] spinning_synchronization(f::typeof(CUDA.isdone), obj::CUDA.CuStream)
    @ CUDA /storage3/navidcy/.julia/packages/CUDA/ja0IX/lib/cudadrv/synchronization.jl:79
  [4] synchronize(stream::CUDA.CuStream; blocking::Bool, spin::Bool)
    @ CUDA /storage3/navidcy/.julia/packages/CUDA/ja0IX/lib/cudadrv/synchronization.jl:196
  [5] synchronize
    @ /storage3/navidcy/.julia/packages/CUDA/ja0IX/lib/cudadrv/synchronization.jl:194 [inlined]
  [6] synchronize
    @ /storage3/navidcy/.julia/packages/CUDA/ja0IX/src/memory.jl:527 [inlined]
  [7] synchronize
    @ /storage3/navidcy/.julia/packages/CUDA/ja0IX/src/array.jl:493 [inlined]
  [8] (::CUDA.var"#1162#1163"{Float32, Vector{Float32}, Int64, CUDA.CuArray{Float32, 1, CUDA.DeviceMemory}, Int64, Int64})()
    @ CUDA /storage3/navidcy/.julia/packages/CUDA/ja0IX/src/array.jl:613
  [9] #context!#1025
    @ /storage3/navidcy/.julia/packages/CUDA/ja0IX/lib/cudadrv/state.jl:168 [inlined]
 [10] context!
    @ /storage3/navidcy/.julia/packages/CUDA/ja0IX/lib/cudadrv/state.jl:163 [inlined]
 [11] unsafe_copyto!(dest::Vector{Float32}, doffs::Int64, src::CUDA.CuArray{Float32, 1, CUDA.DeviceMemory}, soffs::Int64, n::Int64)
    @ CUDA /storage3/navidcy/.julia/packages/CUDA/ja0IX/src/array.jl:608
 [12] copyto!
    @ /storage3/navidcy/.julia/packages/CUDA/ja0IX/src/array.jl:560 [inlined]
 [13] getindex
    @ /storage3/navidcy/.julia/packages/GPUArrays/uiVyU/src/host/indexing.jl:52 [inlined]
 [14] getindex
    @ /storage3/navidcy/.julia/packages/OffsetArrays/b7qpm/src/OffsetArrays.jl:435 [inlined]
 [15] getindex
    @ ./subarray.jl:323 [inlined]
 [16] mapreduce_impl(f::typeof(identity), op::typeof(min), A::SubArray{…}, first::Int64, last::Int64)
    @ Base ./reduce.jl:645
 [17] _mapreduce(f::typeof(identity), op::typeof(min), ::IndexLinear, A::SubArray{Float32, 1, OffsetArrays.OffsetVector{…}, Tuple{…}, true})
    @ Base ./reduce.jl:447
 [18] _mapreduce_dim
    @ ./reducedim.jl:367 [inlined]
 [19] mapreduce
    @ ./reducedim.jl:359 [inlined]
 [20] _minimum
    @ ./reducedim.jl:1017 [inlined]
 [21] _minimum
    @ ./reducedim.jl:1016 [inlined]
 [22] minimum
    @ ./reducedim.jl:1012 [inlined]
 [23] macro expansion
    @ /storage3/navidcy/.julia/packages/GPUArraysCore/aNaXo/src/GPUArraysCore.jl:206 [inlined]
 [24] compute_bounding_nodes(::Nothing, grid::Oceananigans.Grids.ZRegularLLG{…}, LH::Type{…}, hnodes::typeof(λnodes))
    @ ClimaOcean.DataWrangling.JRA55 ~/ClimaOcean.jl/src/DataWrangling/JRA55/JRA55_field_time_series.jl:18
 [25] compute_bounding_indices(longitude::Nothing, latitude::Nothing, grid::Oceananigans.Grids.ZRegularLLG{…}, LX::Type, LY::Type, λc::Vector{…}, φc::Vector{…})
    @ ClimaOcean.DataWrangling.JRA55 ~/ClimaOcean.jl/src/DataWrangling/JRA55/JRA55_field_time_series.jl:63
 [26] set!(fts::FieldTimeSeries{…}, backend::JRA55NetCDFBackend{…})
    @ ClimaOcean.DataWrangling.JRA55 ~/ClimaOcean.jl/src/DataWrangling/JRA55/JRA55_field_time_series.jl:121
 [27] set!
    @ ~/ClimaOcean.jl/src/DataWrangling/JRA55/JRA55_field_time_series.jl:110 [inlined]
 [28] update_field_time_series!
    @ /storage3/navidcy/.julia/dev/Oceananigans/src/OutputReaders/field_time_series_indexing.jl:335 [inlined]
 [29] update_field_time_series!(fts::FieldTimeSeries{…}, time_index::Time{…})
    @ Oceananigans.OutputReaders /storage3/navidcy/.julia/dev/Oceananigans/src/OutputReaders/field_time_series_indexing.jl:324
 [30] time_step!
    @ ~/ClimaOcean.jl/src/OceanSeaIceModels/PrescribedAtmospheres.jl:362 [inlined]
 [31] error in running finalizer: CUDA.CuError(code=CUDA.cudaError_enum(0x000002bc))
throw_api_error at /storage3/navidcy/.julia/packages/CUDA/ja0IX/lib/cudadrv/libcuda.jl:30
check at /storage3/navidcy/.julia/packages/CUDA/ja0IX/lib/cudadrv/libcuda.jl:37 [inlined]
cuStreamDestroy_v2 at /storage3/navidcy/.julia/packages/GPUToolbox/cZlg7/src/ccalls.jl:33 [inlined]
#957 at /storage3/navidcy/.julia/packages/CUDA/ja0IX/lib/cudadrv/stream.jl:89
#context!#1025 at /storage3/navidcy/.julia/packages/CUDA/ja0IX/lib/cudadrv/state.jl:168
context! at /storage3/navidcy/.julia/packages/CUDA/ja0IX/lib/cudadrv/state.jl:163
unsafe_destroy! at /storage3/navidcy/.julia/packages/CUDA/ja0IX/lib/cudadrv/stream.jl:88
jfptr_unsafe_destroyNOT._11875 at /storage3/navidcy/.julia/compiled/v1.10/CUDA/oWw5k_NdVy8.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:3077
run_finalizer at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gc.c:318
jl_gc_run_finalizers_in_list at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gc.c:408
run_finalizers at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gc.c:454
run_finalizers at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gc.c:435 [inlined]
ijl_gc_collect at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gc.c:3545
maybe_collect at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gc.c:937 [inlined]
jl_gc_pool_alloc_inner at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gc.c:1293
jl_gc_pool_alloc_noinline at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gc.c:1350 [inlined]
jl_gc_alloc_ at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/julia_internal.h:478 [inlined]
jl_gc_alloc at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gc.c:3576
ijl_alloc_svec_uninit at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/simplevector.c:63
ijl_alloc_svec at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/simplevector.c:73
jl_type_intersection_env_s at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/subtype.c:4330
jl_type_intersection_env at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/subtype.c:4344 [inlined]
ijl_type_intersection_with_env at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/subtype.c:4357
make_typealias at ./show.jl:617
show_typealias at ./show.jl:793
_show_type at ./show.jl:958
show at ./show.jl:953
jfptr_show_50630.1 at /home/navidcy/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:3077
show_typeparams at ./show.jl:710
show_typealias at ./show.jl:743
show_typealias at ./show.jl:796
_show_type at ./show.jl:958
show at ./show.jl:953
jfptr_show_50630.1 at /home/navidcy/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:3077
show_typeparams at ./show.jl:710
show_datatype at ./show.jl:1130
show_datatype at ./show.jl:1077 [inlined]
_show_type at ./show.jl:961
show at ./show.jl:953
jfptr_show_50630.1 at /home/navidcy/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:3077
show_typeparams at ./show.jl:710
show_datatype at ./show.jl:1130
show_datatype at ./show.jl:1077 [inlined]
_show_type at ./show.jl:961
show at ./show.jl:953
jfptr_show_50630.1 at /home/navidcy/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:3077
show_typeparams at ./show.jl:710
show_datatype at ./show.jl:1130
show_datatype at ./show.jl:1077 [inlined]
_show_type at ./show.jl:961
show at ./show.jl:953
jfptr_show_50630.1 at /home/navidcy/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:3077
show_at_namedtuple at ./show.jl:1143
unknown function (ip: 0x7fdb82d810d0)
_jl_invoke at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:3077
show_datatype at ./show.jl:1116
show_datatype at ./show.jl:1077 [inlined]
_show_type at ./show.jl:961
show at ./show.jl:953
jfptr_show_50630.1 at /home/navidcy/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:3077
show_typeparams at ./show.jl:710
show_datatype at ./show.jl:1130
show_datatype at ./show.jl:1077 [inlined]
_show_type at ./show.jl:961
show at ./show.jl:953
jfptr_show_50630.1 at /home/navidcy/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:3077
show_typeparams at ./show.jl:710
show_datatype at ./show.jl:1130
show_datatype at ./show.jl:1077 [inlined]
_show_type at ./show.jl:961
show at ./show.jl:953
jfptr_show_50630.1 at /home/navidcy/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:3077
show_typeparams at ./show.jl:710
show_datatype at ./show.jl:1130
show_datatype at ./show.jl:1077 [inlined]
_show_type at ./show.jl:961
show at ./show.jl:953
jfptr_show_50630.1 at /home/navidcy/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:3077
#sprint#501 at ./strings/io.jl:112
sprint at ./strings/io.jl:107 [inlined]
#print_type_bicolor#564 at ./show.jl:2672 [inlined]
print_type_bicolor at ./show.jl:2671
jfptr_print_type_bicolor_42693.1 at /home/navidcy/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:3077
#show_tuple_as_call#558 at ./show.jl:2539
show_tuple_as_call at ./show.jl:2507 [inlined]
show_spec_sig at ./stacktraces.jl:260
show_spec_linfo at ./stacktraces.jl:232
print_stackframe at ./errorshow.jl:742
print_stackframe at ./errorshow.jl:706
#show_full_backtrace#942 at ./errorshow.jl:605
show_full_backtrace at ./errorshow.jl:598 [inlined]
show_backtrace at ./errorshow.jl:802
#showerror#919 at ./errorshow.jl:97
showerror at ./errorshow.jl:93
unknown function (ip: 0x7fdb82d7dd94)
_jl_invoke at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:3077
#showerror#920 at ./errorshow.jl:103
showerror at ./errorshow.jl:101
unknown function (ip: 0x7fdb82d7a234)
_jl_invoke at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:3077
show_exception_stack at ./errorshow.jl:975
display_error at ./client.jl:116
jfptr_display_error_82800.1 at /home/navidcy/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:3077
jl_apply at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
jl_f__call_latest at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/builtins.c:812
#invokelatest#2 at ./essentials.jl:892 [inlined]
invokelatest at ./essentials.jl:889 [inlined]
repl_display_error at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/REPL/src/REPL.jl:293
print_response at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/REPL/src/REPL.jl:310
#57 at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/REPL/src/REPL.jl:284
jfptr_YY.57_91705.1 at /home/navidcy/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:3077
with_repl_linfo at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/REPL/src/REPL.jl:569
jfptr_with_repl_linfo_91780.1 at /home/navidcy/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:3077
print_response at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/REPL/src/REPL.jl:282
do_respond at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/REPL/src/REPL.jl:911
jfptr_do_respond_91853.1 at /home/navidcy/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:3077
jl_apply at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
jl_f__call_latest at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/builtins.c:812
#invokelatest#2 at ./essentials.jl:892 [inlined]
invokelatest at ./essentials.jl:889 [inlined]
run_interface at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/REPL/src/LineEdit.jl:2656
jfptr_run_interface_90592.1 at /home/navidcy/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:3077
run_frontend at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/REPL/src/REPL.jl:1312
#62 at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/REPL/src/REPL.jl:386
jfptr_YY.62_91756.1 at /home/navidcy/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/gf.c:3077
jl_apply at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
start_task at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-10/src/task.c:1256
WARNING: Error while freeing DeviceMemory(10.938 MiB at 0x0000000656528e00):
CUDA.CuError(code=CUDA.cudaError_enum(0x000002bc))

Stacktrace:
  [1] throw_api_error(res::CUDA.cudaError_enum)
    @ CUDA /storage3/navidcy/.julia/packages/CUDA/ja0IX/lib/cudadrv/libcuda.jl:30
  [2] check
    @ /storage3/navidcy/.julia/packages/CUDA/ja0IX/lib/cudadrv/libcuda.jl:37 [inlined]
  [3] cuMemFreeAsync
    @ /storage3/navidcy/.julia/packages/GPUToolbox/cZlg7/src/ccalls.jl:33 [inlined]
  [4] free(mem::CUDA.DeviceMemory; stream::CUDA.CuStream)
    @ CUDA /storage3/navidcy/.julia/packages/CUDA/ja0IX/lib/cudadrv/memory.jl:87
  [5] free
    @ /storage3/navidcy/.julia/packages/CUDA/ja0IX/lib/cudadrv/memory.jl:82 [inlined]
  [6] #1135
    @ /storage3/navidcy/.julia/packages/CUDA/ja0IX/src/memory.jl:714 [inlined]
  [7] #context!#1025
    @ /storage3/navidcy/.julia/packages/CUDA/ja0IX/lib/cudadrv/state.jl:168 [inlined]
  [8] context!
    @ /storage3/navidcy/.julia/packages/CUDA/ja0IX/lib/cudadrv/state.jl:163 [inlined]
  [9] _pool_free
    @ /storage3/navidcy/.julia/packages/CUDA/ja0IX/src/memory.jl:713 [inlined]
 [10] macro expansion
    @ ./timing.jl:395 [inlined]
 [11] pool_free(managed::CUDA.Managed{CUDA.DeviceMemory})
    @ CUDA /storage3/navidcy/.julia/packages/CUDA/ja0IX/src/memory.jl:695
 [12] release(::GPUArrays.RefCounted{CUDA.Managed{CUDA.DeviceMemory}})
    @ GPUArrays /storage3/navidcy/.julia/packages/GPUArrays/uiVyU/src/host/abstractarray.jl:42
 [13] unsafe_free!
    @ /storage3/navidcy/.julia/packages/GPUArrays/uiVyU/src/host/abstractarray.jl:100 [inlined]
 [14] unsafe_free!(x::CUDA.CuArray{Float32, 1, CUDA.DeviceMemory})
    @ GPUArrays /storage3/navidcy/.julia/packages/GPUArrays/uiVyU/src/host/abstractarray.jl:115
 [15] make_typealias(x::Type)
    @ Base ./show.jl:617
 [16] show_typealias(io::IOContext{IOBuffer}, x::Type)
    @ Base ./show.jl:793
 [17] _show_type(io::IOContext{IOBuffer}, x::Type)
    @ Base ./show.jl:958
 [18] show(io::IOContext{IOBuffer}, x::Type)
    @ Base ./show.jl:953
 [19] show_typeparams(io::IOContext{IOBuffer}, env::Core.SimpleVector, orig::Core.SimpleVector, wheres::Vector{TypeVar})
    @ Base ./show.jl:710
 [20] show_typealias(io::IOContext{IOBuffer}, name::GlobalRef, x::Type, env::Core.SimpleVector, wheres::Vector{TypeVar})
    @ Base ./show.jl:743
 [21] show_typealias(io::IOContext{IOBuffer}, x::Type)
    @ Base ./show.jl:796
 [22] _show_type(io::IOContext{IOBuffer}, x::Type)
    @ Base ./show.jl:958
 [23] show(io::IOContext{IOBuffer}, x::Type)
    @ Base ./show.jl:953
 [24] show_typeparams(io::IOContext{IOBuffer}, env::Core.SimpleVector, orig::Core.SimpleVector, wheres::Vector{TypeVar})
    @ Base ./show.jl:710
 [25] show_datatype(io::IOContext{IOBuffer}, x::DataType, wheres::Vector{TypeVar})
    @ Base ./show.jl:1130
 [26] show_datatype
    @ ./show.jl:1077 [inlined]
 [27] _show_type(io::IOContext{IOBuffer}, x::Type)
    @ Base ./show.jl:961
--- the last 5 lines are repeated 2 more times ---
 [38] show(io::IOContext{IOBuffer}, x::Type)
    @ Base ./show.jl:953
 [39] show_at_namedtuple(io::IOContext{IOBuffer}, syms::NTuple{11, Symbol}, types::DataType)
    @ Base ./show.jl:1143
 [40] show_datatype(io::IOContext{IOBuffer}, x::DataType, wheres::Vector{TypeVar})
    @ Base ./show.jl:1116
 [41] show_datatype
    @ ./show.jl:1077 [inlined]
 [42] _show_type(io::IOContext{IOBuffer}, x::Type)
    @ Base ./show.jl:961
 [43] show(io::IOContext{IOBuffer}, x::Type)
    @ Base ./show.jl:953
 [44] show_typeparams(io::IOContext{IOBuffer}, env::Core.SimpleVector, orig::Core.SimpleVector, wheres::Vector{TypeVar})
    @ Base ./show.jl:710
 [45] show_datatype(io::IOContext{IOBuffer}, x::DataType, wheres::Vector{TypeVar})
    @ Base ./show.jl:1130
--- the last 5 lines are repeated 2 more times ---
 [56] show_datatype
    @ ./show.jl:1077 [inlined]
 [57] _show_type(io::IOContext{IOBuffer}, x::Type)
    @ Base ./show.jl:961
 [58] show(io::IOContext{IOBuffer}, x::Type)
    @ Base ./show.jl:953
 [59] sprint(f::Function, args::Type; context::IOContext{IOBuffer}, sizehint::Int64)
    @ Base ./strings/io.jl:112
 [60] sprint
    @ ./strings/io.jl:107 [inlined]
 [61] #print_type_bicolor#564
    @ ./show.jl:2672 [inlined]
 [62] show_tuple_as_call(out::IOContext{Base.TTY}, name::Symbol, sig::Type; demangle::Bool, kwargs::Base.Iterators.Zip{Tuple{Vector{Symbol}, Vector{Any}}}, argnames::Vector{Symbol}, qualified::Bool, hasfirst::Bool)
    @ Base ./show.jl:2539
 [63] show_tuple_as_call
    @ ./show.jl:2507 [inlined]
 [64] show_spec_sig(io::IOContext{Base.TTY}, m::Method, sig::Type)
    @ Base.StackTraces ./stacktraces.jl:260
 [65] show_spec_linfo(io::IOContext{Base.TTY}, frame::Base.StackTraces.StackFrame)
    @ Base.StackTraces ./stacktraces.jl:232
 [66] print_stackframe(io::IOContext{Base.TTY}, i::Int64, frame::Base.StackTraces.StackFrame, n::Int64, ndigits_max::Int64, modulecolor::Symbol)
    @ Base ./errorshow.jl:742
 [67] print_stackframe(io::IOContext{Base.TTY}, i::Int64, frame::Base.StackTraces.StackFrame, n::Int64, ndigits_max::Int64, modulecolordict::IdDict{Module, Symbol}, modulecolorcycler::Base.Iterators.Stateful{Base.Iterators.Cycle{Vector{Symbol}}, Union{Nothing, Tuple{Symbol, Int64}}, Int64})
    @ Base ./errorshow.jl:706
 [68] show_full_backtrace(io::IOContext{Base.TTY}, trace::Vector{Any}; print_linebreaks::Bool)
    @ Base ./errorshow.jl:605
 [69] show_full_backtrace
    @ ./errorshow.jl:598 [inlined]
 [70] show_backtrace(io::IOContext{Base.TTY}, t::Vector{Base.StackTraces.StackFrame})
    @ Base ./errorshow.jl:802
 [71] showerror(io::IOContext{Base.TTY}, ex::CUDA.CuError, bt::Vector{Base.StackTraces.StackFrame}; backtrace::Bool)
    @ Base ./errorshow.jl:97
 [72] showerror(io::IOContext{Base.TTY}, ex::LoadError, bt::Vector{Base.StackTraces.StackFrame}; backtrace::Bool)
    @ Base ./errorshow.jl:103
 [73] show_exception_stack(io::IOContext{Base.TTY}, stack::Base.ExceptionStack)
    @ Base ./errorshow.jl:975
 [74] display_error(io::IOContext{Base.TTY}, stack::Base.ExceptionStack)
    @ Base ./client.jl:116
 [75] #invokelatest#2
    @ ./essentials.jl:892 [inlined]
 [76] invokelatest
    @ ./essentials.jl:889 [inlined]
 [77] repl_display_error(errio::IO, errval::Any)
    @ REPL ~/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/share/julia/stdlib/v1.10/REPL/src/REPL.jl:293
 [78] print_response(errio::IO, response::Any, show_value::Bool, have_color::Bool, specialdisplay::Union{Nothing, AbstractDisplay})
    @ REPL ~/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/share/julia/stdlib/v1.10/REPL/src/REPL.jl:310
 [79] (::REPL.var"#57#58"{REPL.LineEditREPL, Pair{Any, Bool}, Bool, Bool})(io::Any)
    @ REPL ~/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/share/julia/stdlib/v1.10/REPL/src/REPL.jl:284
 [80] with_repl_linfo(f::Any, repl::REPL.LineEditREPL)
    @ REPL ~/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/share/julia/stdlib/v1.10/REPL/src/REPL.jl:569
 [81] print_response(repl::REPL.AbstractREPL, response::Any, show_value::Bool, have_color::Bool)
    @ REPL ~/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/share/julia/stdlib/v1.10/REPL/src/REPL.jl:282
 [82] (::REPL.var"#do_respond#80"{Bool, Bool, REPL.var"#93#103"{REPL.LineEditREPL, REPL.REPLHistoryProvider}, REPL.LineEditREPL, REPL.LineEdit.Prompt})(s::REPL.LineEdit.MIState, buf::Any, ok::Bool)
    @ REPL ~/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/share/julia/stdlib/v1.10/REPL/src/REPL.jl:911
 [83] #invokelatest#2
    @ ./essentials.jl:892 [inlined]
 [84] invokelatest
    @ ./essentials.jl:889 [inlined]
 [85] run_interface(terminal::REPL.Terminals.TextTerminal, m::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState)
    @ REPL.LineEdit ~/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/share/julia/stdlib/v1.10/REPL/src/LineEdit.jl:2656
 [86] run_frontend(repl::REPL.LineEditREPL, backend::REPL.REPLBackendRef)
    @ REPL ~/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/share/julia/stdlib/v1.10/REPL/src/REPL.jl:1312
time_step!(coupled_model::OceanSeaIceModel{…}, Δt::Float64; callbacks::Vector{…}, compute_tendencies::Bool)
    @ ClimaOcean.OceanSeaIceModels ~/ClimaOcean.jl/src/OceanSeaIceModels/time_step_ocean_sea_ice_model.jl:26
 [32] time_step!(coupled_model::OceanSeaIceModel{…}, Δt::Float64)
    @ ClimaOcean.OceanSeaIceModels ~/ClimaOcean.jl/src/OceanSeaIceModels/time_step_ocean_sea_ice_model.jl:13
 [33] top-level scope
    @ ~/ClimaOcean.jl/examples/mwe_clock_sync.jl:49
 [34] include(fname::String)
    @ Base.MainInclude ./client.jl:494
 [35] top-level scope
    @ REPL[2]:1
in expression starting at /home/navidcy/ClimaOcean.jl/examples/mwe_clock_sync.jl:47
Some type information was truncated. Use `show(err)` to see complete types.

julia> Unhandled Task ERROR: CUDA error: an illegal memory access was encountered (code 700, ERROR_ILLEGAL_ADDRESS)
Stacktrace:
  [1] throw_api_error(res::CUDA.cudaError_enum)
    @ CUDA /storage3/navidcy/.julia/packages/CUDA/ja0IX/lib/cudadrv/libcuda.jl:30
  [2] isdone
    @ /storage3/navidcy/.julia/packages/CUDA/ja0IX/lib/cudadrv/stream.jl:134 [inlined]
  [3] spinning_synchronization(f::typeof(CUDA.isdone), obj::CUDA.CuStream)
    @ CUDA /storage3/navidcy/.julia/packages/CUDA/ja0IX/lib/cudadrv/synchronization.jl:79
  [4] device_synchronize(; blocking::Bool, spin::Bool)
    @ CUDA /storage3/navidcy/.julia/packages/CUDA/ja0IX/lib/cudadrv/synchronization.jl:180
  [5] device_synchronize
    @ /storage3/navidcy/.julia/packages/CUDA/ja0IX/lib/cudadrv/synchronization.jl:178 [inlined]
  [6] reclaim(sz::Int64)
    @ CUDA /storage3/navidcy/.julia/packages/CUDA/ja0IX/src/memory.jl:743
  [7] reclaim
    @ /storage3/navidcy/.julia/packages/CUDA/ja0IX/src/memory.jl:738 [inlined]
  [8] #1127
    @ /storage3/navidcy/.julia/packages/CUDA/ja0IX/src/memory.jl:300 [inlined]
  [9] context!(f::CUDA.var"#1127#1128", ctx::CUDA.CuContext; skip_destroyed::Bool)
    @ CUDA /storage3/navidcy/.julia/packages/CUDA/ja0IX/lib/cudadrv/state.jl:168
 [10] context!
    @ /storage3/navidcy/.julia/packages/CUDA/ja0IX/lib/cudadrv/state.jl:163 [inlined]
 [11] device!(f::Function, dev::CUDA.CuDevice)
    @ CUDA /storage3/navidcy/.julia/packages/CUDA/ja0IX/lib/cudadrv/state.jl:300
 [12] pool_cleanup()
    @ CUDA /storage3/navidcy/.julia/packages/CUDA/ja0IX/src/memory.jl:299
 [13] (::CUDA.var"#1122#1124")()
    @ CUDA /storage3/navidcy/.julia/packages/CUDA/ja0IX/src/memory.jl:246

[navidcy]

I tried to reduce the MWE more but I wasn't getting always the error.... only sometimes.... It's quite confusing.

@glwagner, @simone-silvestri does the above hint to you to anywhere in the source code I should be looking?

[glwagner]

I think this code is supposed to load the needed atmosphere data:

ClimaOcean.jl/src/OceanSeaIceModels/PrescribedAtmospheres.jl

Lines 355 to 366 in 23f31de

	@inline function time_step!(atmos::PrescribedAtmosphere, Δt)
	tick!(atmos.clock, Δt)
	time = Time(atmos.clock.time)
	ftses = extract_field_time_series(atmos)
	for fts in ftses
	update_field_time_series!(fts, time)
	end
	return nothing
	end

so maybe that is being called too late, or not at all --- not sure. Maybe update_field_time_series should be called in update_state! rather than time_step!?

[simone-silvestri]

maybe trying on the CPU with --check-bounds=yes will help isolate the problem

[taimoorsohail]

Thanks all, @simone-silvestri clarified that when moving the simulation clock to a different time, the atmosphere needs to be updated using time_step!, and so the following must be done:

simulation.model.ocean.model.clock.iteration = iteration
simulation.model.ocean.model.clock.time = time
simulation.model.atmosphere.clock.iteration = iteration
simulation.model.atmosphere.clock.time = time
simulation.model.clock.iteration = iteration
simulation.model.clock.time = time
time_step!(atmosphere, 0)
simulation.model.atmosphere.clock.iteration -= 1
simulation.model.ocean.model.clock.last_Δt = last_Δt

where iteration, time, last_∆t are all user defined variables (presumably read from a checkpoint file)

[navidcy]

To elaborate a bit more: What's essentially needed from the time_step! method is the part of updating the atmosphere, i.e.,

ClimaOcean.jl/src/OceanSeaIceModels/PrescribedAtmospheres.jl

Lines 358 to 363 in 8e26abd

	time = Time(atmos.clock.time)
	ftses = extract_field_time_series(atmos)
	for fts in ftses
	update_field_time_series!(fts, time)
	end

#590 exposes that functionality so that we don't need to do the what-seems-as-bit strange

time_step!(atmosphere, 0)
simulation.model.atmosphere.clock.iteration -= 1
simulation.model.ocean.model.clock.last_Δt = last_Δt

that requires us to take the iteration back because time_step!(atmosphere, 0) ticked the clock.

Instead, one simply needs to do:

update_state!(atmosphere)