Read scan fails with verify: bad magic header 0, wanted acca at.... when encountering unwritten device locations

[blshep]

I am looking for a way to efficiently verify written locations after some writes are not verified due to using runtime=. I have been able to minimize the number of unchecked writes by using verify_backlog=xx, but this changes the behavior of a random write or sequential write workload at the device level. Ideally, it would be good for runtime= to stop issuing IO when that time expires, but allow do_verify=1 to completely check written data before ending. Anyway to accomplish that?

The drives are zeroed before starting any IO workloads. When doing writes, I am using verify=md5, verify_interval=4096 or 512 (depending on the device's block size) which puts the fio header in every drive sector. I would like to periodically read scan sections of the drive to verify previously fio written data are in the correct locations and CRCs are correct. However, when a non-written sector is encountered and all zeroes are read, fio stops checking due to a missing acca header. This error happens when using verify_fatal=0 also. I tried using/not using verify_only with the same result. Is there a way to ignore non-fio written locations short of "initializing" the drive with fio writes before starting test?

The error seen is....

job1: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.19
Starting 1 process
verify: bad magic header 0, wanted acca at file /dev/sdaf offset 52428800, length 4096 (requested block: offset=52428800, length=4096)
       hdr_fail data dumped as sdaf.52428800.hdr_fail
fio: pid=58492, err=84/file:io_u.c:2086, func=io_u_queued_complete, error=Invalid or incomplete multibyte or wide character

job1: (groupid=0, jobs=1): err=84 (file:io_u.c:2086, func=io_u_queued_complete, error=Invalid or incomplete multibyte or wide character): pid=58492: Thu Aug 14 12:09:15 2025
  read: IOPS=12.2k, BW=47.9MiB/s (50.2MB/s)(50.0MiB/1045msec)
    slat (nsec): min=10494, max=69738, avg=11066.25, stdev=911.68
    clat (usec): min=42, max=409, avg=47.18, stdev= 3.39
     lat (usec): min=54, max=421, avg=59.27, stdev= 3.57
    clat percentiles (usec):
     |  1.00th=[   46],  5.00th=[   47], 10.00th=[   47], 20.00th=[   47],
     | 30.00th=[   47], 40.00th=[   47], 50.00th=[   48], 60.00th=[   48],
     | 70.00th=[   48], 80.00th=[   48], 90.00th=[   48], 95.00th=[   48],
     | 99.00th=[   49], 99.50th=[   49], 99.90th=[   65], 99.95th=[   68],
     | 99.99th=[  135]
   bw (  KiB/s): min=49070, max=49070, per=100.00%, avg=49070.00, stdev= 0.00, samples=1
   iops        : min=12267, max=12267, avg=12267.00, stdev= 0.00, samples=1
  lat (usec)   : 50=99.77%, 100=0.21%, 250=0.01%, 500=0.01%
  cpu          : usr=22.80%, sys=28.35%, ctx=12804, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=12801,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=47.9MiB/s (50.2MB/s), 47.9MiB/s-47.9MiB/s (50.2MB/s-50.2MB/s), io=50.0MiB (52.4MB), run=1045-1045msec

Disk stats (read/write):
  sdaf: ios=11465/0, merge=0/0, ticks=563/0, in_queue=564, util=61.84%

The last "readscan" invocation of the jobfile was….

[global]
name=timed_write_verify_log
#filename=testfile
#offset=1G
size=60M
iodepth=1
verify_fatal=0
write_iolog=io_sequence3.log
log_avg_msec=1
#status-interval=1
bs=4K
direct=1
rw=read
ioengine=libaio
#runtime=3
#time_based=1
verify=md5
do_verify=1
verify_interval=4096
verify_dump=1
#verify_only
#verify_backlog=50
#norandommap
#random_generator=tausworthe64
nice=19
prio=0
prioclass=1
#size=100%
[job1]
filename=/dev/sdaf
new_group
#[job2]
#filename=/dev/sdk
#new_group

[sitsofe]

Hello @blshep:

Ideally, it would be good for runtime= to stop issuing IO when that time expires, but allow do_verify=1 to completely check written data before ending. Anyway to accomplish that?

To do exactly that (have a verify phase that ran after "time ran out") would be brand new work and if someone shows up with such a patch/pull request it could be reviewed for inclusion. With fio as it stands if you have two (separate) invocations couldn't the second verify what the first did by using verify_state_load to know where to stop?

However, when a non-written sector is encountered and all zeroes are read, fio stops checking due to a missing acca header. This error happens when using verify_fatal=0 also.

First up verify_fatal already defaults to off/0. Next, verify_fatal=0 won't stop errors being reported nor will it stop fio ultimately aborting - it just controls whether fio will stop at first error seen within a block. For example, if you have --bs=4k --verify_interval=2k --verify_pattern=0xff --verify=pattern then technically two verifications take place per block. With verify_fatal=0 even if the first part of the block doesn't verify the second part of the block is still checked (but after verification of the block finishes fio will still bail out with an error if any portion failed). In the previous example verify_fatal=1 just means fio will bail out after if it finds the first part of the block is wrong. Perhaps you were thinking of continue_on_error/ignore_error but if you're going to ignore all verification errors why do the verification in the first place?

I tried using/not using verify_only with the same result. Is there a way to ignore non-fio written locations short of "initializing" the drive with fio writes before starting test?

Are you saying the write job was random? If so, your "read" verification job would also have to be random and a verification state file loaded so fio regenerates the same sequence of blocks and knows the correct point to stop. The following works for me:

% dd if=/dev/zero of=/tmp/fio.tmp bs=64k count=1 2> /dev/null
 % ./fio --name=scattered --verify_pattern=0xff --verify=pattern --rw=randwrite --filename=/tmp/fio.tmp --verify_state_save=1 --rate_iops=4 --runtime=1
[...]
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,4,0,0 short=0,0,0,0 dropped=0,0,0,0
[...]
% hexdump /tmp/fio.tmp
0000000 0000 0000 0000 0000 0000 0000 0000 0000
*
0008000 ffff ffff ffff ffff ffff ffff ffff ffff
*
0009000 0000 0000 0000 0000 0000 0000 0000 0000
*
000a000 ffff ffff ffff ffff ffff ffff ffff ffff
*
000b000 0000 0000 0000 0000 0000 0000 0000 0000
*
000d000 ffff ffff ffff ffff ffff ffff ffff ffff
*
000e000 0000 0000 0000 0000 0000 0000 0000 0000
*
000f000 ffff ffff ffff ffff ffff ffff ffff ffff
*
0010000
% ./fio --name=scattered --verify_pattern=0xff --verify=pattern --rw=randread --filename=/tmp/fio.tmp --verify_state_save=0 --verify_state_load=1
scattered: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.40-58-g7664d-dirty
Starting 1 process
Stop verify because seq 4 >= 4
Stop verify because seq 4 >= 4

scattered: (groupid=0, jobs=1): err= 0: pid=38501: Mon Aug 18 20:41:50 2025
[...]
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
[...]

Please also see @vincentkfu's excellent blog post about fio verification.

Is there a way to ignore non-fio written locations short of "initializing" the drive with fio writes before starting test?

That depends on the type of job. If memory serves, inline verification of something like a rw=randrw job will not trigger verification of those points that fio only read but never wrote. If you mean you mean:

• Do a write invocation that doesn't fill the whole file
• Do a (separate) read invocation that tries to to verify the whole file (rather than "replaying" the workload you previously did as reads)

Then the answer is no, there's no way to currently do that. Does this answer your questions?