Use DirectIO for sampling

[qizhou]

DirectIO will bypass OS buffer, where buffering is unnecessary in our sampling process. Further, it seems that DirectIO will be faster on some FS such as APFS. E.g., using 32 threads sampling on APFS, DirectIO can yield 1GB/s randread (actual disk IO is 1GB/s), while buffer IO will yield 500GB/s (actual disk IO is 2GB/s, no sure where the overhead comes from).

[qizhou]

A library for Go can be found https://github.com/ncw/directio

[syntrust]

A quick test with Macos (16GB RAM) on an external disk (APFS)
Shard size: 256GB

io type	threads	noncesTried	iostat(MB/s)
non-directio	24	162096(15.5%)	105
directio	24	78888(7.5%)	99.79

Trying to do more research and test on the reason why this is counter-intuitive.

[syntrust]

DirectIO will bypass OS buffer, where buffering is unnecessary in our sampling process. Further, it seems that DirectIO will be faster on some FS such as APFS. E.g., using 32 threads sampling on APFS, DirectIO can yield 1GB/s randread (actual disk IO is 1GB/s), while buffer IO will yield 500GB/s (actual disk IO is 2GB/s, no sure where the overhead comes from).

Hi Qi, what exactly is the fio command used in this test?

[qzhodl]

A quick test with Macos (16GB RAM) on an external disk (APFS) Shard size: 256GB

io type threads noncesTried iostat(MB/s)
non-directio 24 162096(15.5%) 105
directio 24 78888(7.5%) 99.79
Trying to do more research and test on the reason why this is counter-intuitive.

It will be great to include the testing result that fio directIO/no-directIO ran (also the fio command you were using) with the same config in your test env

[qzhodl]

A quick test with Macos (16GB RAM) on an external disk (APFS) Shard size: 256GB

io type threads noncesTried iostat(MB/s)
non-directio 24 162096(15.5%) 105
directio 24 78888(7.5%) 99.79
Trying to do more research and test on the reason why this is counter-intuitive.

Also you'd better clarify the testing result was based on the PR (#121), right?

[qizhou]

Running on my Mac M2 with APFS + USBC 4.0 + Fanxiang S880

• write 128GB test file
  fio --name=random-write --ioengine=psync --rw=write --bs=1m --size=4g --numjobs=32 --iodepth=1 --thread --offset_increment=4g --filename fio.dat --direct=1

• read with 64 threads (direct=1)
  fio --name=random-write --ioengine=psync --rw=randread --bs=4k --size=2g --numjobs=64 --iodepth=1 --thread --offset_increment=2g --filename fio.dat --direct=1

fio reports 1.2GB/s, activity monitor reports 1.2GB/s

• read with 64 threads (direct=0)
  fio --name=random-write --ioengine=psync --rw=randread --bs=4k --size=2g --numjobs=64 --iodepth=1 --thread --offset_increment=2g --filename fio.dat --direct=0

fio reports 700MB/s, activity monitor reports 2.6GB/s

[qizhou]

The performance is closely related to the USB interface. For example, using the above commands with Samsung 990 Pro, I have

• USBC 3.2 yields 141 MB/s
• USBC 3.1 yields 27.2MB/s

Using another USBC4.0 + 990 Pro, I have

• fio reports 1050 MB/s (direct=1) and activity monitor reports 900 MB/s
• fio reports 641 MB/s (direct=0) and activity monitor reports 2.17 GB/s

[syntrust]

Test with exactly fio commands with the following hardware:

• MacBookPro 2.9 GHz 6-Core Intel Core i9

• fanxiang PS2000 USB 3.2

read with 64 threads (direct=1)

• fio reports 138MB/s

• activity monitor reports 141MB/s

read with 64 threads (direct=0)

• fio reports 140MB/s

• activity monitor reports 140MB/s

No big difference has been observed. Could it be that the USB 3.2 capacity has already been reached?

[syntrust]

Another round of tests with local APPLE SSD(APFS) and a file of size 256GB, with the same commands.

read with 64 threads (direct=1)

• fio reports 603MB/s

• activity monitor reports 621MB/s

read with 64 threads (direct=0)

• fio reports 541MB/s

• activity monitor reports 578MB/s

Could see a little better with direct=1.