boot image mmap overwrites the JikesRVM executable mapping #160
Description
The phenomenon is that the process crashes almost immediately after invoking the rvm command. It is very easy to reproduce when using rr. (Note that you need to compile rr from source in order to make rr support the 32-bit x86 architecture on 64-bit OS. Compiling the git head using the default configuration should just work, but the rr from your Linux distro may not have 32-bit support.)
Example 1
In one particular execution, the executable JikesRVM was mapped near address 0x61b4????.
(rr) info proc mappings
process 128175
Mapped address spaces:
Start Addr End Addr Size Offset Perms objfile
0x61b43000 0x61b44000 0x1000 0x0 r--p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/JikesRVM
0x61b44000 0x61b46000 0x2000 0x1000 r-xp /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/JikesRVM
0x61b46000 0x61b48000 0x2000 0x3000 r--p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/JikesRVM
0x61b48000 0x61b49000 0x1000 0x4000 r--p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/JikesRVM
0x61b49000 0x61b4a000 0x1000 0x5000 rw-p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/JikesRVM
0x62287000 0x622a9000 0x22000 0x0 rw-p [heap]
0x68000000 0x68200000 0x200000 0x0 rwxp
0x6fffd000 0x70001000 0x4000 0x0 r-xp /home/wks/tocompile/rr/build-release/install/lib/rr/librrpage_32.so
0x70001000 0x70002000 0x1000 0x0 rw-s /tmp/rr-shared-preload_thread_locals-131561-25
0x70003000 0x70004000 0x1000 0x0 r-xp
0x70004000 0x70104000 0x100000 0x0 rw-s /tmp/rr-shared-syscallbuf.128175-131561-28
0xe8400000 0xe841e000 0x1e000 0x0 r--p /usr/lib32/libc.so.6
0xe841e000 0xe85a0000 0x182000 0x1e000 r-xp /usr/lib32/libc.so.6
0xe85a0000 0xe861b000 0x7b000 0x1a0000 r--p /usr/lib32/libc.so.6
0xe861b000 0xe861d000 0x2000 0x21a000 r--p /usr/lib32/libc.so.6
0xe861d000 0xe861e000 0x1000 0x21c000 rw-p /usr/lib32/libc.so.6
0xe861e000 0xe8623000 0x5000 0x0 rw-p
0xe8800000 0xe881b000 0x1b000 0x0 r--p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libmmtk.so
0xe881b000 0xe89ce000 0x1b3000 0x1b000 r-xp /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libmmtk.so
0xe89ce000 0xe8a9c000 0xce000 0x1ce000 r--p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libmmtk.so
0xe8a9c000 0xe8a9d000 0x1000 0x29c000 rw-p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libmmtk.so
0xe8b2c000 0xe8b2e000 0x2000 0x0 rw-p
0xe8b2e000 0xe8b31000 0x3000 0x0 r--p /usr/lib32/libgcc_s.so.1
0xe8b31000 0xe8b50000 0x1f000 0x3000 r-xp /usr/lib32/libgcc_s.so.1
0xe8b50000 0xe8b56000 0x6000 0x22000 r--p /usr/lib32/libgcc_s.so.1
0xe8b56000 0xe8b57000 0x1000 0x28000 rw-p /usr/lib32/libgcc_s.so.1
0xe8b57000 0xe8b62000 0xb000 0x0 r--p /usr/lib32/libm.so.6
0xe8b62000 0xe8bf0000 0x8e000 0xb000 r-xp /usr/lib32/libm.so.6
0xe8bf0000 0xe8c27000 0x37000 0x99000 r--p /usr/lib32/libm.so.6
0xe8c27000 0xe8c28000 0x1000 0xcf000 r--p /usr/lib32/libm.so.6
0xe8c28000 0xe8c29000 0x1000 0xd0000 rw-p /usr/lib32/libm.so.6
0xe8c29000 0xe8c2c000 0x3000 0x0 r--p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libjvm.so
0xe8c2c000 0xe8c39000 0xd000 0x3000 r-xp /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libjvm.so
0xe8c39000 0xe8c40000 0x7000 0x10000 r--p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libjvm.so
0xe8c40000 0xe8c41000 0x1000 0x16000 r--p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libjvm.so
0xe8c41000 0xe8c42000 0x1000 0x17000 rw-p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libjvm.so
0xe8c42000 0xe8c43000 0x1000 0x0 r--p /usr/lib32/libdl.so.2
0xe8c43000 0xe8c44000 0x1000 0x1000 r-xp /usr/lib32/libdl.so.2
0xe8c44000 0xe8c45000 0x1000 0x2000 r--p /usr/lib32/libdl.so.2
0xe8c45000 0xe8c46000 0x1000 0x2000 r--p /usr/lib32/libdl.so.2
0xe8c46000 0xe8c47000 0x1000 0x3000 rw-p /usr/lib32/libdl.so.2
0xe8c8c000 0xe8c8d000 0x1000 0x0 r--p /home/wks/tocompile/rr/build-release/install/lib/rr/librrpreload_32.so
0xe8c8d000 0xe8c94000 0x7000 0x1000 r-xp /home/wks/tocompile/rr/build-release/install/lib/rr/librrpreload_32.so
0xe8c94000 0xe8c98000 0x4000 0x8000 r--p /home/wks/tocompile/rr/build-release/install/lib/rr/librrpreload_32.so
0xe8c98000 0xe8c99000 0x1000 0xc000 rw-p /home/wks/tocompile/rr/build-release/install/lib/rr/librrpreload_32.so
0xe8c99000 0xe8ca4000 0xb000 0x0 rw-p
0xe8ca4000 0xe8ca6000 0x2000 0x0 r-xp [vdso]
0xe8ca6000 0xe8ca7000 0x1000 0x0 r--p /usr/lib32/ld-linux.so.2
0xe8ca7000 0xe8cca000 0x23000 0x1000 r-xp /usr/lib32/ld-linux.so.2
0xe8cca000 0xe8cd8000 0xe000 0x24000 r--p /usr/lib32/ld-linux.so.2
0xe8cd8000 0xe8cda000 0x2000 0x31000 r--p /usr/lib32/ld-linux.so.2
0xe8cda000 0xe8cdb000 0x1000 0x33000 rw-p /usr/lib32/ld-linux.so.2
0xff98f000 0xff9b2000 0x23000 0x0 rw-p [stack]
Then JikesRVM executed the mmap syscall in mapImageFile to map RVM.data.image. The stack trace is:
(rr) bt
#0 0x70000002 in syscall_traced ()
#1 0xe8c929ad in _raw_syscall () at /home/wks/tocompile/rr/build-release/32/preload/raw_syscall.S:34
#2 0xe8c8dcd3 in traced_raw_syscall (call=call@entry=0x681fffd8) at /home/wks/tocompile/rr/build-release/32/preload/syscallbuf.c:377
#3 0xe8c905e7 in sys_quotactl (call=0x681fffd8) at /home/wks/tocompile/rr/build-release/32/preload/syscallbuf.c:3710
#4 syscall_hook_internal (call=0x681fffd8) at /home/wks/tocompile/rr/build-release/32/preload/syscallbuf.c:4229
#5 syscall_hook (call=0x681fffd8) at /home/wks/tocompile/rr/build-release/32/preload/syscallbuf.c:4355
#6 syscall_hook (call=<optimized out>) at /home/wks/tocompile/rr/build-release/32/preload/syscallbuf.c:4339
#7 0xe8c8d341 in _syscall_hook_trampoline () at /home/wks/tocompile/rr/build-release/32/preload/syscall_hook.S:131
#8 0xe8c8d3b2 in _syscall_hook_trampoline_90_90_90 () at /home/wks/tocompile/rr/build-release/32/preload/syscall_hook.S:211
#9 0x6ffff005 in __kernel_vsyscall ()
#10 0xe851b366 in ?? () from /usr/lib32/libc.so.6
#11 0xe852216c in mmap64 () from /usr/lib32/libc.so.6
#12 0xe8c2d7a1 in mapImageFile (fileName=0xff9b0d0f "/home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/RVM.data.image", targetAddress=0x60000000, executable=1 '\001',
writable=1 '\001', roundedImageSize=0xff9ae8d4) at /home/wks/projects/mmtk-github/jikesrvm/tools/bootloader/jvm.c:319
#13 0xe8c2d8cf in createVM (vmInSeparateThread=0) at /home/wks/projects/mmtk-github/jikesrvm/tools/bootloader/jvm.c:359
#14 0xe8c2dffa in JNI_CreateJavaVM (mainJavaVM=0xff9ae95c, mainJNIEnv=0xff9ae960, initArgs=0xff9ae96c) at /home/wks/projects/mmtk-github/jikesrvm/tools/bootloader/jvm.c:497
#15 0x61b459a0 in main (argc=19, argv=0xff9aea68) at /home/wks/projects/mmtk-github/jikesrvm/tools/bootloader/main.c:594
After that, the mapping becomes:
(rr) info proc mappings
process 128175
Mapped address spaces:
Start Addr End Addr Size Offset Perms objfile
0x60000000 0x6218a000 0x218a000 0x0 rwxp /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/RVM.data.image
0x62287000 0x622a9000 0x22000 0x0 rw-p [heap]
0x68000000 0x68200000 0x200000 0x0 rwxp
0x6fffd000 0x70001000 0x4000 0x0 r-xp /home/wks/tocompile/rr/build-release/install/lib/rr/librrpage_32.so
0x70001000 0x70002000 0x1000 0x0 rw-s /tmp/rr-shared-preload_thread_locals-131561-25
0x70003000 0x70004000 0x1000 0x0 r-xp
0x70004000 0x70104000 0x100000 0x0 rw-s /tmp/rr-shared-syscallbuf.128175-131561-28
0xe8400000 0xe841e000 0x1e000 0x0 r--p /usr/lib32/libc.so.6
0xe841e000 0xe85a0000 0x182000 0x1e000 r-xp /usr/lib32/libc.so.6
0xe85a0000 0xe861b000 0x7b000 0x1a0000 r--p /usr/lib32/libc.so.6
0xe861b000 0xe861d000 0x2000 0x21a000 r--p /usr/lib32/libc.so.6
0xe861d000 0xe861e000 0x1000 0x21c000 rw-p /usr/lib32/libc.so.6
0xe861e000 0xe8623000 0x5000 0x0 rw-p
0xe8800000 0xe881b000 0x1b000 0x0 r--p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libmmtk.so
0xe881b000 0xe89ce000 0x1b3000 0x1b000 r-xp /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libmmtk.so
0xe89ce000 0xe8a9c000 0xce000 0x1ce000 r--p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libmmtk.so
0xe8a9c000 0xe8a9d000 0x1000 0x29c000 rw-p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libmmtk.so
0xe8b2c000 0xe8b2e000 0x2000 0x0 rw-p
0xe8b2e000 0xe8b31000 0x3000 0x0 r--p /usr/lib32/libgcc_s.so.1
0xe8b31000 0xe8b50000 0x1f000 0x3000 r-xp /usr/lib32/libgcc_s.so.1
0xe8b50000 0xe8b56000 0x6000 0x22000 r--p /usr/lib32/libgcc_s.so.1
0xe8b56000 0xe8b57000 0x1000 0x28000 rw-p /usr/lib32/libgcc_s.so.1
0xe8b57000 0xe8b62000 0xb000 0x0 r--p /usr/lib32/libm.so.6
0xe8b62000 0xe8bf0000 0x8e000 0xb000 r-xp /usr/lib32/libm.so.6
0xe8bf0000 0xe8c27000 0x37000 0x99000 r--p /usr/lib32/libm.so.6
0xe8c27000 0xe8c28000 0x1000 0xcf000 r--p /usr/lib32/libm.so.6
0xe8c28000 0xe8c29000 0x1000 0xd0000 rw-p /usr/lib32/libm.so.6
0xe8c29000 0xe8c2c000 0x3000 0x0 r--p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libjvm.so
0xe8c2c000 0xe8c39000 0xd000 0x3000 r-xp /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libjvm.so
0xe8c39000 0xe8c40000 0x7000 0x10000 r--p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libjvm.so
0xe8c40000 0xe8c41000 0x1000 0x16000 r--p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libjvm.so
0xe8c41000 0xe8c42000 0x1000 0x17000 rw-p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libjvm.so
0xe8c42000 0xe8c43000 0x1000 0x0 r--p /usr/lib32/libdl.so.2
0xe8c43000 0xe8c44000 0x1000 0x1000 r-xp /usr/lib32/libdl.so.2
0xe8c44000 0xe8c45000 0x1000 0x2000 r--p /usr/lib32/libdl.so.2
0xe8c45000 0xe8c46000 0x1000 0x2000 r--p /usr/lib32/libdl.so.2
0xe8c46000 0xe8c47000 0x1000 0x3000 rw-p /usr/lib32/libdl.so.2
0xe8c8c000 0xe8c8d000 0x1000 0x0 r--p /home/wks/tocompile/rr/build-release/install/lib/rr/librrpreload_32.so
0xe8c8d000 0xe8c94000 0x7000 0x1000 r-xp /home/wks/tocompile/rr/build-release/install/lib/rr/librrpreload_32.so
0xe8c94000 0xe8c98000 0x4000 0x8000 r--p /home/wks/tocompile/rr/build-release/install/lib/rr/librrpreload_32.so
0xe8c98000 0xe8c99000 0x1000 0xc000 rw-p /home/wks/tocompile/rr/build-release/install/lib/rr/librrpreload_32.so
0xe8c99000 0xe8ca4000 0xb000 0x0 rw-p
0xe8ca4000 0xe8ca6000 0x2000 0x0 r-xp [vdso]
0xe8ca6000 0xe8ca7000 0x1000 0x0 r--p /usr/lib32/ld-linux.so.2
0xe8ca7000 0xe8cca000 0x23000 0x1000 r-xp /usr/lib32/ld-linux.so.2
0xe8cca000 0xe8cd8000 0xe000 0x24000 r--p /usr/lib32/ld-linux.so.2
0xe8cd8000 0xe8cda000 0x2000 0x31000 r--p /usr/lib32/ld-linux.so.2
0xe8cda000 0xe8cdb000 0x1000 0x33000 rw-p /usr/lib32/ld-linux.so.2
0xff98f000 0xff9b2000 0x23000 0x0 rw-p [stack]
Note that the memory mappings for all sections in the JikesRVM executable are overwritten by RVM.data.image which spans from 0x60000000 to 0x6218a000.
It is also interesting that the mmap succeeded silently without warning about existing mmaps at the same location.
The execution went on. It eventually crashed when calling jikesrvm_alloc. That function is defined in glue.asm which is statically lined into the JikesRVM executable. The address that was supposed to hold the function body of jikesrvm_alloc is now mapped to RVM.data.image and the address contains a sequence of 0x00 bytes which may be interpreted as add %al,(%eax). The program crashed with SIGSEGV, with the message
Failing instruction offset: 0xfd11f1fe in method sysAlloc with descriptor (Lorg/vmmagic/unboxed/Address;IIII)Lorg/vmmagic/unboxed/Address; declared by class with descriptor Lorg/jikesrvm/runtime/SysCall;
Example 2
The error is non-deterministic. (I left ASLR enabled by default.) In another execution, the JikesRVM was mapped to another address:
Start Addr End Addr Size Offset Perms objfile
0x64533000 0x64534000 0x1000 0x0 r--p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/JikesRVM
0x64534000 0x64536000 0x2000 0x1000 r-xp /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/JikesRVM
0x64536000 0x64538000 0x2000 0x3000 r--p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/JikesRVM
0x64538000 0x64539000 0x1000 0x4000 r--p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/JikesRVM
0x64539000 0x6453a000 0x1000 0x5000 rw-p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/JikesRVM
0x655c5000 0x655e7000 0x22000 0x0 rw-p [heap]
0x68000000 0x68200000 0x200000 0x0 rwxp
Then it was RVM.code.image that overwrote the mappings for JikesRVM.
Start Addr End Addr Size Offset Perms objfile
0x60000000 0x6218a000 0x218a000 0x0 rwxp /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/RVM.data.image
0x64000000 0x6521e000 0x121e000 0x0 rwxp /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/RVM.code.image
0x655c5000 0x655e6000 0x21000 0x0 rw-p [heap]
0x68000000 0x68200000 0x200000 0x0 rwxp
And the process managed to execute the third mmap for RVM.rmap.image
Start Addr End Addr Size Offset Perms objfile
0x60000000 0x6218a000 0x218a000 0x0 rwxp /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/RVM.data.image
0x64000000 0x6521e000 0x121e000 0x0 rwxp /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/RVM.code.image
0x655c5000 0x655e6000 0x21000 0x0 rw-p [heap]
0x67000000 0x67074000 0x74000 0x0 r--p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/RVM.rmap.image
0x68000000 0x68200000 0x200000 0x0 rwxp
and then it crashed due to SIGSEGV.
430 if (((*hasharr ^ new_hash) >> 1) == 0)
(rr) bt
#0 do_lookup_x (undef_name=undef_name@entry=0xe8e229a2 "checkMalloc", new_hash=new_hash@entry=1550431963, old_hash=old_hash@entry=0xff955b90, ref=0xe8e2221c, result=0xff955b94, scope=0xe8ed2b9c,
i=<optimized out>, version=0xe8e9bd10, flags=1, skip=0x0, type_class=1, undef_map=0xe8e9ab30) at dl-lookup.c:430
#1 0xe8ea852e in _dl_lookup_symbol_x (undef_name=0xe8e229a2 "checkMalloc", undef_map=0xe8e9ab30, ref=0xff955c10, symbol_scope=<optimized out>, version=0xe8e9bd10, type_class=1, flags=1, skip_map=0x0)
at dl-lookup.c:793
#2 0xe8eae1d6 in _dl_fixup (l=0xe8e9ab30, reloc_arg=768) at dl-runtime.c:95
#3 0xe8eb02d4 in _dl_runtime_resolve () at ../sysdeps/i386/dl-trampoline.S:38
#4 0xe8e2f40d in sysMonitorCreate () at /home/wks/projects/mmtk-github/jikesrvm/tools/bootloader/sysThread.c:671
#5 0xe8e2e540 in sysInitialize () at /home/wks/projects/mmtk-github/jikesrvm/tools/bootloader/sysThread.c:125
#6 0xe8e25c1d in createVM (vmInSeparateThread=0) at /home/wks/projects/mmtk-github/jikesrvm/tools/bootloader/jvm.c:459
#7 0xe8e25ffa in JNI_CreateJavaVM (mainJavaVM=0xff955d2c, mainJNIEnv=0xff955d30, initArgs=0xff955d3c) at /home/wks/projects/mmtk-github/jikesrvm/tools/bootloader/jvm.c:497
#8 0x645359a0 in main (argc=19, argv=0xff955e38) at /home/wks/projects/mmtk-github/jikesrvm/tools/bootloader/main.c:594
Line 671 in sysThread.c attempts to call checkMalloc, but the stack trace shows it didn't enter checkMalloc, but it ended up crashing in the system linker/loader when trying to resolve the symbol checkMalloc. It is likely because the linking metadata was overwritten by the mmap of RVM.*.image.
How is this related to other "failing instruction" errors?
I am not sure, but it is quite likely that #108 is caused by some code sections overwritten by irrelevant mmaps.