Migrate to a Linux 6.12 kernel

[krizhanovsky]

We were living with 5.10 for too long, it's time to migrate to the 6.1 longterm kernel.

Please update all the https://github.com/tempesta-tech/tempesta/wiki pages referencing the old kernel.

Please also grep and fix all TODO #1808 comments.

[krizhanovsky]

Probably it makes sense to migrate to 6.8 or later, even not stable yet, to get the TCP performance optimizations

[kingluo]

Probably it makes sense to migrate to 6.8 or later, even not stable yet, to get the TCP performance optimizations

I have some questions:

1. 6.8 is not an LTS release so I want to double-check if it is worth developing this release.
2. We need to create a new repository just like https://github.com/tempesta-tech/linux-5.10.35-tfw, or just a new patch file?
3. Which commit is the end point where we need to migrate to the new kernel version? You know the master branch is growing every day.

[krizhanovsky]

I created new repo for the kernel https://github.com/tempesta-tech/linux-6.8.9-tfw , so please

• apply and merge the tempesta/linux-5.10.35.patch patch to the new kernel
• replace tempesta/linux-5.10.35.patch with a new patch linux-6.8.9.patch
• create a new directory 5.10.35 in https://github.com/tempesta-tech/linux_patches and place all patches since the previous migration (from 4.14.32) as N.diff, where N is the number of patch, use git_patches.sh for this
• update all the wiki pages referencing the 5 kernel to the new 6.8.9
• work with @RomanBelozerov to make our CI migrate to the new kernel

[kingluo]

I divided this issue into 7 parts:

• part 0: rcu stall issue: not sure if it's related to our patch or upstream bug, happens rarely
• part 1,2 and 3: Performance related issues.
  • part 1: fpu invalid opcode: 0000
  • part 2: paged-skb-patch
  • part 3: Assembly problem: endbr64-disallow-indirect-jump
• part 4,5 and 6: Function related issues, all found by tempesta-test.
  • part 4: kernel crash triggered by test cases
  • part 5: userspace segment fault triggered by test cases
  • part 6: test failures
  • part 7: TSO

---

1. fpu invalid opcode: 0000

kernel_fpu_begin_mask() --> asm ldmxcsr will cause an exception in the interrupt context when the system boots at the initrd stage.

Unlike the old version, the new kernel raises softirq even in the boot phase, when fpu related stuff (such as registers or process fpu state) is not ready for manipulation (otherwise exception raised), so it should be enabled in the first kernel_fpu_begin() (i.e. the first crypto API call) instead of __do_softirq().

[    0.000000] Linux version 6.8.9+ (kingluo@test) (gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #50 SMP PREEMPT_DYNAMIC Thu May 23 17:29:03 CST 2024
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-6.8.9+ root=UUID=af9a86ba-8a14-4ad9-b7d2-78508d3c3f1e ro recovery nomodeset dis_ucode_ldr console=ttyS0,38400
[    0.000000] KERNEL supported cpus:
[    0.000000]   Intel GenuineIntel
[    0.000000]   AMD AuthenticAMD
[    0.000000]   Hygon HygonGenuine
[    0.000000]   Centaur CentaurHauls
[    0.000000]   zhaoxin   Shanghai
[    0.000000] BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffdffff] usable
[    0.000000] BIOS-e820: [mem 0x00000000bffe0000-0x00000000bfffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000013fffffff] usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] APIC: Static calls initialized
[    0.000000] SMBIOS 3.0.0 present.
[    0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[    0.000000] last_pfn = 0x140000 max_arch_pfn = 0x400000000
[    0.000000] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs
[    0.000000] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WP  UC- WT
[    0.000000] last_pfn = 0xbffe0 max_arch_pfn = 0x400000000
[    0.000000] found SMP MP-table at [mem 0x000f5480-0x000f548f]
[    0.000000] RAMDISK: [mem 0x33b89000-0x35dbbfff]
[    0.000000] ACPI: Early table checksum verification disabled
[    0.000000] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )
[    0.000000] ACPI: RSDT 0x00000000BFFE1D57 000034 (v01 BOCHS  BXPC     00000001 BXPC 00000001)
[    0.000000] ACPI: FACP 0x00000000BFFE1BF3 000074 (v01 BOCHS  BXPC     00000001 BXPC 00000001)
[    0.000000] ACPI: DSDT 0x00000000BFFE0040 001BB3 (v01 BOCHS  BXPC     00000001 BXPC 00000001)
[    0.000000] ACPI: FACS 0x00000000BFFE0000 000040
[    0.000000] ACPI: APIC 0x00000000BFFE1C67 000090 (v03 BOCHS  BXPC     00000001 BXPC 00000001)
[    0.000000] ACPI: HPET 0x00000000BFFE1CF7 000038 (v01 BOCHS  BXPC     00000001 BXPC 00000001)
[    0.000000] ACPI: WAET 0x00000000BFFE1D2F 000028 (v01 BOCHS  BXPC     00000001 BXPC 00000001)
[    0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1bf3-0xbffe1c66]
[    0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1bf2]
[    0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f]
[    0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c67-0xbffe1cf6]
[    0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1cf7-0xbffe1d2e]
[    0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d2f-0xbffe1d56]
[    0.000000] No NUMA configuration found
[    0.000000] Faking a node at [mem 0x0000000000000000-0x000000013fffffff]
[    0.000000] NODE_DATA(0) allocated [mem 0x13ffd2000-0x13fffcfff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
[    0.000000]   DMA32    [mem 0x0000000001000000-0x00000000ffffffff]
[    0.000000]   Normal   [mem 0x0000000100000000-0x000000013fffffff]
[    0.000000]   Device   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000001000-0x000000000009efff]
[    0.000000]   node   0: [mem 0x0000000000100000-0x00000000bffdffff]
[    0.000000]   node   0: [mem 0x0000000100000000-0x000000013fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x000000013fffffff]
[    0.000000] On node 0, zone DMA: 1 pages in unavailable ranges
[    0.000000] On node 0, zone DMA: 97 pages in unavailable ranges
[    0.000000] On node 0, zone Normal: 32 pages in unavailable ranges
[    0.000000] ACPI: PM-Timer IO Port: 0x608
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
[    0.000000] IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
[    0.000000] ACPI: Using ACPI (MADT) for SMP configuration information
[    0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000
[    0.000000] TSC deadline timer available
[    0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs
[    0.000000] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]
[    0.000000] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x0009ffff]
[    0.000000] PM: hibernation: Registered nosave memory: [mem 0x000a0000-0x000effff]
[    0.000000] PM: hibernation: Registered nosave memory: [mem 0x000f0000-0x000fffff]
[    0.000000] PM: hibernation: Registered nosave memory: [mem 0xbffe0000-0xbfffffff]
[    0.000000] PM: hibernation: Registered nosave memory: [mem 0xc0000000-0xfffbffff]
[    0.000000] PM: hibernation: Registered nosave memory: [mem 0xfffc0000-0xffffffff]
[    0.000000] [mem 0xc0000000-0xfffbffff] available for PCI devices
[    0.000000] Booting paravirtualized kernel on bare hardware
[    0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns
[    0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1
[    0.000000] percpu: Embedded 64 pages/cpu s225280 r8192 d28672 u524288
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.8.9+ root=UUID=af9a86ba-8a14-4ad9-b7d2-78508d3c3f1e ro recovery nomodeset dis_ucode_ldr console=ttyS0,38400
[    0.000000] Booted with the nomodeset parameter. Only the system framebuffer will be available
[    0.000000] Unknown kernel command line parameters "recovery dis_ucode_ldr BOOT_IMAGE=/boot/vmlinuz-6.8.9+", will be passed to user space.
[    0.000000] random: crng init done
[    0.000000] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes, linear)
[    0.000000] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    0.000000] Fallback order for Node 0: 0
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 1031904
[    0.000000] Policy zone: Normal
[    0.000000] mem auto-init: stack:off, heap alloc:on, heap free:off
[    0.000000] software IO TLB: area num 4.
[    0.000000] Memory: 3971448K/4193784K available (18432K kernel code, 3189K rwdata, 7024K rodata, 12736K init, 4384K bss, 222076K reserved, 0K cma-reserved)
[    0.000000] Tempesta: allocated huge pages space (____ptrval____) 512MB at node 0
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] ftrace: allocating 51518 entries in 202 pages
[    0.000000] ftrace: allocated 202 pages with 4 groups
[    0.000000] Dynamic Preempt: voluntary
[    0.000000] rcu: Preemptible hierarchical RCU implementation.
[    0.000000] rcu:     RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4.
[    0.000000]  Trampoline variant of Tasks RCU enabled.
[    0.000000]  Rude variant of Tasks RCU enabled.
[    0.000000]  Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
[    0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16
[    0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[    0.000000] Console: colour VGA+ 80x25
[    0.000000] printk: legacy console [ttyS0] enabled
[    0.000000] ACPI: Core revision 20230628
[    0.000000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns
[    0.004000] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
[    0.004000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 6.8.9+ #50
[    0.004000] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[    0.004000] RIP: 0010:__kernel_fpu_begin_mask+0x55/0xa0
[    0.004000] Code: 65 48 8b 3c 25 c0 36 03 00 f7 47 2c 00 40 20 00 74 3a 65 48 c7 05 7f 4a dc 58 00 00 00 00 f6 c3 02 74 0b c7 45 ec 80 1f 00 00 <0f> ae 55 ec 83 e3 01 75 34 48 8b 45 f0 65 48 2b 04 25 28 00 00 00
[    0.004000] RSP: 0000:ffffba7580003f58 EFLAGS: 00010002
[    0.004000] RAX: 0000000000000000 RBX: 0000000000000002 RCX: 0000000000000000
[    0.004000] RDX: 0000000000000001 RSI: 0000000004200000 RDI: ffffffffa8c0a980
[    0.004000] RBP: ffffba7580003f70 R08: 00000036e863fce8 R09: c8656ec1deb9f021
[    0.004000] R10: ffffba7580003dc0 R11: ffffffffa8d50728 R12: ffffffffa8c03d68
[    0.004000] R13: 0000000000000200 R14: 0000000000000000 R15: 0000000000000000
[    0.004000] FS:  0000000000000000(0000) GS:ffff9ddb3bc00000(0000) knlGS:0000000000000000
[    0.004000] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    0.004000] CR2: ffff9ddb3ffff000 CR3: 0000000006a2e000 CR4: 00000000000000b0
[    0.004000] Call Trace:
[    0.004000]  <IRQ>
[    0.004000]  ? show_regs+0x6e/0x80
[    0.004000]  ? die+0x3c/0xa0
[    0.004000]  ? do_trap+0xd4/0xf0
[    0.004000]  ? do_error_trap+0x75/0xa0
[    0.004000]  ? __kernel_fpu_begin_mask+0x55/0xa0
[    0.004000]  ? exc_invalid_op+0x57/0x80
[    0.004000]  ? __kernel_fpu_begin_mask+0x55/0xa0
[    0.004000]  ? asm_exc_invalid_op+0x1f/0x30
[    0.004000]  ? __kernel_fpu_begin_mask+0x55/0xa0
[    0.004000]  __do_softirq+0x65/0x2af
[    0.004000]  __irq_exit_rcu+0x6b/0x90
[    0.004000]  irq_exit_rcu+0x12/0x20
[    0.004000]  common_interrupt+0x92/0xa0
[    0.004000]  </IRQ>
[    0.004000]  <TASK>
[    0.004000]  asm_common_interrupt+0x2b/0x40
[    0.004000] RIP: 0010:__x86_return_thunk+0x0/0x10
[    0.004000] Code: e8 01 00 00 00 cc e8 01 00 00 00 cc 48 81 c4 80 00 00 00 65 48 c7 04 25 d0 36 03 00 ff ff ff ff c3 cc 0f 1f 84 00 00 00 00 00 <c3> cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc e9 db 7d 10 ff 0f
[    0.004000] RSP: 0000:ffffffffa8c03e18 EFLAGS: 00000206
[    0.004000] RAX: 0000000000000001 RBX: ffff9ddb001a2360 RCX: 0000000000015a00
[    0.004000] RDX: 0000000000000001 RSI: 0000000000000246 RDI: ffff9ddb001a22a4
[    0.004000] RBP: ffffffffa8c03e20 R08: 00000000ffffffea R09: 0000000000000000
[    0.004000] R10: 0000000000000246 R11: ffff9ddb001a22a4 R12: ffff9ddb001a2200
[    0.004000] R13: ffff9ddb00197e00 R14: 0000000000000000 R15: 0000000000000000
[    0.004000]  ? _raw_spin_unlock_irqrestore+0x21/0x40
[    0.004000]  __setup_irq+0x4ce/0x7b0
[    0.004000]  request_threaded_irq+0x116/0x180
[    0.004000]  hpet_time_init+0x3e/0x60
[    0.004000]  x86_late_time_init+0x1f/0x40
[    0.004000]  start_kernel+0x442/0x780
[    0.004000]  x86_64_start_reservations+0x1c/0x30
[    0.004000]  x86_64_start_kernel+0x80/0x80
[    0.004000]  secondary_startup_64_no_verify+0x175/0x17b
[    0.004000]  </TASK>
[    0.004000] Modules linked in:
[    0.004000] ---[ end trace 0000000000000000 ]---
[    0.004000] RIP: 0010:__kernel_fpu_begin_mask+0x55/0xa0
[    0.004000] Code: 65 48 8b 3c 25 c0 36 03 00 f7 47 2c 00 40 20 00 74 3a 65 48 c7 05 7f 4a dc 58 00 00 00 00 f6 c3 02 74 0b c7 45 ec 80 1f 00 00 <0f> ae 55 ec 83 e3 01 75 34 48 8b 45 f0 65 48 2b 04 25 28 00 00 00
[    0.004000] RSP: 0000:ffffba7580003f58 EFLAGS: 00010002
[    0.004000] RAX: 0000000000000000 RBX: 0000000000000002 RCX: 0000000000000000
[    0.004000] RDX: 0000000000000001 RSI: 0000000004200000 RDI: ffffffffa8c0a980
[    0.004000] RBP: ffffba7580003f70 R08: 00000036e863fce8 R09: c8656ec1deb9f021
[    0.004000] R10: ffffba7580003dc0 R11: ffffffffa8d50728 R12: ffffffffa8c03d68
[    0.004000] R13: 0000000000000200 R14: 0000000000000000 R15: 0000000000000000
[    0.004000] FS:  0000000000000000(0000) GS:ffff9ddb3bc00000(0000) knlGS:0000000000000000
[    0.004000] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    0.004000] CR2: ffff9ddb3ffff000 CR3: 0000000006a2e000 CR4: 00000000000000b0
[    0.004000] Kernel panic - not syncing: Fatal exception in interrupt
[    0.004000] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---

2. paged-skb-patch

The skb stuff is refactored by the latest kernel a lot, for example, now the kernel uses kmalloc_reserve() to alloc skb and return the actual size by the way, so the simple migration at this part will cause memory error, e.g. memcpy() segfault.

[kingluo]

3. Assembly problem: endbr64-disallow-indirect-jump

We have below assembly functions:

fw/str_avx2.S:SYM_FUNC_START(__tfw_strtolower_avx2)
fw/str_avx2.S:SYM_FUNC_START(__tfw_stricmp_avx2)
fw/str_avx2.S:SYM_FUNC_START(__tfw_stricmp_avx2_2lc)
fw/str_avx2.S:SYM_FUNC_START(__tfw_match_custom)
fw/str_avx2.S:SYM_FUNC_START(__tfw_match_ctext_vchar)
fw/str_avx2.S:SYM_FUNC_START(tfw_match_uri)
fw/str_avx2.S:SYM_FUNC_START(tfw_match_token)
fw/str_avx2.S:SYM_FUNC_START(tfw_match_token_lc)
fw/str_avx2.S:SYM_FUNC_START(tfw_match_qetoken)
fw/str_avx2.S:SYM_FUNC_START(tfw_match_nctl)
fw/str_avx2.S:SYM_FUNC_START(tfw_match_xff)
fw/str_avx2.S:SYM_FUNC_START(tfw_match_cookie)
fw/str_avx2.S:SYM_FUNC_START(__tfw_match_etag)
lib/str_simd.S:SYM_FUNC_START(__memcpy_fast)
lib/str_simd.S:SYM_FUNC_START(__memcmp_fast)
lib/str_simd.S:SYM_FUNC_START(__bzero_fast)
tls/bignum_x86-64.S:SYM_FUNC_START(mpi_cmp_x86_64_4)
tls/bignum_x86-64.S:SYM_FUNC_START(mpi_add_x86_64)
tls/bignum_x86-64.S:SYM_FUNC_START(mpi_add_mod_p256_x86_64)
tls/bignum_x86-64.S:SYM_FUNC_START(mpi_sub_x86_64_5_4)
tls/bignum_x86-64.S:SYM_FUNC_START(mpi_sub_x86_64_4_4)
tls/bignum_x86-64.S:SYM_FUNC_START(mpi_sub_mod_p256_x86_64)
tls/bignum_x86-64.S:SYM_FUNC_START(mpi_sub_x86_64_3_3)
tls/bignum_x86-64.S:SYM_FUNC_START(mpi_sub_x86_64_2_2)
tls/bignum_x86-64.S:SYM_FUNC_START(mpi_shift_l_x86_64)
tls/bignum_x86-64.S:SYM_FUNC_START(mpi_shift_l_x86_64_4)
tls/bignum_x86-64.S:SYM_FUNC_START(mpi_shift_l1_mod_p256_x86_64)
tls/bignum_x86-64.S:SYM_FUNC_START(mpi_shift_r_x86_64)
tls/bignum_x86-64.S:SYM_FUNC_START(mpi_shift_r_x86_64_4)
tls/bignum_x86-64.S:SYM_FUNC_START(mpi_div2_x86_64_4)

1. In the new kernel, assembly functions uniformly return from __x86_return_thunk. However, our assembly code uses the original ret instruction, so objtool in the kernel will notice this is a naked return during compilation.

/home/kingluo/tempesta/tls/tempesta_tls.o: warning: objtool: mpi_cmp_x86_64_4+0x35: 'naked' return found in RETHUNK build
/home/kingluo/tempesta/tls/tempesta_tls.o: warning: objtool: mpi_add_x86_64+0x4a: 'naked' return found in RETHUNK build
/home/kingluo/tempesta/tls/tempesta_tls.o: warning: objtool: mpi_add_mod_p256_x86_64+0x63: 'naked' return found in RETHUNK build
/home/kingluo/tempesta/tls/tempesta_tls.o: warning: objtool: mpi_sub_x86_64+0xc9: 'naked' return found in RETHUNK build
/home/kingluo/tempesta/tls/tempesta_tls.o: warning: objtool: mpi_sub_x86_64_5_4+0x3d: 'naked' return found in RETHUNK build
/home/kingluo/tempesta/tls/tempesta_tls.o: warning: objtool: mpi_sub_x86_64_4_4+0x31: 'naked' return found in RETHUNK build
/home/kingluo/tempesta/tls/tempesta_tls.o: warning: objtool: mpi_sub_mod_p256_x86_64+0x63: 'naked' return found in RETHUNK build
/home/kingluo/tempesta/tls/tempesta_tls.o: warning: objtool: mpi_sub_x86_64_3_3+0x25: 'naked' return found in RETHUNK build
/home/kingluo/tempesta/tls/tempesta_tls.o: warning: objtool: mpi_sub_x86_64_2_2+0x19: 'naked' return found in RETHUNK build
/home/kingluo/tempesta/tls/tempesta_tls.o: warning: objtool: mpi_shift_l_x86_64+0x38: 'naked' return found in RETHUNK build
/home/kingluo/tempesta/tls/tempesta_tls.o: warning: objtool: mpi_shift_l_x86_64_4+0x3f: 'naked' return found in RETHUNK build
/home/kingluo/tempesta/tls/tempesta_tls.o: warning: objtool: mpi_shift_l1_mod_p256_x86_64+0x60: 'naked' return found in RETHUNK build
/home/kingluo/tempesta/tls/tempesta_tls.o: warning: objtool: mpi_shift_r_x86_64+0x25: 'naked' return found in RETHUNK build
/home/kingluo/tempesta/tls/tempesta_tls.o: warning: objtool: mpi_shift_r_x86_64_4+0x25: 'naked' return found in RETHUNK build
/home/kingluo/tempesta/tls/tempesta_tls.o: warning: objtool: mpi_div2_x86_64_4+0x72: 'naked' return found in RETHUNK build
/home/kingluo/tempesta/tls/tempesta_tls.o: warning: objtool: mpi_tpl_mod_p256_x86_64+0x9d: 'naked' return found in RETHUNK build
/home/kingluo/tempesta/tls/tempesta_tls.o: warning: objtool: mpi_mul_x86_64_4+0x17d: 'naked' return found in RETHUNK build
/home/kingluo/tempesta/tls/tempesta_tls.o: warning: objtool: mpi_sqr_x86_64_4+0x112: 'naked' return found in RETHUNK build
/home/kingluo/tempesta/tls/tempesta_tls.o: warning: objtool: ecp_mod_p256_x86_64+0x1f9: 'naked' return found in RETHUNK build
/home/kingluo/tempesta/tls/tempesta_tls.o: warning: objtool: mpi_mul_int_x86_64_4+0x3f: 'naked' return found in RETHUNK build
/home/kingluo/tempesta/tls/tempesta_tls.o: warning: objtool: mpi_mul_mod_p256_x86_64_4+0x351: 'naked' return found in RETHUNK build
/home/kingluo/tempesta/tls/tempesta_tls.o: warning: objtool: mpi_sqr_mod_p256_x86_64_4+0x2e6: 'naked' return found in RETHUNK build
/home/kingluo/tempesta/tls/tempesta_tls.o: warning: objtool: mpi_from_mont_p256_x86_64+0xf8: 'naked' return found in RETHUNK build
/home/kingluo/tempesta/tls/tempesta_tls.o: warning: objtool: mpi_mul_mont_mod_p256_x86_64+0x227: 'naked' return found in RETHUNK build
/home/kingluo/tempesta/tls/tempesta_tls.o: warning: objtool: mpi_sqr_mont_mod_p256_x86_64+0x1c1: 'naked' return found in RETHUNK build
/home/kingluo/tempesta/tls/tempesta_tls.o: warning: objtool: .rodata+0x10c0: data relocation to !ENDBR: mpi_sub_x86_64+0x9f
/home/kingluo/tempesta/tls/tempesta_tls.o: warning: objtool: .rodata+0x10c8: data relocation to !ENDBR: mpi_sub_x86_64+0x90
/home/kingluo/tempesta/tls/tempesta_tls.o: warning: objtool: .rodata+0x10d0: data relocation to !ENDBR: mpi_sub_x86_64+0x81
/home/kingluo/tempesta/tls/tempesta_tls.o: warning: objtool: .rodata+0x10d8: data relocation to !ENDBR: mpi_sub_x86_64+0x72

2. SYM_FUNC_START in the new kernel will add endbr64 to the head of the assembly function, and all indirect jumps to ENDBR instructions, that is, the code snippet within the same function, will fail, but we use jump tables in the assembly function to perform indirect jumps. It will raise CET exception.

When IBT is enabled, an indirect branch (jump, call, return) to any instruction that is not an ENDBR32/64 instruction will cause a #CP exception.

[  680.737353] Missing ENDBR: .sic2lc_len1+0x0/0x4 [tempesta_fw]
[  680.737396] ------------[ cut here ]------------
[  680.737405] kernel BUG at arch/x86/kernel/cet.c:102!
[  680.737417] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
[  680.737427] CPU: 3 PID: 2676 Comm: curl Kdump: loaded Tainted: G        W  OE      6.8.9+ #84
[  680.737438] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[  680.737450] RIP: 0010:exc_control_protection+0xbf/0xd0
[  680.737463] Code: 4c 89 e7 e8 d3 2d 1c ff 44 89 f6 4c 89 e7 e8 18 08 00 00 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 49 c7 44 24 50 00 00 00 00 eb de <0f> 0b 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 90 90 90 90 90
[  680.737481] RSP: 0018:ffffb302401605b0 EFLAGS: 00010002
[  680.737491] RAX: 0000000000000031 RBX: 0000000000000000 RCX: 0000000000000000
[  680.737500] RDX: 0000000000000000 RSI: ffff94ed3bda1840 RDI: ffff94ed3bda1840
[  680.737509] RBP: ffffb302401605c8 R08: 0000000000000000 R09: ffffb30240160438
[  680.737518] R10: ffffb30240160430 R11: ffffffff9b150728 R12: ffffb302401605d8
[  680.737527] R13: 0000000000000003 R14: 0000000000000000 R15: 0000000000000000
[  680.737535] FS:  00007f324b92d740(0000) GS:ffff94ed3bd80000(0000) knlGS:0000000000000000
[  680.737544] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  680.737553] CR2: 0000560795bd9a08 CR3: 0000000005d6e000 CR4: 0000000000b50ef0
[  680.737563] Call Trace:
[  680.737571]  <IRQ>
[  680.737578]  ? show_regs+0x6e/0x80
[  680.737589]  ? die+0x3c/0xa0
[  680.737596]  ? do_trap+0xd4/0xf0
[  680.737605]  ? do_error_trap+0x75/0xa0
[  680.737613]  ? exc_control_protection+0xbf/0xd0
[  680.737622]  ? exc_invalid_op+0x57/0x80
[  680.737632]  ? exc_control_protection+0xbf/0xd0
[  680.737640]  ? asm_exc_invalid_op+0x1f/0x30
[  680.737650]  ? exc_control_protection+0xbf/0xd0
[  680.737659]  asm_exc_control_protection+0x2b/0x30
[  680.737667] RIP: 0010:.sic2lc_len1+0x0/0x4 [tempesta_fw]
[  680.737691] Code: 88 40 62 ba c2 32 4e 01 0f b6 c1 09 c2 0f b6 07 0f b6 80 40 62 ba c2 32 06 0f b6 c0 09 d0 c3 cc cc cc cc 31 c0 c3 cc cc cc cc <31> d2 eb df 31 d2 eb c8 31 c0 eb af 31 c0 eb 98 31 d2 eb 81 31 d2
[  680.737883] RSP: 0018:ffffb30240160680 EFLAGS: 00010297
[  680.738036] RAX: ffffffffc0b59ade RBX: 0000000000000001 RCX: 0000000000000003
[  680.738183] RDX: 0000000000000001 RSI: ffffffffc2baa1ef RDI: ffff94ec1e690061
[  680.738379] RBP: ffffb302401606a8 R08: ffffffffc2baa1ef R09: 0000000000000001
[  680.738585] R10: 0000000000000001 R11: ffff94ec1bc5f198 R12: ffff94ec1bc5f1b8
[  680.738788] R13: ffff94ec1bc5f198 R14: ffffffffc2baa1ef R15: ffff94ec1e690061
[  680.738995]  ? .sic2lc_len0+0x7/0x7 [tempesta_fw]
[  680.739214]  ? __try_str+0x50/0xb0 [tempesta_fw]
[  680.739488]  __h2_req_parse_accept+0x31f/0xdb0 [tempesta_fw]
[  680.739705]  tfw_h2_parse_req_hdr_val+0x330/0x76e0 [tempesta_fw]
[  680.739921]  ? tfw_huffman_decode+0x414/0x6d0 [tempesta_fw]
[  680.740135]  tfw_hpack_decode+0x738/0x1f70 [tempesta_fw]
[  680.740345]  tfw_h2_parse_req+0x164/0x270 [tempesta_fw]
[  680.740556]  ss_skb_process+0xf9/0x140 [tempesta_fw]
[  680.740765]  ? __pfx_tfw_h2_parse_req+0x10/0x10 [tempesta_fw]
[  680.740973]  tfw_http_req_process+0x8d/0xaa0 [tempesta_fw]
[  680.741181]  ? alloc_pages_mpol+0x95/0x1f0
[  680.741377]  ? alloc_pages+0x58/0xa0
[  680.741569]  ? __get_free_pages+0x15/0x40
[  680.741761]  ? tfw_pool_alloc_pages+0x7c/0x90 [tempesta_fw]
[  680.741972]  ? __tfw_pool_new+0x31/0x80 [tempesta_fw]
[  680.742173]  tfw_http_msg_process_generic+0x18d/0x6d0 [tempesta_fw]
[  680.742371]  ? ss_skb_chop_head_tail+0xc9/0x1e0 [tempesta_fw]
[  680.742567]  tfw_h2_frame_process+0x4d6/0x6e0 [tempesta_fw]
[  680.742756]  tfw_http_msg_process+0x50/0x60 [tempesta_fw]
[  680.742935]  tfw_connection_recv+0xc3/0x140 [tempesta_fw]
[  680.743112]  tfw_tls_connection_recv+0x31b/0x430 [tempesta_fw]
[  680.743290]  ss_tcp_process_data+0x1e1/0x470 [tempesta_fw]
[  680.743470]  ss_tcp_data_ready+0x83/0x170 [tempesta_fw]
[  680.743646]  tcp_data_ready+0x35/0xe0
[  680.743813]  tcp_data_queue+0x8d5/0xe20
[  680.743976]  tcp_rcv_established+0x244/0x790
[  680.744137]  tcp_v4_do_rcv+0x16a/0x2a0
[  680.744295]  tcp_v4_rcv+0xf01/0xf70
[  680.744451]  ? raw_local_deliver+0xcd/0x240
[  680.744611]  ip_protocol_deliver_rcu+0x37/0x180
[  680.744766]  ip_local_deliver_finish+0x8a/0xb0
[  680.744921]  ip_local_deliver+0x73/0x120
[  680.745075]  ? nf_nat_setup_info+0xb81/0xc20 [nf_nat]
[  680.745235]  ? __pfx_ip_local_deliver_finish+0x10/0x10
[  680.745392]  ip_rcv+0x18f/0x1b0
[  680.745549]  ? __pfx_ip_rcv_finish+0x10/0x10
[  680.745706]  __netif_receive_skb_one_core+0x8a/0xa0
[  680.745715] Missing ENDBR: .sic2lc_len3+0x0/0x4 [tempesta_fw]
[  680.745866]  __netif_receive_skb+0x15/0x60
[  680.746034] ------------[ cut here ]------------
[  680.746194]  process_backlog+0x9a/0x140
[  680.746359] kernel BUG at arch/x86/kernel/cet.c:102!
[  680.746519]  __napi_poll+0x31/0x1d0
[  680.746820]  net_rx_action+0x29d/0x310
[  680.746963]  __do_softirq+0xcd/0x2a5
[  680.747105]  do_softirq.part.0+0x41/0x60
[  680.747246]  </IRQ>
[  680.747382]  <TASK>
[  680.747514]  __local_bh_enable_ip+0x6e/0x70
[  680.747648]  __dev_queue_xmit+0x33d/0xde0
[  680.747801]  ? hash_conntrack_raw+0x6b/0xe0 [nf_conntrack]
[  680.747943]  ip_finish_output2+0x2dc/0x550
[  680.748075]  ? nf_conntrack_in+0xeb/0x6c0 [nf_conntrack]
[  680.748213]  __ip_finish_output+0xb7/0x190
[  680.748344]  ip_finish_output+0x2d/0xe0
[  680.748474]  ip_output+0x63/0xf0
[  680.748603]  ? __pfx_ip_finish_output+0x10/0x10
[  680.748732]  ip_local_out+0x62/0x70
[  680.748860]  __ip_queue_xmit+0x19b/0x4f0
[  680.748988]  ip_queue_xmit+0x19/0x20
[  680.749113]  __tcp_transmit_skb+0xada/0xc90
[  680.749240]  tcp_write_xmit+0x5d0/0x1420
[  680.749367]  __tcp_push_pending_frames+0x3b/0x110
[  680.749494]  tcp_push+0x10c/0x120
[  680.749620]  tcp_sendmsg_locked+0x91a/0xdd0
[  680.749745]  tcp_sendmsg+0x31/0x50
[  680.749868]  inet_sendmsg+0x47/0x80
[  680.749989]  sock_write_iter+0x163/0x190
[  680.750112]  vfs_write+0x38a/0x430
[  680.750235]  ksys_write+0xb9/0xf0
[  680.750354]  __x64_sys_write+0x1d/0x30
[  680.750473]  x64_sys_call+0x1681/0x20c0
[  680.750593]  do_syscall_64+0x54/0x120
[  680.750709]  entry_SYSCALL_64_after_hwframe+0x78/0x80
[  680.750828] RIP: 0033:0x7f324bb14887
[  680.750946] Code: 10 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24
[  680.751213] RSP: 002b:00007ffd5bc22778 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[  680.751354] RAX: ffffffffffffffda RBX: 000000000000004a RCX: 00007f324bb14887
[  680.751495] RDX: 000000000000004a RSI: 0000560795bd5953 RDI: 0000000000000005
[  680.751638] RBP: 0000560795ba4ae0 R08: 0000000000000001 R09: 00007f324b6eb340
[  680.751783] R10: 0000000000000068 R11: 0000000000000246 R12: 0000560795bd5953
[  680.751927] R13: 000000000000004a R14: 00007ffd5bc22820 R15: 0000560795ba5c50
[  680.751965] Missing ENDBR: .sic2lc_len3+0x0/0x4 [tempesta_fw]
[  680.752072]  </TASK>
[  680.752186] ------------[ cut here ]------------

Solution

• 1. Substitute the ret with RET, a macro added in the new kernel to ensure the correct return.
• 2. notrack jmp and enable notrack in CPU setting: wrmsrl(MSR_IA32_S_CET, CET_ENDBR_EN | CET_NO_TRACK_EN)

Prefix used with indirect CALL/JMP near instructions (opcodes FF /2 and FF /4) to indicate that the branch target is not required to start with an ENDBR32/64 instruction. Prefix only honored when NO_TRACK_EN flag is set.

As an aside, interestingly, if a user-mode C program uses a switch statement that meets the conditions for generating a jump table (gcc uses -fcf-protection=full by default), the generated jump table will use a jmp with the notrack prefix, and IBT will be marked as true in the .note.gnu.property section of the compiled elf file, so that the NO_TRACK_EN of the MSR will be set to true when the kernel loads and executes this user program. So user mode can use notrack to bypass CET without caring about setting or not setting NO_TRACK_EN.