JVM Crash while starting

[knokio]

So we've been using the Datadog Agent for more than 12 months and everything seemed ok.
We've recently added a wrapper around our cron jobs and we're able to pick ocasional JVM crashes when starting these jobs.

We used to have dd-trace-java 0.8 but we tried upgrading to the latest 0.18 and the crashes still occurred.
We disabled the agent for a few days, to test if this could be the issue and the crashes disappeared.
There seems to be a connection. The other variables where not changed (no new deploy, same machine, etc).

Attached is a simple log of the JVM starting. Do you have any similar cases?
We found the crash might be related to bytebuddy which is used in this agent.

java -Djava.awt.headless=true -Xms128m -Xmx1024m -XX:+UseConcMarkSweepGC -Dlog4j.configuration=file:/folder/utils.properties -javaagent:/folder/shared/dd-java-agent.jar -jar /folder/current --config /folder/server.properties job-name --threads=1 

[main] INFO datadog.trace.agent.ot.DDTraceOTInfo - dd-trace - version: 0.18.0~d1e8ed12
[main] INFO datadog.trace.agent.ot.DDTracer - New instance: DDTracer-4416d64f{ serviceName=utils, writer=DDAgentWriter { api=DDApi { tracesEndpoint=http://localhost:8126/v0.4/traces } }, sampler=AllSampler { sample=true }, runtimeId=98ce9072, defaultSpanTags={}}
[main] INFO datadog.trace.agent.tooling.VersionLogger - dd-trace-ot - version: 0.18.0~d1e8ed12
[main] INFO datadog.trace.agent.tooling.VersionLogger - dd-trace-api - version: 0.18.0~d1e8ed12
[main] INFO datadog.trace.agent.tooling.VersionLogger - dd-java-agent - version: 0.18.0~d1e8ed12
[main] INFO datadog.trace.agent.jmxfetch.JMXFetch - JMXFetch is disabled

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fdc3f3ab9f2, pid=20229, tid=0x00007fdbdbbfb700
#
# JRE version: OpenJDK Runtime Environment (8.0_181-b13) (build 1.8.0_181-8u181-b13-2~deb9u1-b13)
# Java VM: OpenJDK 64-Bit Server VM (25.181-b13 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x7b89f2]
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/user/hs_err_pid20229.log
#
# Compiler replay data is saved as:
# /home/user/replay_pid20229.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
/folder/job.sh: line 47: 20216 Aborted               

Bytebuddy class:

C2: 5798 1763 4 net.bytebuddy.description.type.TypeDescription$Generic$OfTypeVariable::accept (8 bytes)

[tylerbenson]

@knokio Thanks for the report. This is deeply concerning and should not be happening. Does this happen immediately on startup, or some time after the application has been running? Is there a consistent way to reproduce this?

[mar-kolya]

@knokio would you mind opening a support ticket for this problem and providing agent debug log along with /home/user/hs_err_pid20229.log and /home/user/replay_pid20229.log - they might have some useful information there.
Also, it might be useful to gather core dump from that failure and upload it to support ticket as well.

You may also want to consider updating to latest java8 version - looks like there is build 201 available now - maybe this problem would be resolved there.

[knokio]

@knokio Thanks for the report. This is deeply concerning and should not be happening. Does this happen immediately on startup, or some time after the application has been running? Is there a consistent way to reproduce this?

It happens immediately on startup. There is no consistent way to reproduce but it happens sporadically.

[knokio]

@knokio would you mind opening a support ticket for this problem and providing agent debug log along with /home/user/hs_err_pid20229.log and /home/user/replay_pid20229.log - they might have some useful information there.
Also, it might be useful to gather core dump from that failure and upload it to support ticket as well.

You may also want to consider updating to latest java8 version - looks like there is build 201 available now - maybe this problem would be resolved there.

I will open a ticket with the support. Is there any sensitive information I should exclude from these logs?

[mar-kolya]

@knokio let's move this question to support ticket - I think that would be easier.
Thanks!

[tylerbenson]

@raphw, Do you have any suggestions for how to troubleshoot these kinds of reports? A different customer isolated the version difference that is causing this to when we updated from bb 1.8.14 to 1.9.2. I've asked them to try a new build with the latest version of Byte Buddy to see if it might help. (Let me know if you'd prefer I file an issue in your repo.)

[raphw]

Can I get access to the crash dumps? Looks like a bug in HotSpot to me, to be honest.

[tylerbenson]

@raphw What's the best way to privately message you?

[raphw]

From looking into the dump file, this has nothing to do with Byte Buddy. It is a bug in the C2 compiler that seems to reference an invalid machine code adress in the compiled code. I would guess that this issue is fixed today; you could for example try running with a Java 8 VM to see if the application starts up. Maybe Oracle also backported the patch to Java 7 if you have a support subscription.

The reason that the problem occurs after the update is that the changed code now triggers this update.

I would report this issue to Oracle but please note that it is unlikely that it gets fixed if it is already fixed on Java 11 as older VMs are no longer supported at this point.

[tylerbenson]

@raphw Thanks for looking into this. Do you know if there is anything specific we could do to avoid triggering it?

[raphw]

This is kind of difficult as the program has no real control over how the JVM is translating a program to machine code. Any program might run into this bug if only the profile is set right. You can of course influence how C2 handles code (or even turn it of via -Xint or -client).

If I remember correctly, you can even turn off compilation for specific methods only. You can probably find a lot in this flag explorer and try out a few things: https://chriswhocodes.com/hotspot_options_jdk7.html

[knokio]

Thank you both for these comments. Is there any specific JVM that we could try to run to avoid this? @raphw

[raphw]

Could you try on the most recent Java 8 VM?

[mrproliu]

Could you try on the most recent Java 8 VM?

I has the same issue on the Java 8 VM.

dump file:
--------------- T H R E A D ---------------

Current thread (0x00007fa354a28800): JavaThread "C2 CompilerThread0" daemon [_thread_in_native, id=22053, stack(0x00007fa33695b000,0x00007fa336a5c000)]

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000008

Registers:
RAX=0x0000000000000000, RBX=0x00007fa336a585f0, RCX=0x0000000000000000, RDX=0x00000000000002be
RSP=0x00007fa336a56310, RBP=0x00007fa336a56340, RSI=0x0000000000000000, RDI=0x00007fa319178eb0
R8 =0x0000000000000000, R9 =0x00007fa30c3866a0, R10=0x00007fa30c5a3bb0, R11=0x00007fa35a644e4d
R12=0x00007fa318b9f1c8, R13=0x00007fa318ba1470, R14=0x00007fa30c5a3b18, R15=0x0000000000000086
RIP=0x00007fa359deb1d4, EFLAGS=0x0000000000010202, CSGSFS=0x0000000000000033, ERR=0x0000000000000004
TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007fa336a56310)
0x00007fa336a56310: 00007fa336a58610 00007fa336a56630
0x00007fa336a56320: 00007fa30c5a3b18 00007fa336a56670
0x00007fa336a56330: 0000000000000001 0000000000000086
0x00007fa336a56340: 00007fa336a563a0 00007fa359deb78c
0x00007fa336a56350: 00007fa300000001 00007fa30c2c8180
0x00007fa336a56360: 00007fa336a56650 00007fa336a585f0
0x00007fa336a56370: 00007fa336a58fa8 0000000000000001
0x00007fa336a56380: 00007fa30d49f7c0 00007fa336a585f0
0x00007fa336a56390: 00007fa336a56650 00007fa354a29da0
0x00007fa336a563a0: 00007fa336a566c0 00007fa359dee3a5
0x00007fa336a563b0: 8000584c00000000 00007fa336a56670
0x00007fa336a563c0: 00007fa336a56630 01007fa35a04fe6d
0x00007fa336a563d0: 00007fa336a58610 00007fa300000001
0x00007fa336a563e0: 00007fa354a29da0 0000007c00000004
0x00007fa336a563f0: 00007fa3188abc30 00007fa300000000
0x00007fa336a56400: 00007fa3183cbd90 00007fa3183cbe10
0x00007fa336a56410: 00007fa336a56440 00007fa35a059feb
0x00007fa336a56420: 00007fa35a57a7b8 0000000000000001
0x00007fa336a56430: 0000000000000002 00007fa30c0506c0
0x00007fa336a56440: 00007fa336a56450 00007fa35a04fe6d
0x00007fa336a56450: 00007fa336a564b0 00007fa35a63b379
0x00007fa336a56460: 0000000000000005 000000007fffffff
0x00007fa336a56470: 00007fa30c45fe78 00007fa30c45fe78
0x00007fa336a56480: 000000760000000c 0000000000000200
0x00007fa336a56490: 0000000000000040 0000000000000000
0x00007fa336a564a0: 0000000000000001 0000000000000008
0x00007fa336a564b0: 0000000000008040 0000023000000802
0x00007fa336a564c0: 0000007c0000009b 0000007600000078
0x00007fa336a564d0: 00007fa318f5efb0 00007fa30c000020
0x00007fa336a564e0: 0000000000008010 0000000000008010
0x00007fa336a564f0: 0000000000000000 0000000000000000
0x00007fa336a56500: 00007fa336a56570 00007fa35a63bb1c

Instructions: (pc=0x00007fa359deb1d4)
0x00007fa359deb1b4: 00 00 8b 70 28 49 89 c4 0f 1f 40 00 89 f0 48 8d
0x00007fa359deb1c4: 3c c5 00 00 00 00 48 03 bb f8 09 00 00 48 8b 0f
0x00007fa359deb1d4: 48 8b 41 08 48 83 38 00 75 2b 8b 73 28 0f 1f 80
0x00007fa359deb1e4: 00 00 00 00 8b 41 28 31 c9 39 f0 73 0e 89 c2 48

Register to memory mapping:

RAX=0x0000000000000000 is an unknown value
RBX=0x00007fa336a585f0 is pointing into the stack for thread: 0x00007fa354a28800
RCX=0x0000000000000000 is an unknown value
RDX=0x00000000000002be is an unknown value
RSP=0x00007fa336a56310 is pointing into the stack for thread: 0x00007fa354a28800
RBP=0x00007fa336a56340 is pointing into the stack for thread: 0x00007fa354a28800
RSI=0x0000000000000000 is an unknown value
RDI=0x00007fa319178eb0 is an unknown value
R8 =0x0000000000000000 is an unknown value
R9 =0x00007fa30c3866a0 is an unknown value
R10=0x00007fa30c5a3bb0 is an unknown value
R11=0x00007fa35a644e4d: <offset 0x83e4d> in /lib64/libc.so.6 at 0x00007fa35a5c1000
R12=0x00007fa318b9f1c8 is an unknown value
R13=0x00007fa318ba1470 is an unknown value
R14=0x00007fa30c5a3b18 is an unknown value
R15=0x0000000000000086 is an unknown value

Stack: [0x00007fa33695b000,0x00007fa336a5c000], sp=0x00007fa336a56310, free space=1004k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x8181d4] PhaseIdealLoop::build_loop_late_post(Node*)+0x144
V [libjvm.so+0x81878c] PhaseIdealLoop::build_loop_late(VectorSet&, Node_List&, Node_Stack&)+0x13c
V [libjvm.so+0x81b3a5] PhaseIdealLoop::build_and_optimize(bool, bool)+0x835
V [libjvm.so+0x4a25f0] Compile::Optimize()+0xbe0
V [libjvm.so+0x4a684b] Compile::Compile(ciEnv*, C2Compiler*, ciMethod*, int, bool, bool, bool)+0x13cb
V [libjvm.so+0x3f20a8] C2Compiler::compile_method(ciEnv*, ciMethod*, int)+0x198
V [libjvm.so+0x4af61a] CompileBroker::invoke_compiler_on_method(CompileTask*)+0xc9a
V [libjvm.so+0x4b05c6] CompileBroker::compiler_thread_loop()+0x5d6
V [libjvm.so+0xa7343f] JavaThread::thread_main_inner()+0xdf
V [libjvm.so+0xa7356c] JavaThread::run()+0x11c
V [libjvm.so+0x924698] java_start(Thread*)+0x108
C [libpthread.so.0+0x7aa1] start_thread+0xd1

Current CompileTask:
C2: 22280 7861 4 dep.com.lagou.pard.dependencies.net.bytebuddy.description.type.TypeDescription$Generic$OfTypeVariable::accept (8 bytes)

System Info
OS:CentOS release 6.2 (Final)

uname:Linux 2.6.32-279.19.27.el6.ucloud.x86_64 #1 SMP Fri Aug 14 16:10:19 CST 2015 x86_64
libc:glibc 2.12 NPTL 2.12
rlimit: STACK 10240k, CORE infinity, NPROC 655350, NOFILE 655350, AS infinity
load average:0.98 1.16 1.73

CPU:total 2 (1 cores per cpu, 1 threads per core) family 6 model 42 stepping 1, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, aes, clmul, erms, tsc

Memory: 4k page, physical 3860700k(1533216k free), swap 524280k(376964k free)

vm_info: Java HotSpot(TM) 64-Bit Server VM (25.101-b13) for linux-amd64 JRE (1.8.0_101-b13), built on Jun 22 2016 02:59:44 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)

[mar-kolya]

@mliu111477 This dump seems to have been created by build 101 of Java8. This page suggests that current build number is 201. These kinds of bugs may be fixed in incrementa JVM builds.

Also dep.com.lagou.pard.dependencies.net.bytebuddy.description.type.TypeDescription$Generic$OfTypeVariable::accept - this looks like repackaged Bytebuddy, and I do not immediately recognized which project would use this prefix. Is this some proprietary code? If so, I'm not sure this would be directly related to Datadog APM. Are you using some other dependencies that use Bytebuddy?