JDK19 HCRLateAttachWorkload_previewEnabled_1_FAILED StackOverflowError: operating system stack overflow #15250

JasonFengJ9 · 2022-06-07T19:11:11Z

Failure link

From an internal build job/Test_openjdknext_j9_extended.system_aarch64_linux_Personal/1/tapResults/(cent7-aarch64-6):

11:32:17  openjdk version "19-internal" 2022-09-20
11:32:17  OpenJDK Runtime Environment (build 19-internal-adhoc.jenkins.BuildJDKnextaarch64linuxPersonal)
11:32:17  Eclipse OpenJ9 VM (build exclude19-52f04efbff5, JRE 19 Linux aarch64-64-Bit Compressed References 20220607_35 (JIT enabled, AOT enabled)
11:32:17  OpenJ9   - 52f04efbff5
11:32:17  OMR      - c60867497c6
11:32:17  JCL      - 5ccf02de16a based on jdk-19+25)

Rerun in Grinder - Change TARGET to run only the failed test targets.

Optional info

Failure output (captured from console output)

===============================================
Running test HCRLateAttachWorkload_previewEnabled_1 ...
===============================================
HCRLateAttachWorkload_previewEnabled_1 Start Time: Tue Jun  7 13:00:41 2022 Epoch Time (ms): 1654621241268
variation: Mode650
JVM_OPTIONS:  -XX:-UseCompressedOops 

AG  13:00:47 > Will attempt to attach to JVM with pid 7807
AG  13:00:47 > Attach will be done using agent jar file: /home/jenkins/workspace/Test_openjdknext_j9_extended.system_aarch64_linux_Personal_testList_2/jvmtest/system/aqa-systemtest/openjdk.test.debugging/bin/StringModifierAgent.jar
LT  The agent's agentmain method was called.
LT  To re-run this agent with these options, pass this option to the agent on the command line: "percentage=50,threads=10,threadsTimeout=300000,duration=240,seed=3429801749770138821"
LT  Initialising all of the threads.
LT  Starting thread 0
LT  Starting thread 1
LT  Starting thread 2
LT  Starting thread 3
LT  Starting thread 4
LT  Starting thread 5
LT  Starting thread 6
LT  Starting thread 7
LT  Starting thread 8
LT  Starting thread 9
LT  Threads are all initialised.
LT  Entering the loop that will replace dead threads with live ones until we run out of time.
LT  stderr Exception in thread "Thread-9" Exception in thread "Thread-13" java/lang/StackOverflowError: operating system stack overflow
LT  stderr 	at java/lang/Thread.getAllThreads (java.base@19-internal/Thread.java:2695)
LT  stderr 	at java/lang/ThreadGroup.activeCount (java.base@19-internal/ThreadGroup.java:377)
LT  stderr 	at java/lang/Thread.activeCount (java.base@19-internal/Thread.java:2041)
LT  stderr 	at java/lang/String.length (java.base@19-internal/String.java:4294967295)
LT  stderr 	at java/lang/String.startsWith (java.base@19-internal/String.java:3010)
LT  stderr 	at java/lang/String.startsWith (java.base@19-internal/String.java:3053)
LT  stderr 	at java/lang/invoke/MethodType.fromDescriptor (java.base@19-internal/MethodType.java:1212)
LT  stderr 	at java/lang/invoke/MethodType.fromMethodDescriptorString (java.base@19-internal/MethodType.java:1195)
LT  stderr 	at java/lang/invoke/MethodTypeHelper.vmResolveFromMethodDescriptorString (java.base@19-internal/MethodTypeHelper.java:256)
LT  stderr 	at java/lang/invoke/MethodHandleResolver.resolveInvokeDynamic (java.base@19-internal/MethodHandleResolver.java:166)
LT  stderr 	at java/lang/Thread.getAllThreads (java.base@19-internal/Thread.java:2695)

LT  stderr 	at java.base/java.lang.Thread.getAllThreads(Thread.java:2695)
LT  stderr 	at java.base/java.lang.ThreadGroup.activeCount(ThreadGroup.java:377)
LT  stderr 	at java.base/java.lang.Thread.activeCount(Thread.java:2041)
LT  stderr 	at java.base/java.lang.String.length(String.java)
LT  stderr 	at java.base/java.lang.String.startsWith(String.java:3010)
LT  stderr 	at java.base/java.lang.String.startsWith(String.java:3053)
LT  stderr 	at java.base/java.lang.invoke.MethodType.fromDescriptor(MethodType.java:1212)
LT  stderr 	at java.base/java.lang.invoke.MethodType.fromMethodDescriptorString(MethodType.java:1195)
LT  stderr 	at java.base/java.lang.invoke.MethodTypeHelper.vmResolveFromMethodDescriptorString(MethodTypeHelper.java:256)
LT  stderr 	at java.base/java.lang.invoke.MethodHandleResolver.resolveInvokeDynamic(MethodHandleResolver.java:166)
LT  stderr 	at java.base/java.lang.Thread.getAllThreads(Thread.java:2695)
LT  stderr 	at java.base/java.lang.ThreadGroup.activeCount(ThreadGroup.java:377)
LT  stderr 	at java.base/java.lang.Thread.activeCount(Thread.java:2041)
LT  stderr 	at java.base/java.lang.String.length(String.java)
LT  stderr 	at java.base/java.lang.String.startsWith(String.java:3010)
LT  stderr 	at java.base/java.lang.String.startsWith(String.java:3053)
LT  stderr 	at java.base/java.lang.invoke.MethodType.fromDescriptor(MethodType.java:1212)
LT  stderr 	at java.base/java.lang.invoke.MethodType.fromMethodDescriptorString(MethodType.java:1195)
LT  stderr JVMDUMP010I Snap dump written to /home/jenkins/workspace/Test_openjdknext_j9_extended.system_aarch64_linux_Personal_testList_2/aqa-tests/TKG/output_16546195099989/HCRLateAttachWorkload_previewEnabled_1/20220607-130041-HCRLateAttachWorkload/results/Snap.20220607.130051.7807.0003.trc
LT  stderr getOriginalROMMethodUnchecked+0x2c (0x0000FFFFAD866A0C [libj9jit29.so+0x7b3a0c])
LT  stderr getOriginalROMMethod+0x54 (0x0000FFFFAD866B84 [libj9jit29.so+0x7b3b84])
LT  stderr _ZN11TR_J9MethodC1EP11TR_FrontEndP9TR_MemoryP20TR_OpaqueMethodBlock+0x70 (0x0000FFFFAD1F0190 [libj9jit29.so+0x13d190])
LT  stderr _ZN19TR_ResolvedJ9MethodC2EP20TR_OpaqueMethodBlockP11TR_FrontEndP9TR_MemoryP17TR_ResolvedMethodj+0x40 (0x0000FFFFAD1F0310 [libj9jit29.so+0x13d310])
LT  stderr _ZN11TR_J9VMBase33createResolvedMethodWithSignatureEP9TR_MemoryP20TR_OpaqueMethodBlockP19TR_OpaqueClassBlockPciP17TR_ResolvedMethodj.localalias+0x154 (0x0000FFFFAD1F3B14 [libj9jit29.so+0x140b14])
LT  stderr _ZN2J920SymbolReferenceTable31findOrCreateDynamicMethodSymbolEPN2TR20ResolvedMethodSymbolEiPbS4_+0x24 (0x0000FFFFAD1A1DE4 [libj9jit29.so+0xeede4])
LT  stderr _ZN24TR_J9ByteCodeIlGenerator16genInvokeDynamicEi+0x60 (0x0000FFFFAD256840 [libj9jit29.so+0x1a3840])
LT  stderr _ZN24TR_J9ByteCodeIlGenerator6walkerEPN2TR5BlockE+0x1964 (0x0000FFFFAD25C444 [libj9jit29.so+0x1a9444])
LT  stderr _ZN24TR_J9ByteCodeIlGenerator18genILFromByteCodesEv+0x300 (0x0000FFFFAD238FF4 [libj9jit29.so+0x185ff4])
LT  stderr _ZN24TR_J9ByteCodeIlGenerator5genILEv+0x60 (0x0000FFFFAD23AA70 [libj9jit29.so+0x187a70])
LT  stderr _ZN3OMR20ResolvedMethodSymbol5genILEP11TR_FrontEndPN2TR11CompilationEPNS3_20SymbolReferenceTableERNS3_12IlGenRequestE+0x220 (0x0000FFFFAD4C0A70 [libj9jit29.so+0x40da70])
LT  stderr _ZN3OMR11Compilation7compileEv+0x1d4 (0x0000FFFFAD4A7C34 [libj9jit29.so+0x3f4c34])
LT  stderr _ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadPNS_11CompilationEP17TR_ResolvedMethodR11TR_J9VMBaseP19TR_OptimizationPlanRKNS_16SegmentAllocatorE+0x3cc (0x0000FFFFAD1BF930 [libj9jit29.so+0x10c930])
LT  stderr _ZN2TR28CompilationInfoPerThreadBase14wrappedCompileEP13J9PortLibraryPv+0x2e4 (0x0000FFFFAD1C0634 [libj9jit29.so+0x10d634])
LT  stderr omrsig_protect+0x21c (0x0000FFFFAE4B525C [libj9prt29.so+0x2825c])
LT  stderr _ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadP21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x2c0 (0x0000FFFFAD1BE854 [libj9jit29.so+0x10b854])
LT  stderr _ZN2TR24CompilationInfoPerThread12processEntryER21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x16c (0x0000FFFFAD1BED50 [libj9jit29.so+0x10bd50])
LT  stderr _ZN2TR24CompilationInfoPerThread14processEntriesEv+0x2bc (0x0000FFFFAD1BDC4C [libj9jit29.so+0x10ac4c])
LT  stderr _ZN2TR24CompilationInfoPerThread3runEv+0x6c (0x0000FFFFAD1BDF70 [libj9jit29.so+0x10af70])
LT  stderr _Z30protectedCompilationThreadProcP13J9PortLibraryPN2TR24CompilationInfoPerThreadE+0x7c (0x0000FFFFAD1BDFFC [libj9jit29.so+0x10affc])
LT  stderr omrsig_protect+0x21c (0x0000FFFFAE4B525C [libj9prt29.so+0x2825c])
LT  stderr _Z21compilationThreadProcPv+0x140 (0x0000FFFFAD1BE3C0 [libj9jit29.so+0x10b3c0])
LT  stderr thread_wrapper+0xcc (0x0000FFFFAE4633BC [libj9thr29.so+0x73bc])
LT  stderr start_thread+0xb0 (0x0000FFFFAEC4C088 [libpthread.so.0+0x7088])
LT  stderr ---------------------------------------
LT  stderr JVMDUMP039I Processing dump event "gpf", detail "" at 2022/06/07 13:00:57 - please wait.
LT  stderr 	at java.base/java.lang.invoke.MethodHandleResolver.resolveInvokeDynamic(MethodHandleResolver.java:166)
LT  stderr Exception in thread "Thread-19" 	at java.base/java.lang.Thread.getAllThreads(Thread.java:2695)
LT  stderr JVMDUMP032I JVM requested JIT dump using '/home/jenkins/workspace/Test_openjdknext_j9_extended.system_aarch64_linux_Personal_testList_2/aqa-tests/TKG/output_16546195099989/HCRLateAttachWorkload_previewEnabled_1/20220607-130041-HCRLateAttachWorkload/results/jitdump.20220607.130051.7807.0004.dmp' in response to an event
LT  stderr 
LT  stderr 	at java.base/java.lang.invoke.MethodTypeHelper.vmResolveFromMethodDescriptorString(MethodTypeHelper.java:256)	at java.base/java.lang.invoke.MethodTypeHelper.vmResolveFromMethodDescriptorString(MethodTypeHelper.java:256)
LT  stderr java.lang.StackOverflowError: operating system stack overflow
LT  stderr JVMDUMP051I JIT dump occurred in 'JIT Compilation Thread-005' thread 0x0000FFFEF8002400
LT  stderr 	at java.base/java.lang.ThreadGroup.activeCount(ThreadGroup.java:377)
LT  stderr 	at java.base/java.lang.Thread.activeCount(Thread.java:2041)
LT  stderr 	at java.base/java.lang.String.length(String.java)
LT  stderr 	at java.base/java.lang.String.startsWith(String.java:3010)
LT  stderr JVMDUMP049I JIT dump notified all waiting threads of the current method to be compiled
LT  stderr 	at java.base/java.lang.String.startsWith(String.java:3053)
LT  stderr 	at java.base/java.lang.invoke.MethodType.fromDescriptor(MethodType.java:1212)
LT  stderr 	at java.base/java.lang.invoke.MethodType.fromMethodDescriptorString(MethodType.java:1195)
LT  stderr 	at java.base/java.lang.invoke.MethodTypeHelper.vmResolveFromMethodDescriptorString(MethodTypeHelper.java:256)
LT  stderr 	at java.base/java.lang.invoke.MethodHandleResolver.resolveInvokeDynamic(MethodHandleResolver.java:166)
LT  stderr 	at java.base/java.lang.Thread.getAllThreads(Thread.java:2695)
LT  stderr 	at java.base/java.lang.ThreadGroup.activeCount(ThreadGroup.java:377)
LT  stderr 	at java.base/java.lang.Thread.activeCount(Thread.java:2041)
LT  stderr 	at java.base/java.lang.String.length(String.java)JVMDUMP054I JIT dump is tracing the IL of the method on the crashed compilation thread
LT  stderr 
LT  stderr 	at java.base/java.lang.String.startsWith(String.java:3010)
LT  stderr 	at java.base/java.lang.String.startsWith(String.java:3053)
LT  stderr 	at java.base/java.lang.invoke.MethodType.fromDescriptor(MethodType.java:1212)
LT  stderr 	at java.base/java.lang.invoke.MethodType.fromMethodDescriptorString(MethodType.java:1195)
LT  stderr 	at java.base/java.lang.invoke.MethodTypeHelper.vmResolveFromMethodDescriptorString(MethodTypeHelper.java:256)
LT  stderr 	at java.base/java.lang.invoke.MethodHandleResolver.resolveInvokeDynamic(MethodHandleResolver.java:166)
LT  stderr 	at java.base/java.lang.Thread.getAllThreads(Thread.java:2695)
LT  stderr 	at java.base/java.lang.ThreadGroup.activeCount(ThreadGroup.java:377)
LT  stderr 	at java.base/java.lang.Thread.activeCount(Thread.java:2041)
LT  stderr 	at java.base/java.lang.String.length(String.java)
LT  stderr 	at java.base/java.lang.String.startsWith(String.java:3010)
LT  stderr 	at java.base/java.lang.String.startsWith(String.java:3053)
LT  stderr 	at java.base/java.lang.invoke.MethodType.fromDescriptor(MethodType.java:1212)
LT  stderr JVMDUMP048I JIT dump method being compiled is an ordinary method
LT  stderr 	at java.base/java.lang.invoke.MethodType.fromMethodDescriptorString(MethodType.java:1195)
LT  stderr 	at java.base/java.lang.invoke.MethodTypeHelper.vmResolveFromMethodDescriptorString(MethodTypeHelper.java:256)
LT  stderr 	at java.base/java.lang.invoke.MethodHandleResolver.resolveInvokeDynamic(MethodHandleResolver.java:166)
LT  stderr 	at java.base/java.lang.Thread.getAllThreads(Thread.java:2695)
LT  stderr 	at java.base/java.lang.ThreadGroup.activeCount(ThreadGroup.java:377)
LT  stderr 	at java.base/java.lang.Thread.activeCount(Thread.java:2041)
LT  stderr JVMDUMP053I JIT dump is recompiling java/lang/Thread.getAllThreads()[Ljava/lang/Thread;
LT  stderr 	at java.base/java.lang.String.length(String.java)
LT  stderr 	at java.base/java.lang.String.startsWith(String.java:3010)
LT  stderr 	at java.base/java.lang.String.startsWith(String.java:3053)
LT  stderr 	at java.base/java.lang.invoke.MethodType.fromDescriptor(MethodType.java:1212)
LT  stderr 	at java.base/java.lang.invoke.MethodType.fromMethodDescriptorString(MethodType.java:1195)
LT  stderr 	at java.base/java.lang.invoke.MethodTypeHelper.vmResolveFromMethodDescriptorString(MethodTypeHelper.java:256)
LT  stderr 	at java.base/java.lang.invoke.MethodHandleResolver.resolveInvokeDynamic(MethodHandleResolver.java:166)
LT  stderr 	at java.base/java.lang.Thread.getAllThreads(Thread.java:2695)
LT  stderr 	at java.base/java.lang.ThreadGroup.activeCount(ThreadGroup.java:377)
LT  stderr 	at java.base/java.lang.Thread.activeCount(Thread.java:2041)
LT  stderr 	at java.base/java.lang.String.length(String.java)
LT  stderr 	at java.base/java.lang.String.startsWith(String.java:3010)
LT  stderr 	at java.base/java.lang.String.startsWith(String.java:3053)
LT  stderr 	at java.base/java.lang.invoke.MethodType.fromDescriptor(MethodType.java:1212)
LT  stderr 	at java.base/java.lang.invoke.MethodType.fromMethodDescriptorString(MethodType.java:1195)
LT  stderr 	at java.base/java.lang.invoke.MethodTypeHelper.vmResolveFromMethodDescriptorString(MethodTypeHelper.java:256)
LT  stderr 	at java.base/java.lang.invoke.MethodHandleResolver.resolveInvokeDynamic(MethodHandleResolver.java:166)
LT  stderr 	at java.base/java.lang.Thread.getAllThreads(Thread.java:2695)
LT  stderr 	at java.base/java.lang.ThreadGroup.activeCount(ThreadGroup.java:377)
LT  stderr 	at java.base/java.lang.Thread.activeCount(Thread.java:2041)
LT  stderr 	at java.base/java.lang.String.length(String.java)
LT  stderr 	at java.base/java.lang.String.startsWith(String.java:3010)
LT  stderr 	at java.base/java.lang.String.startsWith(String.java:3053)
LT  stderr 	at java.base/java.lang.invoke.MethodType.fromDescriptor(MethodType.java:1212)
LT  stderr 	at java.base/java.lang.invoke.MethodType.fromMethodDescriptorString(MethodType.java:1195)
LT  stderr 	at java.base/java.lang.invoke.MethodTypeHelper.vmResolveFromMethodDescriptorString(MethodTypeHelper.java:256)
LT  stderr 	at java.base/java.lang.invoke.MethodHandleResolver.resolveInvokeDynamic(MethodHandleResolver.java:166)
LT  stderr JVMDUMP052I JIT dump recursive crash occurred on diagnostic thread
LT  stderr 	at java.base/java.lang.Thread.getAllThreads(Thread.java:2695)
LT  stderr 	at java.base/java.lang.ThreadGroup.activeCount(ThreadGroup.java:377)
LT  stderr 	at java.base/java.lang.Thread.activeCount(Thread.java:2041)
LT  stderr 	at java.base/java.lang.String.length(String.java)
LT  stderr JVMDUMP010I JIT dump written to /home/jenkins/workspace/Test_openjdknext_j9_extended.system_aarch64_linux_Personal_testList_2/aqa-tests/TKG/output_16546195099989/HCRLateAttachWorkload_previewEnabled_1/20220607-130041-HCRLateAttachWorkload/results/jitdump.20220607.130051.7807.0004.dmp
LT  stderr 	at java.base/java.lang.String.startsWith(String.java:3010)
LT  stderr 	at java.base/java.lang.String.startsWith(String.java:3053)
LT  stderr 	at java.base/java.lang.invoke.MethodType.fromDescriptor(MethodType.java:1212)
LT  stderr 	at java.base/java.lang.invoke.MethodType.fromMethodDescriptorString(MethodType.java:1195)
LT  stderr 	at java.base/java.lang.invoke.MethodTypeHelper.vmResolveFromMethodDescriptorString(MethodTypeHelper.java:256)
LT  stderr JVMDUMP013I Processed dump event "gpf", detail "".
LT  stderr 	at java.base/java.lang.invoke.MethodHandleResolver.resolveInvokeDynamic(MethodHandleResolver.java:166)
LT  stderr 	at java.base/java.lang.Thread.getAllThreads(Thread.java:2695)
LT  stderr 	at java.base/java.lang.ThreadGroup.activeCount(ThreadGroup.java:377)
LT  stderr 	at java.base/java.lang.Thread.activeCount(Thread.java:2041)
LT  stderr 	at java.base/java.lang.String.length(String.java)
LT  stderr 	at java.base/java.lang.String.startsWith(String.java:3010)
LT  stderr 	at java.base/java.lang.String.startsWith(String.java:3053)
LT  stderr 	at java.base/java.lang.invoke.MethodType.fromDescriptor(MethodType.java:1212)
LT  stderr 	at java.base/java.lang.invoke.MethodType.fromMethodDescriptorString(MethodType.java:1195)
LT  stderr 	at java.base/java.lang.invoke.MethodTypeHelper.vmResolveFromMethodDescriptorString(MethodTypeHelper.java:256)
LT  stderr 	at java.base/java.lang.invoke.MethodHandleResolver.resolveInvokeDynamic(MethodHandleResolver.java:166)
LT  stderr 	at java.base/java.lang.Thread.getAllThreads(Thread.java:2695)
LT  stderr 	at java.base/java.lang.ThreadGroup.activeCount(ThreadGroup.java:377)
LT  stderr 	at java.base/java.lang.Thread.activeCount(Thread.java:2041)
**FAILED** at step 3 (Wait for Agent to complete). Expected return value=0 Actual=1 at /home/jenkins/workspace/Test_openjdknext_j9_extended.system_aarch64_linux_Personal_testList_2/aqa-tests/TKG/../TKG/output_16546195099989/HCRLateAttachWorkload_previewEnabled_1/20220607-130041-HCRLateAttachWorkload/execute.pl line 155.
STF 13:00:57.670 - **FAILED** execute script failed. Expected return value=0 Actual=1
STF 13:00:57.670 - 
STF 13:00:57.670 - ====================   T E A R D O W N   ====================
STF 13:00:57.670 - Running teardown: perl /home/jenkins/workspace/Test_openjdknext_j9_extended.system_aarch64_linux_Personal_testList_2/aqa-tests/TKG/../TKG/output_16546195099989/HCRLateAttachWorkload_previewEnabled_1/20220607-130041-HCRLateAttachWorkload/tearDown.pl
STF 13:00:57.753 - TEARDOWN stage completed
STF 13:00:57.756 - 
STF 13:00:57.756 - =====================   R E S U L T S   =====================
STF 13:00:57.756 - Stage results:
STF 13:00:57.756 -   setUp:     pass
STF 13:00:57.756 -   execute:  *fail*
STF 13:00:57.756 -   teardown:  pass
STF 13:00:57.756 - 
STF 13:00:57.756 - Overall result: **FAILED**

HCRLateAttachWorkload_previewEnabled_1_FAILED

The text was updated successfully, but these errors were encountered:

tajila · 2022-10-17T19:50:52Z

@JasonFengJ9 is this still failing

JasonFengJ9 · 2022-10-17T20:06:25Z

The test has been excluded, launched an internal grinder

LT  Entering the loop that will replace dead threads with live ones until we run out of time.
LT  stderr Exception in thread "Thread-7" java.lang.StackOverflowError: operating system stack overflow
LT  stderr 	at java.base/java.lang.Thread.getAllThreads(Thread.java:2702)
LT  stderr 	at java.base/java.lang.ThreadGroup.activeCount(ThreadGroup.java:377)
LT  stderr 	at java.base/java.lang.Thread.activeCount(Thread.java:2048)
LT  stderr 	at java.base/java.lang.String.replace(String.java)
LT  stderr 	at java.base/sun.invoke.util.BytecodeDescriptor.parseSig(BytecodeDescriptor.java:91)

Similar errors were reported on hotspot JDK19 as well HCRLateAttachWorkload_previewEnabled_0 and HCRLateAttachWorkload_previewEnabled_1 failing with StackOverflow Errors

JasonFengJ9 · 2022-10-18T12:51:55Z

The StackOverflowError is still seen in OpenJ9 10x internal grinder

07:50:48  openjdk version "19-beta" 2022-09-20
07:50:48  IBM Semeru Runtime Open Edition 19+36-202210060535 (build 19-beta+36-202210060535)
07:50:48  Eclipse OpenJ9 VM 19+36-202210060535 (build master-c3faa0a56, JRE 19 Linux ppc64le-64-Bit Compressed References 20221006_50 (JIT enabled, AOT enabled)
07:50:48  OpenJ9   - c3faa0a56
07:50:48  OMR      - 773e9b51a
07:50:48  JCL      - b9e3daf582 based on jdk-19+36)

RI JDK19 passed 10x internal grinder

08:07:44  OpenJDK Runtime Environment Temurin-19+36 (build 19+36)
08:07:44  OpenJDK 64-Bit Server VM Temurin-19+36 (build 19+36, mixed mode, sharing)

FYI @tajila

tajila · 2022-10-18T13:03:48Z

is it all platforms?

tajila · 2022-10-18T13:10:27Z

Actually, this makes me think it may be a jit issue

LT  stderr 	at java.base/java.lang.ThreadGroup.activeCount(ThreadGroup.java:377)
LT  stderr 	at java.base/java.lang.Thread.activeCount(Thread.java:2041)
LT  stderr 	at java.base/java.lang.String.length(String.java) //<-- shouldnt be possible for length() to call activeCount()
LT  stderr 	at java.base/java.lang.String.startsWith(String.java:3010)

@JasonFengJ9 Can you see if this reproduces with -Xint

JasonFengJ9 · 2022-10-18T13:18:12Z

Launched 10x -Xint internal grinder

JasonFengJ9 · 2022-10-18T21:01:28Z

Tried -Xint as EXTRA_OPTIONS, and also added mode 100 at https://github.com/JasonFengJ9/aqa-tests/blob/8b3a9959d7988a26673b7dce2135a9ebdb4b873b/system/otherLoadTest/playlist.xml#L294-L296

HCRLateAttachWorkload_previewEnabled_0_FAILED w/ StackOverflowError, from the javacore.20221018.131051.1747.0002.txt file, it wasn't running at -Xint.

@llxia @Mesbah-Alam any insights on why the test won't take the grinder option or follow the mode specified?

llxia · 2022-10-18T23:15:15Z

From the console output, Mode100 did not get passed in:

14:24:18  variation: Mode100
14:24:18  JVM_OPTIONS: -Xdump:system:events=user

...
... -java-args='-Xdump:system:events=user'

This issue affects all system tests with Mode. It seems no Modes were passed in the system tests for the past month. The issue was introduced between Sep 18 (good build) - Sep 25 (bad build).

good build

19:01:09  variation: Mode122
19:01:09  JVM_OPTIONS:  -Xgcpolicy:optavgpause -Xjit:count=0,optlevel=warm,gcOnResolve,rtResolve -Xnocompressedrefs
...-java-args=' -Xgcpolicy:optavgpause -Xjit:count=0,optlevel=warm,gcOnResolve,rtResolve -Xnocompressedrefs  -Xdump:system:events=user'

bad build

16:39:09  variation: Mode122
16:39:09  JVM_OPTIONS: -Xdump:system:events=user
...-java-args='-Xdump:system:events=user'

I suspect https://github.com/adoptium/aqa-tests/blob/d1c960685d0e4f043a11675608048eb7bd682f60/system/system.mk#L63 is incorrect. I think it is related to PR adoptium/aqa-tests#3972

Related: eclipse-openj9/openj9#15250 Signed-off-by: Lan Xia <[email protected]>

Related: eclipse-openj9/openj9#15250 Signed-off-by: Lan Xia <[email protected]> Signed-off-by: Lan Xia <[email protected]>

Related: eclipse-openj9/openj9#15250 Signed-off-by: Lan Xia <[email protected]> Signed-off-by: Lan Xia <[email protected]> Signed-off-by: Lan Xia <[email protected]>

llxia · 2022-10-19T00:32:47Z

@JasonFengJ9 the above issue is resolved via adoptium/aqa-tests#4052. Please try to rebase your branch and try again. Thanks

JasonFengJ9 · 2022-10-19T12:07:52Z

StackOverflowError: operating system stack overflow also occurred at HCRLateAttachWorkload_previewEnabled_0_FAILED Mode100 which is -Xint.
FYI @tajila

tajila · 2022-10-20T01:19:40Z

I added -Xdump:system:events=throw,filter=*StackOverflowError and now the stackoverflow is gone, but it hangs instead.

https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/28807/tapResults/

JasonFengJ9 · 2022-10-20T12:53:03Z

With -Xtrace:trigger=method{java/lang/StackOverflowError,sysdump}

08:16:20  AG  stderr core file generated - /home/jenkins/workspace/Grinder/aqa-tests/TKG/output_16662681658036/HCRLateAttachWorkload_previewEnabled_0/20221020-051607-HCRLateAttachWorkload/results/core.20221020.051619.12033.0001.dmp

08:16:22  HCRLateAttachWorkload_previewEnabled_0_FAILED

Result file
FYI @tajila

tajila · 2022-10-20T14:09:59Z

I see something very strange here:

The bytecodes for String::length() is:

public int length();
    Code:
       0: aload_0
       1: getfield      #50                 // Field value:[B
       4: arraylength
       5: aload_0
       6: invokevirtual #355                // Method coder:()B
       9: ishr
      10: ireturn

but when I look at the runtime version its

> !bytecodes 0x00000000004994D0
  Name: length
  Signature: ()I
  Access Flags (50001): public
  Internal Attribute Flags:
  Max Stack: 12
  Argument Count: 1
  Temp Count: 0

    0 invokestatic 31 java/lang/Thread.activeCount()I
    3 pop
    4 aload0getfield
    5 getfield 27 java/lang/String.value [B
    8 arraylength
    9 aload0
   10 invokevirtual 139 java/lang/String.coder()B
   13 ishr
   14 return1

So there are some extra bytecodes at the start

tajila · 2022-10-20T14:12:23Z

The class is retransformed, the original bytecodes before the transformation is correct

> !bytecodes 0x00000000000A4F50
  Name: length
  Signature: ()I
  Access Flags (50001): public
  Internal Attribute Flags:
  Max Stack: 2
  Argument Count: 1
  Temp Count: 0

    0 aload0getfield
    1 getfield 18 java/lang/String.value [B
    4 arraylength
    5 aload0
    6 invokevirtual 117 java/lang/String.coder()B
    9 ishr
   10 return1

tajila · 2022-10-20T14:15:27Z

I dont know much about this test, but if that transformation is intended then the StackOverflow makes sense.

tajila · 2022-10-20T14:20:51Z

Based on the README, https://github.com/adoptium/aqa-systemtest/blob/0eb9f57f198fe194c070295c261e8bafeaed08aa/openjdk.test.debugging/docs/HCR_README.md

The test randomly adds a method to String API, I think in this particular case there is nothing we can do about it because it is infinitely recursive. I think we can close and re-enable the test, with a different seed value we shouldn't see the failure.

tajila · 2022-11-03T18:33:58Z

@JasonFengJ9 can you please unexclude the test

JasonFengJ9 · 2022-11-07T13:27:59Z

HCRLateAttachWorkload_previewEnabled has been un-excluded.

pshipton · 2022-11-07T14:28:44Z

The test is failing on all platforms, https://openj9-jenkins.osuosl.org/job/Pipeline-Build-Test-JDK19/194/
It needs to be excluded again, today if possible.

Example failure,
https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_extended.system_aarch64_linux_Nightly/43

LT  stderr *** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message transform method call failed at src/java.instrument/share/native/libinstrument/JPLISAgent.c line: 873
LT  stderr Exception in thread "Thread-11" java/lang/StackOverflowError: operating system stack overflow
LT  stderr 	at java/lang/Thread.getAllThreads ([email protected]/Thread.java:2704)
LT  stderr 	at java/lang/ThreadGroup.activeCount ([email protected]/ThreadGroup.java:377)
LT  stderr 	at java/lang/Thread.activeCount ([email protected]/Thread.java:2046)
LT  stderr 	at java/lang/String.startsWith ([email protected]/String.java:4294967295)
LT  stderr 	at java/lang/invoke/MethodType.fromDescriptor ([email protected]/MethodType.java:1212)
LT  stderr 	at java/lang/invoke/MethodType.fromMethodDescriptorString ([email protected]/MethodType.java:1195)
LT  stderr 	at java/lang/invoke/MethodTypeHelper.vmResolveFromMethodDescriptorString ([email protected]/MethodTypeHelper.java:256)
LT  stderr 	at java/lang/invoke/MethodHandleResolver.resolveInvokeDynamic ([email protected]/MethodHandleResolver.java:168)
LT  stderr 	at java/lang/Thread.getAllThreads ([email protected]/Thread.java:2704)
LT  stderr 	at java/lang/ThreadGroup.activeCount ([email protected]/ThreadGroup.java:377)
LT  stderr 	at java/lang/Thread.activeCount ([email protected]/Thread.java:2046)
LT  stderr 	at java/lang/String.startsWith ([email protected]/String.java:4294967295)
LT  stderr 	at java/lang/invoke/MethodType.fromDescriptor ([email protected]/MethodType.java:1212)
LT  stderr 	at java/lang/invoke/MethodType.fromMethodDescriptorString ([email protected]/MethodType.java:1195)
LT  stderr 	at java/lang/invoke/MethodTypeHelper.vmResolveFromMethodDescriptorString ([email protected]/MethodTypeHelper.java:256)
LT  stderr 	at java/lang/invoke/MethodHandleResolver.resolveInvokeDynamic ([email protected]/MethodHandleResolver.java:168)
LT  stderr 	at java/lang/Thread.getAllThreads ([email protected]/Thread.java:2704)

JasonFengJ9 · 2022-11-07T15:25:14Z

Created adoptium/aqa-tests#4121

JasonFengJ9 · 2022-11-07T16:32:30Z

Segmentation error vmState=0x00000000 at an internal build(win16x64rt2-2)

[2022-11-05T21:25:02.244Z] variation: Mode150
[2022-11-05T21:25:02.681Z] JVM_OPTIONS:  -XX:+UseCompressedOops 

[2022-11-05T21:25:17.971Z] LT  14:25:17.847 - Starting thread. Suite=0 thread=2
[2022-11-05T21:25:19.565Z] LT  stderr Unhandled exception
[2022-11-05T21:25:19.566Z] LT  stderr Type=Segmentation error vmState=0x00000000
[2022-11-05T21:25:19.566Z] LT  stderr Windows_ExceptionCode=c0000005 J9Generic_Signal=00000004 ExceptionAddress=00007FFBAC436F38 ContextFlags=0010005f
[2022-11-05T21:25:19.566Z] LT  stderr Handler1=00007FFBAC4EC230 Handler2=00007FFBB23EAA50 =0000000000000000
[2022-11-05T21:25:19.566Z] LT  stderr RDI=0000002DC8F5F168 RSI=00000007FFD38D38 RAX=0000000000435B00 RBX=0000014DB24D3FC0
[2022-11-05T21:25:19.566Z] LT  stderr RCX=0000000040000000 RDX=000000000065BF58 R8=0000002DC8F5F178 R9=0000002DC8F5F190
[2022-11-05T21:25:19.566Z] LT  stderr R10=00007FFBAC5DB374 R11=0000002DC8F5F180 R12=000000000065BF50 R13=0000000000000004
[2022-11-05T21:25:19.566Z] LT  stderr R14=0000002DC8F5F170 R15=00000007FFD15198
[2022-11-05T21:25:19.566Z] LT  stderr RIP=00007FFBAC436F38 RSP=0000002DC8F5EE60 RBP=FFFA71ABFFFA719A EFLAGS=0000000000010217
[2022-11-05T21:25:19.566Z] LT  stderr FS=0053 ES=002B DS=002B
[2022-11-05T21:25:19.566Z] LT  stderr XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-05T21:25:19.566Z] LT  stderr XMM1 00000000003b9560 (f: 3904864.000000, d: 1.929259e-317)
[2022-11-05T21:25:19.566Z] LT  stderr XMM2 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-05T21:25:19.566Z] LT  stderr XMM3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-05T21:25:19.566Z] LT  stderr XMM4 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-05T21:25:19.566Z] LT  stderr XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-05T21:25:19.566Z] LT  stderr XMM6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-05T21:25:19.566Z] LT  stderr XMM7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-05T21:25:19.566Z] LT  stderr XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-05T21:25:19.566Z] LT  stderr XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-05T21:25:19.566Z] LT  stderr XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-05T21:25:19.566Z] LT  stderr XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-05T21:25:19.566Z] LT  stderr XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-05T21:25:19.566Z] LT  stderr XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-05T21:25:19.566Z] LT  stderr XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-05T21:25:19.566Z] LT  stderr XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-05T21:25:19.566Z] LT  stderr Module=C:\Users\jenkins\workspace\Test_openjdk19_j9_extended.system_x86-64_windows_testList_0\openjdkbinary\j2sdk-image\bin\default\j9vm29.dll
[2022-11-05T21:25:19.566Z] LT  stderr Module_base_address=00007FFBAC420000 Offset_in_DLL=0000000000016f38
[2022-11-05T21:25:19.566Z] LT  stderr Target=2_90_20221018_69 (Windows Server 2016 10.0 build 14393)
[2022-11-05T21:25:19.566Z] LT  stderr CPU=amd64 (8 logical CPUs) (0x3fff78000 RAM)
[2022-11-05T21:25:19.566Z] LT  stderr ----------- Stack Backtrace -----------
[2022-11-05T21:25:19.566Z] STF 14:25:19.245 - Found dump at: C:\Users\jenkins\workspace\Test_openjdk19_j9_extended.system_x86-64_windows_testList_0\aqa-tests\TKG\output_16676811979167\HCRLateAttachWorkload_previewEnabled_0\20221105-142503-HCRLateAttachWorkload\results\core.20221105.142519.4344.0001.dmp
[2022-11-05T21:25:19.566Z] LT  stderr (0x00007FFBAC436F38 [j9vm29+0x16f38])
[2022-11-05T21:25:19.566Z] LT  stderr (0xFFFFFFFFFFFFFFFD)
[2022-11-05T21:25:19.566Z] LT  stderr (0x0000002DC8F5F180)
[2022-11-05T21:25:19.566Z] LT  stderr (0x0000002DC8F5F170)
[2022-11-05T21:25:19.566Z] LT  stderr (0x0000014DBDE59A68)
[2022-11-05T21:25:19.566Z] LT  stderr (0x0000014DB2528BB0)
[2022-11-05T21:25:19.566Z] LT  stderr (0x0000014DB2513EC0)
[2022-11-05T21:25:19.566Z] LT  stderr ---------------------------------------
[2022-11-05T21:25:19.566Z] LT  stderr JVMDUMP039I Processing dump event "gpf", detail "" at 2022/11/05 14:25:19 - please wait.

[2022-11-05T21:25:51.048Z] HCRLateAttachWorkload_previewEnabled_0_FAILED

JasonFengJ9 added test failure project:loom Used to track Project Loom related work jdk19 labels Jun 7, 2022

pshipton added this to the Release 0.34 (Java 19) milestone Jun 8, 2022

JasonFengJ9 mentioned this issue Jun 9, 2022

JDK19/18 excludes adoptium/aqa-tests#3750

Merged

JasonFengJ9 added the test excluded label Jun 9, 2022

llxia added a commit to llxia/aqa-tests that referenced this issue Oct 18, 2022

Revert to JAVA_ARGS

55224fd

Related: eclipse-openj9/openj9#15250 Signed-off-by: Lan Xia <[email protected]>

llxia mentioned this issue Oct 18, 2022

Revert to JAVA_ARGS adoptium/aqa-tests#4052

Merged

smlambert pushed a commit to adoptium/aqa-tests that referenced this issue Oct 19, 2022

Revert to JAVA_ARGS (#4052)

fa73e7b

Related: eclipse-openj9/openj9#15250 Signed-off-by: Lan Xia <[email protected]> Signed-off-by: Lan Xia <[email protected]>

llxia added a commit to llxia/aqa-tests that referenced this issue Oct 19, 2022

Revert to JAVA_ARGS (adoptium#4052)

e7354a5

Related: eclipse-openj9/openj9#15250 Signed-off-by: Lan Xia <[email protected]> Signed-off-by: Lan Xia <[email protected]>

llxia mentioned this issue Oct 19, 2022

(v0.9.5-release) Revert to JAVA_ARGS (#4052) adoptium/aqa-tests#4053

Merged

tajila removed project:loom Used to track Project Loom related work jdk19 labels Oct 20, 2022

tajila modified the milestones: Java 19, Release 0.36 (Java 8, 11, 17, 19) Jan refresh Oct 20, 2022

tajila modified the milestones: Release 0.36 (Java 8, 11, 17, 19) Jan refresh, Release 0.38 (Java 8, 11, 17, 20) Apr refresh Nov 3, 2022

JasonFengJ9 mentioned this issue Nov 3, 2022

Enable tests JSR166TestCase.java & HCRLateAttachWorkload_previewEnabled adoptium/aqa-tests#4116

Merged

JasonFengJ9 removed the test excluded label Nov 3, 2022

JasonFengJ9 mentioned this issue Nov 7, 2022

JDK19 re-exclude HCRLateAttachWorkload_previewEnabled adoptium/aqa-tests#4121

Merged

JasonFengJ9 added the test excluded label Nov 7, 2022

tajila modified the milestones: Release 0.38 (Java 8, 11, 17) Apr refresh, Backlog Feb 28, 2023

psoujany mentioned this issue Aug 20, 2024

Disable HCRLateAttachWorkload_previewEnabled on z/OS for JDK21 adoptium/aqa-tests#5514

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

JDK19 HCRLateAttachWorkload_previewEnabled_1_FAILED StackOverflowError: operating system stack overflow #15250

JDK19 HCRLateAttachWorkload_previewEnabled_1_FAILED StackOverflowError: operating system stack overflow #15250

JasonFengJ9 commented Jun 7, 2022 •

edited

Loading

tajila commented Oct 17, 2022

JasonFengJ9 commented Oct 17, 2022 •

edited

Loading

JasonFengJ9 commented Oct 18, 2022

tajila commented Oct 18, 2022

tajila commented Oct 18, 2022

JasonFengJ9 commented Oct 18, 2022 •

edited

Loading

JasonFengJ9 commented Oct 18, 2022

llxia commented Oct 18, 2022 •

edited

Loading

llxia commented Oct 19, 2022

JasonFengJ9 commented Oct 19, 2022

tajila commented Oct 20, 2022

JasonFengJ9 commented Oct 20, 2022 •

edited

Loading

tajila commented Oct 20, 2022

tajila commented Oct 20, 2022

tajila commented Oct 20, 2022

tajila commented Oct 20, 2022

tajila commented Nov 3, 2022

JasonFengJ9 commented Nov 7, 2022

pshipton commented Nov 7, 2022 •

edited

Loading

JasonFengJ9 commented Nov 7, 2022

JasonFengJ9 commented Nov 7, 2022

JDK19 HCRLateAttachWorkload_previewEnabled_1_FAILED StackOverflowError: operating system stack overflow #15250

JDK19 HCRLateAttachWorkload_previewEnabled_1_FAILED StackOverflowError: operating system stack overflow #15250

Comments

JasonFengJ9 commented Jun 7, 2022 • edited Loading

Failure link

Optional info

Failure output (captured from console output)

tajila commented Oct 17, 2022

JasonFengJ9 commented Oct 17, 2022 • edited Loading

JasonFengJ9 commented Oct 18, 2022

tajila commented Oct 18, 2022

tajila commented Oct 18, 2022

JasonFengJ9 commented Oct 18, 2022 • edited Loading

JasonFengJ9 commented Oct 18, 2022

llxia commented Oct 18, 2022 • edited Loading

llxia commented Oct 19, 2022

JasonFengJ9 commented Oct 19, 2022

tajila commented Oct 20, 2022

JasonFengJ9 commented Oct 20, 2022 • edited Loading

tajila commented Oct 20, 2022

tajila commented Oct 20, 2022

tajila commented Oct 20, 2022

tajila commented Oct 20, 2022

tajila commented Nov 3, 2022

JasonFengJ9 commented Nov 7, 2022

pshipton commented Nov 7, 2022 • edited Loading

JasonFengJ9 commented Nov 7, 2022

JasonFengJ9 commented Nov 7, 2022

JasonFengJ9 commented Jun 7, 2022 •

edited

Loading

JasonFengJ9 commented Oct 17, 2022 •

edited

Loading

JasonFengJ9 commented Oct 18, 2022 •

edited

Loading

llxia commented Oct 18, 2022 •

edited

Loading

JasonFengJ9 commented Oct 20, 2022 •

edited

Loading

pshipton commented Nov 7, 2022 •

edited

Loading