High CPU usage caused by TrackerProcessor from apminsight

High CPU usage caused by TrackerProcessor from apminsight

Is it normal for the TrackerProcessor thread of the apminsight Java agent to consume a high CPU usage?

My current setting is having 2 Java processes running on a AIX machine with 2 CPU cores. Both of the Java processes have specified apminsight as the Java agent and as checked from the apminsight_agent.log.0 and the OpManager, they're working properly.

The high CPU usage occurs frequently (more than 5 times a day) on one of the process. When it happens on both cores, the CPU usage will be over 90%.

I've used the tprof tool to do CPU profiling when the CPU usage is over 40% and identified the TrackerProcessor thread as the culprit.

As seen from the stack trace of the TrackerProcessor, it was obviously doing something related to Regular Expression that could be costly. But I'm not sure how and why apminsight performs inefficient RegExp searches that use high CPU usage from several minutes to over ten minutes.

Here is the recorded stack trace of the TrackerProcessor thread(various IDs are masked to preserve privacy) when the CPU usage was above 40%:

3XMTHREADINFO      "TrackerProcessor" (TID:0xXXXXXXXX, sys_thread_t:0xXXXXXXXX, state:CW, native ID:0x00XXXXXX) prio=5
4XESTACKTRACE          at java/util/regex/Pattern$Curly.match0(Pattern.java:4259(Compiled Code))
4XESTACKTRACE          at java/util/regex/Pattern$Curly.match(Pattern.java:4233(Compiled Code))
4XESTACKTRACE          at java/util/regex/Pattern$Start.match(Pattern.java:3055(Compiled Code))
4XESTACKTRACE          at java/util/regex/Matcher.search(Matcher.java:1108(Compiled Code))
4XESTACKTRACE          at java/util/regex/Matcher.find(Matcher.java:544(Compiled Code))
4XESTACKTRACE          at com/manageengine/apminsight/agent/trackers/db/QueryParser.getQueryInfo(QueryParser.java:93(Compiled Code))
4XESTACKTRACE          at com/manageengine/apminsight/agent/trackers/db/StatementTracker.getTrackerSignature(StatementTracker.java:306(Compiled Code))
4XESTACKTRACE          at com/manageengine/apminsight/agent/sequence/SequenceTrace.buildTrace(SequenceTrace.java:91(Compiled Code))
4XESTACKTRACE          at com/manageengine/apminsight/agent/sequence/SequenceTrace.buildTrace(SequenceTrace.java:89(Compiled Code))
4XESTACKTRACE          at com/manageengine/apminsight/agent/sequence/SequenceTrace.buildTrace(SequenceTrace.java:89(Compiled Code))
4XESTACKTRACE          at com/manageengine/apminsight/agent/sequence/SequenceTrace.buildTrace(SequenceTrace.java:89(Compiled Code))
4XESTACKTRACE          at com/manageengine/apminsight/agent/sequence/SequenceTrace.buildTrace(SequenceTrace.java:89(Compiled Code))
4XESTACKTRACE          at com/manageengine/apminsight/agent/sequence/SequenceTrace.buildTrace(SequenceTrace.java:89(Compiled Code))
4XESTACKTRACE          at com/manageengine/apminsight/agent/sequence/SequenceTrace.formTreeAndGetRoot(SequenceTrace.java:79)
4XESTACKTRACE          at com/manageengine/apminsight/agent/sequence/SequenceTrace.constructSeqTrace(SequenceTrace.java:73)
4XESTACKTRACE          at com/manageengine/apminsight/agent/sequence/SequenceTrace.(SequenceTrace.java:50)
4XESTACKTRACE          at com/manageengine/apminsight/agent/sequence/SequenceTraceStore.constructTraceAndPut(SequenceTraceStore.java:92)
4XESTACKTRACE          at com/manageengine/apminsight/agent/trackers/TrackerProcessor.produceTrace(TrackerProcessor.java:155(Compiled Code))
4XESTACKTRACE          at com/manageengine/apminsight/agent/trackers/TrackerProcessor.doExtract(TrackerProcessor.java:112(Compiled Code))
4XESTACKTRACE          at com/manageengine/apminsight/agent/trackers/TrackerProcessor.doWork(TrackerProcessor.java:72)
4XESTACKTRACE          at com/manageengine/apminsight/agent/trackers/TrackerProcessor.run(TrackerProcessor.java:58)
                New to ADManager Plus?

                  New to ADSelfService Plus?