Feature request: Listener API to eavesdrop on trace events

[eostermueller]

Expose a set of interfaces that allows a programmer to write a program (called the Implementor) that receives all InTrace trace events.

The classes java.util.Observable and java.util.Observer could be used as a model for some, but not all, of the interaction between InTrace and an Implementor.

Each Implementor would run in the same JVM as the InTrace GUI.
Some kind of configuration would store class names of all Implementors. -D parameters would be acceptable for a "first attempt." Once InTrace has all the configuration for a single Implementor, that Implementor will be considered "registered"

The interface must allow the InTrace GUI to ask the Implementor for a human-friendly name that will be used for display in the InTrace GUI.

The interface must allow the InTrace GUI to ask the Implementor for an Icon unique for that Implementor. If none is provided, a default (but unique one) will be used.

At GUI startup time, InTrace will ask each registered "Implementor" for a list of
-- classes to be instrumented. Include/exclude functionality should be incorporated.
-- Filters to be used. Include/exclude functionality should be incorporated.
Once the above and any other initialization is complete, InTrace GUI should invoke some kind of "startup" method on each Implementor.

When the end user closes the GUI, InTrace should notify each Implementor of the shutdown request.

Implementor display/output

Each implementor will have its own "section" in the InTrace GUI.
When InTrace UI starts, the end user will need a menu option to open a particular Implementor with a particular host/port.

Example Implementors

Each of these ideas imitates functionality found in commercially offered APM (Application Performance Monitoring) tools, like CompuWare DynaTrace and CA Introscope.

1. Facility to display Gantt charts for each request of a SOA application being traced. Gantt chart will display response time for each traced method. Important to show "nested" methods.
2. Response time/throughput graphs of instrumented classes
3. List of all SQL statements executed by the application being traced.

Option 1), above, could be expanded to show the following for each SOA request:
a) input xml
b) response xml
c) all SQL requests made during the request
d) all exceptions thrown during the request.

[mchr3k]

Hi eostermueller,

Thanks for this feature suggestion (and the other two you raised). I am pleased to see that you are interested in InTrace.

Can you give me a bit more background about yourself and these suggestion you have raised? I would be interested to know all of the following.

• Have you used InTrace much or have you just had a quick play?
• What are you using InTrace for? Personal/commercial projects?
• Are you interested in getting involved in the development of these features or are you hoping I will implement these?
• Do you have a timeline in mind? This month? This year?

[eostermueller]

mchr3k wrote:

Have you used InTrace much or have you just had a quick play?
Just a quick play. I verified that I can capture SQL statements from the
Java PetStore application.
I like the way you can dynamically add/remove the instrumentation and the
filters.

What are you using InTrace for? Personal/commercial projects?
Both commercial and personal, actually. The company I work for relies
heavily on big-dollar APM tools for a number of things.
InTrace can't completely replace those tools, but it could reduce the
number of licenses we need for those tools.
What kind of big dollar tools? CompuWare DynaTrace and CA Introscope.
They cost tens of thousands of USD.
So that is the business side....on the personal side, I'm toying with the
idea of writing a little book on performance testing.
That book would use some techniques that partially rely on the InTrace
feature requests I've made.

Are you interested in getting involved in the development of these
features or are you hoping I will implement these?
Do you have a timeline in mind? This month? This year?
Yes, I'm ready to get involved.
When? How about tomorrow? But I do have a few concerns, before getting
started.

a) I need to run some tests with an IBM jvm on linux. If I can get those
tests to work (hopefully by tomorrow), I'll be ready start helping.

b) If InTrace is monitoring a remote application, it seems that a separate
network request is made for every trace event. Is that right? -- I haven't
looked closely enough at the code to see. If that is the case, I don't
think that kind of model will scale when medium amounts of load is applied.
Perhaps we could consider some kind of burst mode where trace events
collect in a queue and are sent to the GUI every X seconds?
CA Introscope, for instance, has a 15 second delay. I'd be happy with 5.

c) The systems we deploy have multiple, identical JVMs that process traffic
(often, but not always, deployed on separate machines). I'd like to see a
single InTrace GUI capture events from multiple JVMs. Obviously, the
traffic from each would have to be somehow labeled so the user can
distinguish traffic from each JVM.

d) You have support for millisecond response time. I'd really like to
provide support (perhaps optionally) for nanoTime
http://docs.oracle.com/javase/1.5.0/docs/api/java/lang/System.html#nanoTime()

Only item a) is a show stopper for me. I need to make sure that is
resolved before investing any time. I'd like to hear your opinion on the
other two. If needed, I'd be ok to add support for b) and c) and d) at a
later date.

Since this is your code base, perhaps you could propose a design for these
enhancements.
We could agree on the design then you could slice off a few pieces for me
to work on.
What do you think?

So I mentioned these three "plugins" or "implementors":

1. Facility to display Gantt charts for each request of a SOA application
   being traced. Gantt chart will display response time for each traced
   method. Important to show "nested" methods.
2. Response time/throughput graphs of instrumented classes
3. List of all SQL statements executed by the application being traced.

For plugin 1), I've found two decent libraries we could use to create Gantt
charts:
http://www.jfree.org/jfreechart/
http://www.simile-widgets.org/timeline/

The second tool, Simile-Timeline has better scrolling/navigation, but it is
in JavaScript which would make things messy.
So I'd like to start with JFreeChart.

In regards to plugin 2). I'm sick of tools that use "averages" to plot
response time. Averages hide outliers that are often very important.
So instead, I'd like to somehow display the counts of responses in various
time ranges.

For instance, say we had the following response times for 3 requests in a
single 5 second interval:

500ms
95ms
105ms

The average of these numbers is about 233 -- which doesn't some close to
representing the 500, the 95 or the 105ms.

Instead, I'd like the graph to somehow show the counts of fast and slow
responses in various ranges, something like this:

2 requests faster than 128ms
0 requests between 128 and 256ms
1 request between 256 and 512ms.

The ranges will need to be configurable, but this gives you the idea. We'd
have to work out whether we used stacked histograms, or perhaps color to
show these counts.

Let me know what you think.

--Erik

On Mon, Jan 2, 2012 at 2:36 PM, mchr3k <
[email protected]

wrote:

Hi eostermueller,

Thanks for this feature suggestion (and the other two you raised). I am
pleased to see that you are interested in InTrace.

Can you give me a bit more background about yourself and these suggestion
you have raised? I would be interested to know all of the following.

• Have you used InTrace much or have you just had a quick play?
• What are you using InTrace for? Personal/commercial projects?
• Are you interested in getting involved in the development of these
  features or are you hoping I will implement these?
• Do you have a timeline in mind? This month? This year?

---

Reply to this email directly or view it on GitHub:
#12 (comment)

[eostermueller]

mchr3k,

Rats. I'm having problems getting an IBM jdk working with InTrace.
When I start your example.TraceExample program with the javaagent
parameter, I'm getting the stack trace below.
Suggestions?
I really need to have IBM jdk support before I can help with anything else.
It looks like this is the source that is throwing the error:

http://www.docjar.com/html/api/sun/instrument/InstrumentationImpl.java.html*

---

How much InTrace functionality relies on "retransforming" ?

Perhaps removing InTrace support for "on the fly" instrumentation on IBM
jdk would help?

--Erik

---

Buildfile: build.xml****

---

init:****

---

run:****

 [java] ## Loaded InTrace Agent.****

 [java] Exception in thread "main"

java.lang.UnsupportedOperationException: cannot get the capability,
performing dispose of the retransforming environment****

 [java]     at

sun.instrument.InstrumentationImpl.isRetransformClassesSupported0(Native
Method)****

 [java]     at

sun.instrument.InstrumentationImpl.isRetransformClassesSupported(InstrumentationImpl.java:124)

---

 [java]     at

sun.instrument.InstrumentationImpl.addTransformer(InstrumentationImpl.java:79)

---

 [java]     at org.intrace.agent.AgentInit.initialize(AgentInit.java:41)

---

 [java]     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native

Method)****

 [java]     at

sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:48)

---

 [java]     at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)

---

 [java]     at java.lang.reflect.Method.invoke(Method.java:600)****

 [java]     at org.intrace.agent.Agent.initialize(Agent.java:66)****

 [java]     at org.intrace.agent.Agent.premain(Agent.java:26)****

 [java]     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native

Method)****

 [java]     at

sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:48)

---

 [java]     at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)

---

 [java]     at java.lang.reflect.Method.invoke(Method.java:600)****

 [java]     at

sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:335)

---

 [java]     at

sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:350)

---

 [java] Exception in thread "main"

java.lang.reflect.InvocationTargetException****

 [java]     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native

Method)****

 [java]     at

sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:48)

---

 [java]     at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)

---

 [java]     at java.lang.reflect.Method.invoke(Method.java:600)****

 [java]     at

sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:335)

---

 [java]     at

sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:350)

---

 [java] Caused by: java.lang.RuntimeException:

java.lang.reflect.InvocationTargetException****

 [java]     at org.intrace.agent.Agent.initialize(Agent.java:70)****

 [java]     at org.intrace.agent.Agent.premain(Agent.java:26)****

 [java]     ... 6 more****

 [java] Caused by: java.lang.reflect.InvocationTargetException****

 [java]     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native

Method)****

 [java]     at

sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:48)

---

 [java]     at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)

---

[java]     at java.lang.reflect.Method.invoke(Method.java:600)****

 [java]     at org.intrace.agent.Agent.initialize(Agent.java:66)****

 [java]     ... 7 more****

 [java] Caused by: java.lang.UnsupportedOperationException: adding

retransformable transformers is not supported in this environment****

 [java]     at

sun.instrument.InstrumentationImpl.addTransformer(InstrumentationImpl.java:80)

---

 [java]     at org.intrace.agent.AgentInit.initialize(AgentInit.java:41)

---

 [java]     ... 12 more****

---

 [java] Fatal error: processing of -javaagent failed****

---

BUILD SUCCESSFUL****

Total time: 0 seconds

On Mon, Jan 2, 2012 at 5:55 PM, Erik Ostermueller
[email protected]:

mchr3k wrote:

Have you used InTrace much or have you just had a quick play?
Just a quick play. I verified that I can capture SQL statements from the
Java PetStore application.
I like the way you can dynamically add/remove the instrumentation and the
filters.

What are you using InTrace for? Personal/commercial projects?
Both commercial and personal, actually. The company I work for relies
heavily on big-dollar APM tools for a number of things.
InTrace can't completely replace those tools, but it could reduce the
number of licenses we need for those tools.
What kind of big dollar tools? CompuWare DynaTrace and CA Introscope.
They cost tens of thousands of USD.
So that is the business side....on the personal side, I'm toying with the
idea of writing a little book on performance testing.
That book would use some techniques that partially rely on the InTrace
feature requests I've made.

Are you interested in getting involved in the development of these
features or are you hoping I will implement these?
Do you have a timeline in mind? This month? This year?
Yes, I'm ready to get involved.
When? How about tomorrow? But I do have a few concerns, before getting
started.

a) I need to run some tests with an IBM jvm on linux. If I can get those
tests to work (hopefully by tomorrow), I'll be ready start helping.

b) If InTrace is monitoring a remote application, it seems that a separate
network request is made for every trace event. Is that right? -- I haven't
looked closely enough at the code to see. If that is the case, I don't
think that kind of model will scale when medium amounts of load is applied.
Perhaps we could consider some kind of burst mode where trace events
collect in a queue and are sent to the GUI every X seconds?
CA Introscope, for instance, has a 15 second delay. I'd be happy with 5.

c) The systems we deploy have multiple, identical JVMs that process
traffic (often, but not always, deployed on separate machines). I'd like
to see a single InTrace GUI capture events from multiple JVMs. Obviously,
the traffic from each would have to be somehow labeled so the user can
distinguish traffic from each JVM.

d) You have support for millisecond response time. I'd really like to
provide support (perhaps optionally) for nanoTime

http://docs.oracle.com/javase/1.5.0/docs/api/java/lang/System.html#nanoTime()

Only item a) is a show stopper for me. I need to make sure that is
resolved before investing any time. I'd like to hear your opinion on the
other two. If needed, I'd be ok to add support for b) and c) and d) at a
later date.

Since this is your code base, perhaps you could propose a design for these
enhancements.
We could agree on the design then you could slice off a few pieces for me
to work on.
What do you think?

So I mentioned these three "plugins" or "implementors":

1. Facility to display Gantt charts for each request of a SOA application
   being traced. Gantt chart will display response time for each traced
   method. Important to show "nested" methods.
2. Response time/throughput graphs of instrumented classes
3. List of all SQL statements executed by the application being traced.

For plugin 1), I've found two decent libraries we could use to create
Gantt charts:
http://www.jfree.org/jfreechart/
http://www.simile-widgets.org/timeline/

The second tool, Simile-Timeline has better scrolling/navigation, but it
is in JavaScript which would make things messy.
So I'd like to start with JFreeChart.

In regards to plugin 2). I'm sick of tools that use "averages" to plot
response time. Averages hide outliers that are often very important.
So instead, I'd like to somehow display the counts of responses in various
time ranges.

For instance, say we had the following response times for 3 requests in a
single 5 second interval:

500ms
95ms
105ms

The average of these numbers is about 233 -- which doesn't some close to
representing the 500, the 95 or the 105ms.

Instead, I'd like the graph to somehow show the counts of fast and slow
responses in various ranges, something like this:

2 requests faster than 128ms
0 requests between 128 and 256ms
1 request between 256 and 512ms.

The ranges will need to be configurable, but this gives you the idea.
We'd have to work out whether we used stacked histograms, or perhaps color
to show these counts.

Let me know what you think.

--Erik

On Mon, Jan 2, 2012 at 2:36 PM, mchr3k <
[email protected]

wrote:

Hi eostermueller,

Thanks for this feature suggestion (and the other two you raised). I am
pleased to see that you are interested in InTrace.

Can you give me a bit more background about yourself and these suggestion
you have raised? I would be interested to know all of the following.

• Have you used InTrace much or have you just had a quick play?
• What are you using InTrace for? Personal/commercial projects?
• Are you interested in getting involved in the development of these
  features or are you hoping I will implement these?
• Do you have a timeline in mind? This month? This year?

---

Reply to this email directly or view it on GitHub:
#12 (comment)

[mchr3k]

How much InTrace functionality relies on "retransforming" ?

The dynamic selection of which classes to trace at runtime depends on retransforming. However, it would be possible to modify InTrace to specify all classes to trace at startup.

Without retransforming support it won't be possible to instrument any Java classes which are loaded before InTrace. However, application classes should be possible to instrument.

Perhaps removing InTrace support for "on the fly" instrumentation on IBM jdk would help?

This sounds like it is necessary for an IBM JDK. This will require code and build changes.

I can provide limited amounts of time/support about what changes need to be made. However, InTrace is something which I do in my spare time so I won't be able to promise you much.

I wonder if you would be interested in agreeing to a basic contract for me to provide a higher level of time/support for implementing the changes you need. If this is something you want to discuss further then it may be better to do it directly over email. You can contact me at mchr3k at gmail.com.