-
-
Notifications
You must be signed in to change notification settings - Fork 83
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Recycled buffers in XMLStreamWriter are colliding between multiple writer instances when used in an Akka stream #82
Comments
Currently using woodstox-core:5.3.0, but also tested 5.2.1 and 6.0.0 and they are also affected. |
Ok. Unfortunate timing, on one hand (since I just pushed 6.0.0 recently), but possibly fortunate since I think I went through similar problem with Jackson. So: one assumption with current buffer recycling is that parser/generator instances are only ever used from a single thread: not just at any given time (i.e. single-threaded access), but during life-cycle of parser/generator instance. If so, I was actually wondering if I should port changes to Woodstox' recycler (which is where Jackson originally got it impl from), but since I had not heard of any issues I hadn't. But sounds like there is now use case. Now: do you think it would be easy for you to test a fix, if I come up with what I think should work? Does not have to be automated. |
Yeah, I'm able to consistently reproduce within my application using an aggressive gatling script. |
Hmmh. That is interesting -- code in Finding out which buffer is causing problems might help. |
One thing wrt Aalto being less prone could be that it does fewer copies in general, and bundles local-name/namespace-URI combination so it could just have less time to have collision or something. So perhaps it is just significantly less common to get corruption, but not immune. |
That's sorta what I thought might be the case. And why I didn't just switch over to aalto and move on. |
:) It'd be good to resolve the problem, still. One more question: how is the shared access from multiple threads done? As in, are parsers/generators (problem is on generator side, right?) being shared or passed across processing threads? Since both operations are sequential, I assume there is some hand-over or. ..? |
Problem is on the generator side, yes. Handoff is happening entirely by the akka stream/actors framework. I'm not exactly sure under the covers how it assigns threads to actors, etc. |
To answer your earlier question, it appears to be the medium cbuffer that I'm having issues with.
and I can see the problem pretty clearly in the log:
|
Thank you for additional information. |
Do you know what sort of timeline you'd be looking at for publishing fixes to both woodstox and aalto? I'm currently looking into how to incorporate a temporary fork w/the ThreadLocals disabled into my project, but if you expect a pretty quick and easy fix to be available soon, I'd probably hold off on that. |
This is bit difficult to say unfortunately as I am busy trying to finish up Jackson 2.10. |
Ahhhhh. One simple, obvious thing that I must have done for Jackson a while ago was to actually clear the buffer references. That alone would probably remove obvious failures. I think I'll dig into this bit more, and that should result in a fix this week I think -- it's more a question of how to make small enough refactoring. This is internal functionality so API changes should not be problematic. |
I'll give that a test run and let you know how it goes |
Several test runs produced no errors. Looks good. Thanks! |
Was included in |
I'll start w/the caveat that I have not been able to produce a unit test case to illustrate this issue. But what I'm observing in our application is that the recycled char buffers (pretty sure it's the small copy buffers, but could potentially be the others), are being concurrently used by multiple threads in an Akka stream, and the result is corrupted output. In particular in the namespace writing, in my application.
I've so far been unable to reproduce the problem using aalto, but I'm not certain it also isn't affected given that it also uses the ThreadLocal recycled buffers.
Hoping you might have an inkling as to how to attempt to reproduce outside of Akka and my application? Or perhaps this is already a known issue that woodstox should not be used under fork-join parallelism? (However, I did try changing my akka configuration to use a fixed threadpool dispatcher and the problem still occurs.)
I've locally recompiled woodstox w/some debug logging added and I have some log files that demonstrate the problem. Don't want to presume upon you to review them, but I mention in case it might help.
The text was updated successfully, but these errors were encountered: