mavenExecute (go) breaks build when output is too long

[frzme]

Our maven test execution is broken since mavenExecute is using go to execute maven.

08:30:43  time="2020-04-28T08:30:43+02:00" level=error msg="Error while reading from Writer: bufio.Scanner: token too long" library=SAP/jenkins-library stepName=mavenExecute

Also it seems that all maven output is now wrapped in go log statements, can this be reverted and the maven output displayed "as is" again?

[daniel-kurzynski]

#1432 will clear the log statements to only show the message.

Not sure if it will solve the "token too long" issue. Could you please provide more information about which mvn command you are going to run. Looks like this command generates to much output.

[ThommySte]

Switch back to the version before this piper-lib-os@b335387eac3de9f7c68290503beebe2988644d5f "solves" the issue. We will see if the #1432 is merged in master - is there an time line for?

[frzme]

@daniel-kurzynski see #sap-piper-users on Slack for a link, I mentioned you.
It's output (a http message) generated by our java application while running junit tests through surefire.

[stippi2]

To me, it looks like this issue: sirupsen/logrus#564, i.e. it isn't specific to mavenExecute, but could happen to any step, as they all redirect the stdout of an executed command to the logging framework. But of course we'll need to fix it. The linked ticket provides some pointers.

[OliverNocon]

@stippi2 that could really be it.

What I am wondering about though is that the root of the issue seems to be input without line endings.
Piping the Maven log should not pose such an issue by default since typically there are line breaks available.

What could cause such long lines in a Maven execution? Do we have an example of a Maven output which is now broken (before-after)?

[stippi2]

From the build log, it seems to be WireMock, and in particular a class called c.s.h.c.l.s.f.RequestLogger which directly forwards request meta data and payloads to the log. I am not sure about your assumption that linebreaks are typically available, I guess it depends on the particular Maven plugin producing the output. This restriction to 64K of output without linebreaks is in the logrus implementation. I have no clue how Jenkins or the sh step handled such output.

[frzme]

To be fair: at least for our case we very (if at all) rarely look at this very long WireMock log output. However so far no one was bothered enough by it to do something about it.
At least for this specific case truncating the line would be acceptable

I think that maven and Jenkins/sh don't buffer per line and can do unlimited sized output messages

[marcusholl]

A solution for that could be not to route log output from underlying tools (here: maven) through the log-framework but route it directly to stderr.

[OliverNocon]

okay, that means we have a case with Wiremock output and thus rather specific to a dedicated tool used in the maven build cycle and not a general issue with a typical maven build which is good.

When reading that the log is basically not used, what about adding an option to not write the log at all?
Alternatively, we could write the log to a file instead of to the logging framework.

Polluting the log generates also other issues, e.g. display issues in Jenkins which are independent of using a logging framework or not ...

I don't know the step well enough to judge where the error appears. I would expect the error message to be wrapped with something from the Piper step ...
If there is no wrapping, I guess we should add it ...
@stippi2 do you have an idea at which place in the code this happens ...

[OliverNocon]

after a short discussion with @nevskrem we should ideally go with an own writer instead of using the default logrus writer.

[daniel-kurzynski]

The fix was merged. Could you please check whether the issue is solved?

[ThommySte]

Thanks, issue is solved now