Hadoop error

[chenxuhao]

Hi all,
I have setup a 3-node cluster (1 namenode, 2 datanode).
I was trying to run Arabesque like this:
./run_arabesque.sh cluster.yaml cliques.yaml

in cluster.yaml I changed num_workers to 2.

I got this output, wihch I don't understand:

...
3343 [main] INFO org.apache.giraph.job.GiraphJob - Waiting for resources... Job will start only when it gets all 3 mappers
2019-10-24 23:24:11,775 INFO mapreduce.Job: Running job: job_1571961586402_0007
13874 [main] INFO org.apache.hadoop.mapreduce.Job - Running job: job_1571961586402_0007
2019-10-24 23:24:11,783 INFO mapreduce.Job: Job job_1571961586402_0007 running in uber mode : false
13882 [main] INFO org.apache.hadoop.mapreduce.Job - Job job_1571961586402_0007 running in uber mode : false
2019-10-24 23:24:11,786 INFO mapreduce.Job: map 0% reduce 0%
13885 [main] INFO org.apache.hadoop.mapreduce.Job - map 0% reduce 0%
2019-10-24 23:24:11,807 INFO mapreduce.Job: Job job_1571961586402_0007 failed with state FAILED due to: Application application_1571961586402_0007 failed 2 times due to AM Container for appattempt_1571961586402_0007_000002 exited with exitCode: 1
Failing this attempt.Diagnostics: [2019-10-24 23:24:11.725]Exception from container-launch.
Container id: container_1571961586402_0007_02_000001
Exit code: 1

[2019-10-24 23:24:11.731]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/hadoop-xchen/nm-local-dir/usercache/xchen/appcache/application_1571961586402_0007/filecache/11/job.jar/job.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/workspace/hadoop-3.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
...

full output is attached.
No error in the hadoop logs.

Any idea?

output-example.txt

[chenxuhao]

I found two errors in file: logs/userlogs/application_1571981443289_0001/container_1571981443289_0001_01_000001/syslog

ERROR [main] org.apache.hadoop.metrics2.lib.MethodMetric: Error invoking method numOpenConnectionsPerUser

ERROR [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster

Any idea?

[chenxuhao]

115 2019-10-25 00:32:19,629 ERROR [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
116 java.lang.NoSuchMethodError: org.apache.commons.lang3.StringUtils.isAnyBlank([Ljava/lang/CharSequence;)Z
117 at org.apache.hadoop.yarn.webapp.WebApp.getPrefix(WebApp.java:278)
118 at org.apache.hadoop.yarn.webapp.WebApp.parseRoute(WebApp.java:244)
119 at org.apache.hadoop.yarn.webapp.WebApp.route(WebApp.java:232)
120 at org.apache.hadoop.mapreduce.v2.app.webapp.AMWebApp.setup(AMWebApp.java:36)
121 at org.apache.hadoop.yarn.webapp.WebApp.configureServlets(WebApp.java:160)
122 at com.google.inject.servlet.ServletModule.configure(ServletModule.java:55)
123 at com.google.inject.AbstractModule.configure(AbstractModule.java:59)
124 at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:223)
125 at com.google.inject.spi.Elements.getElements(Elements.java:101)
126 at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:133)
127 at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:103)
128 at com.google.inject.Guice.createInjector(Guice.java:95)
129 at com.google.inject.Guice.createInjector(Guice.java:72)
130 at com.google.inject.Guice.createInjector(Guice.java:62)
131 at org.apache.hadoop.yarn.webapp.WebApps$Builder.build(WebApps.java:387)
132 at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:432)
133 at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:428)
134 at org.apache.hadoop.mapreduce.v2.app.client.MRClientService.serviceStart(MRClientService.java:147)
135 at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
136 at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1290)
137 at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
138 at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(MRAppMaster.java:1761)
139 at java.security.AccessController.doPrivileged(Native Method)
140 at javax.security.auth.Subject.doAs(Subject.java:422)
141 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
142 at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1757)
143 at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1691)
144 2019-10-25 00:32:19,630 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting with status 1: java.lang.NoSuchMethodError: org.apache.commons.lang3.Str ingUtils.isAnyBlank([Ljava/lang/CharSequence;)Z