java.lang.NoSuchMethodError: org.apache.logging.log4j.Logger.info

[abhishek6s]

I'm using hive 2.0.1 and hadoop 2.10.0. when I used yauaa-hive-7.19.2-SNAPSHOT-udf.jar -

hive (default)> ADD JAR hdfs:///user/abc/yauaa-hive-7.19.2-SNAPSHOT-udf.jar;
converting to local hdfs:///user/abc/yauaa-hive-7.19.2-SNAPSHOT-udf.jar
Added [/mnt/hive/tmp/319e971b-0c54-46c2-803d-26c64f48c962_resources/yauaa-hive-7.19.2-SNAPSHOT-udf.jar] to class path
Added resources: [hdfs:///user/abc/yauaa-hive-7.19.2-SNAPSHOT-udf.jar]

hive (default)> CREATE FUNCTION ParseUserAgent_v3
              > AS 'nl.basjes.parse.useragent.hive.ParseUserAgent'
              > USING JAR 'hdfs:///user/abc/yauaa-hive-7.19.2-SNAPSHOT-udf.jar';
converting to local hdfs:///user/abc/yauaa-hive-7.19.2-SNAPSHOT-udf.jar
Added [/mnt/hive/tmp/319e971b-0c54-46c2-803d-26c64f48c962_resources/yauaa-hive-7.19.2-SNAPSHOT-udf.jar] to class path
Added resources: [hdfs:///user/abc/yauaa-hive-7.19.2-SNAPSHOT-udf.jar]

hive (default)> select parseuseragent_v3('Mozilla/5.0 (Linux\; Android 6.0\; Nexus 6 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.76 Mobile Safari/537.36');
Exception in thread "de12ba54-5bd7-4f94-bd39-9291652b4e12 main" java.lang.NoSuchMethodError: org.apache.logging.log4j.Logger.info(Ljava/lang/String;Ljava/lang/Object;)V
	at nl.basjes.parse.useragent.utils.YauaaVersion.logVersion(YauaaVersion.java:129)
	at nl.basjes.parse.useragent.utils.YauaaVersion.logVersion(YauaaVersion.java:109)
	at nl.basjes.parse.useragent.utils.YauaaVersion.logVersion(YauaaVersion.java:102)
	at nl.basjes.parse.useragent.AbstractUserAgentAnalyzerDirect$AbstractUserAgentAnalyzerDirectBuilder.build(AbstractUserAgentAnalyzerDirect.java:545)
	at nl.basjes.parse.useragent.AbstractUserAgentAnalyzer$AbstractUserAgentAnalyzerBuilder.build(AbstractUserAgentAnalyzer.java:339)
	at nl.basjes.parse.useragent.hive.ParseUserAgent.lambda$static$0(ParseUserAgent.java:85)
	at java.lang.ThreadLocal$SuppliedThreadLocal.initialValue(ThreadLocal.java:305)
	at java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:195)
	at java.lang.ThreadLocal.get(ThreadLocal.java:172)
	at nl.basjes.parse.useragent.hive.ParseUserAgent.initialize(ParseUserAgent.java:114)
	at org.apache.hadoop.hive.ql.udf.generic.GenericUDF.initializeAndFoldConstants(GenericUDF.java:139)
	at org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.newInstance(ExprNodeGenericFuncDesc.java:230)
	at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:1097)
	at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1316)
	at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
	at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
	at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
	at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:158)
	at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
	at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:213)
	at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:157)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:10519)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:10475)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:3843)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:3622)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:8895)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8850)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9703)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9596)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:10092)
	at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:353)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10103)
	at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:228)
	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:239)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:473)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:319)
	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1249)
	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1295)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1178)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1166)
	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:236)
	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:187)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
	at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:782)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:721)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:648)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:244)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:158)

hive is having below logging lib

log4j-slf4j-impl-2.4.1.jar
log4j-api-2.4.1.jar
log4j-web-2.4.1.jar
log4j-1.2-api-2.4.1.jar
log4j-core-2.4.1.jar

can anyone guide me on this what needs to be done in order to tix this?

Also, how do I maintain this library for my hive and hadoop versions ? ( will test run respective of my hive and hadoop versions when we change in respective versions pom.xml ? )

[nielsbasjes]

Few things I noticed:

• Why have you built Yauaa yourself instead of downloading the prebuilt version?
• Hive 2.0.1 is from 2016 and the Hadoop version you have is from 2019. All of these are really old, which may cause incompatibilities.
• Hive is a distributed system. You have to make sure that all instances have these dependencies.

[nielsbasjes]

Perhaps a simpler way:

• Put all Log4j jars on hdfs
• ADD JAR each one
  Go from there.

[abhishek6s]

what do we mean here by putting all log4j jars on hdfs? If I understand correctly, we already have log4j jars coming from hive lib (as mentioned in the header) so we can't overwrite them.

[nielsbasjes]

I meant something like hdfs dfs -put log4j-api-2.20.0.jar followed by ADD JAR hdfs:///user/abc/log4j-api-2.20.0.jar ; in Hive.
If you were already having these libraries then you would not get the error you asked about.

[abhishek6s]

as an experiment, we remove the old log4j and put in the new version of log4j jars. And it worked. But we can't remove old log4j jars in production as all the components like Hadoop, HIVE, HMS, and HS2 are on the old log4j versions.

[nielsbasjes]

Hmmm, ok. Tricky. I have to think about this.

[nielsbasjes]

Since you are already building it yourself perhaps making these changes to the sources helps in your case:

diff --git a/udfs/hive/pom.xml b/udfs/hive/pom.xml
index 6f6efda13..20867b0e2 100644
--- a/udfs/hive/pom.xml
+++ b/udfs/hive/pom.xml
@@ -53,6 +53,16 @@
       <artifactId>yauaa</artifactId>
     </dependency>
 
+    <dependency>
+      <groupId>org.apache.logging.log4j</groupId>
+      <artifactId>log4j-api</artifactId>
+    </dependency>
+
+    <dependency>
+      <groupId>org.apache.logging.log4j</groupId>
+      <artifactId>log4j-core</artifactId>
+    </dependency>
+
     <dependency>
       <groupId>org.apache.commons</groupId>
       <artifactId>commons-collections4</artifactId>

[abhishek6s]

above dependency didn't work, Probably we have classes already in hive and when we add the yauaa jar, hive classload didn't load logger classes again. So I'm trying to shade the log4j dependency but somehow isn't working

<plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>${maven-shade-plugin.version}</version>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>shade</goal>
            </goals>
            <configuration>
              <minimizeJar>true</minimizeJar>
              <createDependencyReducedPom>true</createDependencyReducedPom>
              <filters>
                <filter>
                  <artifact>org.apache.logging.log4j:log4j-core</artifact>
                </filter>
                <filter>
                  <artifact>org.apache.logging.log4j:log4j-api</artifact>
                </filter>
              </filters>
              <relocations>
                <relocation>
                  <pattern>org.apache.logging.log4j</pattern>
                  <shadedPattern>shaded.yauaa.org.apache.logging.log4j</shadedPattern>
                </relocation>
              </relocations>
            </configuration>
          </execution>
        </executions>
      </plugin>

[abhishek6s]

The shading isn't working somehow. Unable to figure out whats the issue. @nielsbasjes can you help

[nielsbasjes]

First thought is you also need <useDependencyReducedPomInJar>true</useDependencyReducedPomInJar>

[nielsbasjes]

Have your checked the logging dependencies are actually IN the Jar you are using? If you have the ordering of the assembly and shade plugins in the incorrect order you will not get the desired endresult.

[nielsbasjes]

Curious to hear if this hack works on your machine
https://github.com/nielsbasjes/yauaa/tree/HiveLog4jDependencies