Critical vulnerability in Apache Log4j library (CVE-2021-44228)

[maziyarpanahi]

Hi everyone,

As it is explained in Apache Log4j Security Vulnerabilities, versions of log4j 2.14.1 and under of the library, JNDI features used in the configuration, log messages, and parameters, do not protect against attacker-controlled LDAP and other JNDI-related endpoints. It is also known as Log4shell.

This allows attackers that gain control over log messages or log message parameters to execute arbitrary code loaded from LDAP servers when message lookup substitution is enabled. Since this was already discovered, Apache quickly fixed this issue and released the 2.15.0 version, where this behavior has been disabled by default.

How does this affect Spark NLP?

Short answer:

It does not! The issue is with log4j version 2.x hence the name log4j2 vulnerability and Apache Spark uses log4j 1.x.

Longer answer:

This project (spark-nlp) doesn't have any dependency on log4j. As you may know, this dependency is inherited from spark-core from the Apache Spark project. That's being said, all the major and minor releases of Apache Spark (even the latest 3.2.0) are using log4j 1.x releases.

Is log4j 1.x vulnerable? As log4j 1.x does not offer a look-up mechanism, it does not suffer from CVE-2021-44228.

So when it comes to Spark NLP and Apache Spark the log4j used is 1.x and doesn't need any further action regarding CVE-2021-44228 vulnerability. However, since you might be using other third-party dependencies that might use log4j 2.x I would ike to share a possible solution/workaround that will keep you and your clusters safe until everything is updated everywhere:

spark.driver.extraJavaOptions "-Dlog4j2.formatMsgNoLookups=true"
spark.executor.extraJavaOptions "-Dlog4j2.formatMsgNoLookups=true"

If any dependency in your Apache Spark session uses log4j 2.x this config will disable the problematic lookup. You can also keep an eye on the list of the affected software to upgrade quickly and be safe: https://github.com/NCSC-NL/log4shell/tree/main/software

We will keep updating this thread with any new future development.

UPDATE 1:
Databricks Log4j2 Vulnerability (CVE-2021-44228) Research and Assessment