libstdc++.so.6: version `GLIBCXX_3.4.20' not found

[drbonis]

This error was found when trying to run sparknlp_jsl (3.0.3) over spark 3.0.1 in a RedHat 7 linux machine and specifically when loading the T5Transformer;

t5 = T5Transformer().pretrained("t5_base")

The error:

py4j.protocol.Py4JJavaError: An error occurred while calling z:com.johnsnowlabs.nlp.pretrained.PythonResourceDownloader.downloadModel.
: java.lang.UnsatisfiedLinkError: /tmp/libsentencepiece_jni7602875099172035543.so: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /tmp/libsentencepiece_jni7602875099172035543.so)

I found that by adding /usr/local/lib64 to the dynamic link loader path:

export LD_LIBRARY_PATH=/usr/local/lib64:$LD_LIBRARY_PATH

the problem can be solved.

[maziyarpanahi]

In addition to T5Transformer, the libsentencepiece_jni file is also required in the following annotators/models as well:

• MarianTransformer
• XlnetEmbeddings
• AlbertEmbeddings
• XlmRoBertaEmbeddings

Similar issue with workaround/solution: #813

[dengbo22]

I met this problem when execute at cluster mode, how can I fix it ?