EMR spark - Unsatisfied link for libcrypto.so.1.0.0

I am trying to use xgboost4j with Spark 2.4. I have verified that the libcrypto.so.1.0.0 is present inside the /dev/so folder of the master and the worker nodes. The file was copied into /dev/so/ during cluster creation time through bootstrapping. I have also added the /dev/so/* to the spark.executor.extraLibraryPath and
spark.driver.extraClassPath. But when trying to train the model I am getting this error -

java.lang.UnsatisfiedLinkError: /mnt3/yarn/usercache/zeppelin/appcache/application_1583546919274_0001/container_1583546919274_0001_01_000034/tmp/libxgboost4j6574794715384475912.so: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory

Does the libcrypto.so.1.0.0 file needs to be in some specific directory ? Any pointers would be much appreciated.

1 Like

I have the same problem. I manage to solve the problem, in runtime at least, in 2 steps:

  • Add libcrypto.so.1.0.0, libhdfs.so.0.0.0 or libgomp.so.1 to library path
LIB_PATH=... # path to libs 
export LD_LIBRARY_PATH=$LIB_PATH
  • When launching my job with Spark submit:
spark-submit-2.4.3 \
    --files ${LIB_PATH}/libcrypto.so.1.0.0,${LIB_PATH}/libhdfs.so.0.0.0,${LIB_PATH}/libgomp.so.1 \
    ...

During unit test, however, I cannot find a good solution. The problem comes from the fact that I can’t find the way to reproduce the trick above when the job/spark session is started by my unit tests.