I am trying to use xgboost4j with Spark 2.4. I have verified that the libcrypto.so.1.0.0 is present inside the /dev/so folder of the master and the worker nodes. The file was copied into /dev/so/ during cluster creation time through bootstrapping. I have also added the /dev/so/* to the spark.executor.extraLibraryPath and
spark.driver.extraClassPath. But when trying to train the model I am getting this error -
java.lang.UnsatisfiedLinkError: /mnt3/yarn/usercache/zeppelin/appcache/application_1583546919274_0001/container_1583546919274_0001_01_000034/tmp/libxgboost4j6574794715384475912.so: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory
Does the libcrypto.so.1.0.0 file needs to be in some specific directory ? Any pointers would be much appreciated.