Xgboost4j - support for Alpine Linux and/or macos/arm64 planned?

snoopcheri · January 6, 2022, 4:40pm

Hello!

The current version 1.5.1 of xgboost4j published on Maven Central [1] contains native libraries for 3 platforms (OS+architecture):

lib/linux/x86_64/libxgboost4j.so
lib/windows/x86_64/xgboost4j.dll
lib/macos/x86_64/libxgboost4j.dylib

Unfortunately, our setup also requires support for alpine-based Linux (MUSL instead of glibc) and ARM-based Macs in the near future.

Our current plan is to build the library on Alpine-Linux (via Docker) and ARM-based Macs, add the two native libraries to the published JAR from Maven Central and store it in our own repository. Of course, this also involves patching the NativeLibLoader.java class.

While this is doable, it’s kinda ugly. Therefore, I’m asking whether there are any plans yet to add support for those additional native platforms in a future release?

The ARM-based Mac version would easily fit into the current structure, as it would simply be an additional architecture (lib/macos/arm64/libxgboost4j.dylib).

The Alpine-based Linux version doesn’t fit well into the current scheme. Maybe something like
lib/alpine_linux/x86_64/libxgboost4j.so would make sense. Or alternatively
lib/linux/x86_64/libxgboost4j_alpine.so.

[1] https://search.maven.org/artifact/ml.dmlc/xgboost4j_2.12/1.5.1/jar

hcho3 · January 6, 2022, 5:08pm

No plan yet. The reason is that we don’t have easy access to ARM-based Macs. (As for Alpine, it should be set up a testing pipeline, but I don’t have bandwidth for it.)

Feel free to set up your own repository in Maven Central.

Craigacp · January 10, 2022, 3:19pm

Is there a simple way to programmatically detect that a JVM is running on top of musl? If so then we can add the detection support to the native loader, which will at least mean you don’t have to hack on the code any, only rebuild the jar. I’m not sure it’s visible from any JVM property though which is how the current detection logic works.

marcphilipp · January 10, 2022, 3:38pm

There seems to be a way that’s employed by other libraries:

github.com

xerial/sqlite-jdbc/blob/ada2b147dc858ef557a995e2f4ef25f7c65fc6a7/src/main/java/org/sqlite/util/OSInfo.java#L115-L131


public static boolean isMusl() {
    Path mapFilesDir = Paths.get("/proc/self/map_files");
    try (Stream<Path> dirStream = Files.list(mapFilesDir)) {
        return dirStream
                .map(
                        path -> {
                            try {
                                return path.toRealPath().toString();
                            } catch (IOException e) {
                                return "";
                            }
                        })
                .anyMatch(s -> s.toLowerCase().contains("musl"));
    } catch (IOException ignored) {
    }
    return false;
}

snoopcheri · January 10, 2022, 3:54pm

There are no differences for the values of the system properties, so this solution falls flat. The suggested solution above by marcphilipp to check /proc/self/map_files seems to do the trick. And it would even have the advantage, that all MUSL-based Linux distributions are recognized, not just Alpine.

snoopcheri · February 2, 2022, 1:28pm

I’ve created a Pull Request for adding support of MUSL-based Linux systems, based on the proposed solution above: https://github.com/dmlc/xgboost/pull/7624