problem：
I loaded xgboost model files based on java spring to provide online model prediction services. However, while the service was running, the memory kept increasing. Through JVM NMT, I found that it was Java_ml_dmlc_xgboost4j_java_XGBoostJNI_XGDMatrixCreateFromCSR that caused Internal Memory to keep increasing.
java version:11
verison:
ml.dmlc
xgboost4j_2.12
1.5.0
code:
dMatrix = new DMatrix(ListUtil.toLongArray(headers), ListUtil.toIntArray(indices), ListUtil.toFloatArray(data), DMatrix.SparseType.CSR, (int) numFeature);
try code:
dMatrix = new DMatrix(ListUtil.toLongArray(headers), ListUtil.toIntArray(indices), ListUtil.toFloatArray(data), DMatrix.SparseType.CSR, (int) numFeature);
dMatrix.dispose();
In order to avoid being affected by my other code, I tried to process the matrix immediately after new, but the memory was still not released. I don’t know if there is any problem with dispose.
NMT log:
Native Memory Tracking:
Total: reserved=7363MB +31MB, committed=4289MB +31MB
- Java Heap (reserved=3072MB, committed=3072MB)
(mmap: reserved=3072MB, committed=3072MB)
- Class (reserved=1119MB, committed=105MB)
(classes #16959)
( instance classes #15884, array classes #1075)
(malloc=3MB #46273 +33)
(mmap: reserved=1116MB, committed=103MB)
( Metadata: )
( reserved=92MB, committed=91MB)
( used=89MB)
( free=2MB)
( waste=0MB =0.00%)
( Class space:)
( reserved=1024MB, committed=12MB)
( used=11MB)
( free=1MB)
( waste=0MB =0.00%)
- Thread (reserved=2046MB, committed=185MB)
(thread #2029)
(stack: reserved=2037MB, committed=175MB)
(malloc=7MB #10156)
(arena=2MB #4056)
- Code (reserved=244MB, committed=45MB)
(malloc=3MB #12101 +46)
(mmap: reserved=242MB, committed=42MB)
- GC (reserved=163MB, committed=163MB)
(malloc=17MB #40973 +4)
(mmap: reserved=146MB, committed=146MB)
- Compiler (reserved=3MB, committed=3MB)
(malloc=3MB #1913 +8)
- Internal (reserved=614MB +30MB, committed=614MB +30MB)
(malloc=614MB +30MB #37792 +272)
- Other (reserved=70MB, committed=70MB)
(malloc=70MB #251)
- Symbol (reserved=22MB, committed=22MB)
(malloc=19MB #233751 +11)
(arena=3MB #1)
- Native Memory Tracking (reserved=8MB +1MB, committed=8MB +1MB)
(malloc=1MB +1MB #19766 +7291)
(tracking overhead=6MB)
[0x00007fc52398d029] jni_GetIntArrayElements+0x169
[0x00007fc4c1341db4] Java_ml_dmlc_xgboost4j_java_XGBoostJNI_XGDMatrixCreateFromCSR+0x74
[0x00007fc50ca77cf9]
(malloc=290MB type=Internal +16MB #1606 +88)
[0x00007fc52398daa9] jni_GetFloatArrayElements+0x169
[0x00007fc4c1341dce] Java_ml_dmlc_xgboost4j_java_XGBoostJNI_XGDMatrixCreateFromCSR+0x8e
[0x00007fc50ca77cf9]
(malloc=290MB type=Internal +16MB #1605 +87)