useExternalMemory no longer does anything

mallman · April 29, 2022, 7:24am

Hello,

It appears that this change:

(Unfortunately the forum seems to be snipping off the specific parts of that URL. The change I’m referring to is located at lines 788 through 822 of file src/data/data.cc.)

makes this parameter:

github.com

dmlc/xgboost/blob/v1.6.0/jvm-packages/xgboost4j-spark/src/main/scala/ml/dmlc/xgboost4j/scala/spark/params/GeneralParams.scala#L54


 * number of threads used by per worker. default 1
 */
final val nthread = new IntParam(this, "nthread", "number of threads used by per worker",
  ParamValidators.gtEq(1))


final def getNthread: Int = $(nthread)


/**
 * whether to use external memory as cache. default: false
 */
final val useExternalMemory = new BooleanParam(this, "useExternalMemory",
  "whether to use external memory as cache")


final def getUseExternalMemory: Boolean = $(useExternalMemory)


/**
 * Deprecated. Please use verbosity instead.
 * 0 means printing running messages, 1 means silent mode. default: 0
 */
final val silent = new IntParam(this, "silent",
  "Deprecated. Please use verbosity instead. " +

impotent.

I am trying to debug memory allocation failures in a distributed xgboost4j-spark app, and one thing that comes to mind is whether SparsePageDMatrix might be worth trying as an alternative to SimpleDMatrix. I thought setting the useExternalMemory parameter might do that, but it appears that the PR above removed that functionality in 1.5.0. Could someone explain why?

Cheers.

Michael