The random number engine is not reset during each run.

If you remove the colsample_bytree the results should be the same (within floating point errors). Whether to reset the random engine for each run has different effects. For example, the result of:

booster_0 = train({...}, dtrain, num_boost_round=1)
booster_1 = train({...}, dtrain, num_boost_round=1, xgb_model=booster_0)

should equal to:

booster = train({...}, dtrain, num_boost_round=2)

If the random number engine were reset during each run, the above wouldn't hold.

Yes, resetting the random number engine would be a bad idea. Because yes, you want to be able to interrupt training and reach the same results. In other words, we are in agreement that:

booster_0 = train({...}, dtrain, num_boost_round=1)
booster_1 = train({...}, dtrain, num_boost_round=1, xgb_model=booster_0)

should always come up with the same results as

booster = train({...}, dtrain, num_boost_round=2)

However, the current code does NOT guarantee this as my post demonstrates! Because IF the user makes any calls to xgb.train() in between the booster_0 and booster_1 calls above, on an entirely different booster instance, it can mess up the RNG state so that booster_1 does not end up the same as booster (again, within floating point errors).

In other words, this simple change will keep booster from equalling booster_1 as we both agree it should:

booster_0 = train({...}, dtrain, num_boost_round=1)
booster_irrelevant = train({ params with colsample_* values < 1.0}, dtrain, num_boost_round=1, xgb_model=booster_0)
booster_1 = train({...}, dtrain, num_boost_round=1, xgb_model=booster_0)

It 100% should not be the case that interrupting training on one booster by training another should change the outcome on the first booster. And yet, it does. Because that call to train booster_irrelevant is far from irrelevant: it tweaks the RNG state in xgb thereby changing the fate of booster_1 from what it should have been.

If you cannot train in bursts (let's say two bursts of 10 rounds each) and achieve the same results as if you trained in a single burst of 20, then that seems like a bug that should be fixed. Otherwise, the only way to guarantee the same model outcome in xgboost is to always start training from the very start. Which is untenable for boosters that require hundreds of rounds of training. And yet, in the current code, that is exactly what users must do if they want repeatable results (assuming they use any colsample_* parameters that tweak the RNG state).

My suggestion is not to reset the random number engine with each call to xgb.train() but instead to save the random state on a per-booster basis. Then, even if the param set includes colsample_* or other stochastic settings, when you return to training the model in question you reset the RNG to wherever it left off before and achieve repeatable results, even if you train in bursts.

I am not familiar with the internals of the code so I am not clear on whether my fix is easy or even possible. But right now anyone who trains boosters in bursts using xgboost, and who work with more than one booster at a time, is not guaranteed to get the same results between their runs unless they avoid using the parameters that adjust the RNG state!

Because IF the user makes any calls to xgb.train() in between the booster_0 and booster_1 calls above, on an entirely different booster instance

That's true.

Making the state local to the booster is possible with a few changes. But saving it might be more involved. Let me figure something out.

Thank you!