0

I've been working on validating models developed using h2o.

Specificially I've been testing a neural net implemented using h2o.deeplearning. I've been attempting to generate consistent results by setting a seed in the H2O function, but even doing this I see correlation coefficients of between 0.6 and 0.85 between different versions of the same model, even ones with identical seeds.

I did some reading, and saw that I could force reproducibility by setting the reproducible flag to TRUE, but at a significant performance cost. The input to this model is too large for that to be a feasible method.

Has anyone else ever had to solve a similar problem/found a way to force H2O neural nets to be reproducible with less performance impact?

Matthieu Brucher
  • 21,634
  • 7
  • 38
  • 62
bsab
  • 1

1 Answers1

2

From the technical note on this topic

Why Deep learning results are not reproducible:

Motivation

H2O's Deep Learning uses a technique called HOGWILD! which greatly increases the speed of training, but is not reproducible by default.

Solution

In order to obtain reproducible results, you must set reproducible = TRUE and seed = 1 (for example, but you can use any seed as long as you use the same one each time). If you force reproducibility, it will slow down the training because this only works on a single thread. By default, H2O clusters are started with the same number of threads as number of cores (e.g. 8 is typical on a laptop).

The R example below demonstrates how to produce reproducible deep learning models:

library(h2o)
h2o.init(nthreads = -1)

# Import a sample binary outcome train/test set into R
train <- read.table("http://www.stat.berkeley.edu/~ledell/data/higgs_10k.csv", sep=",")
test <- read.table("http://www.stat.berkeley.edu/~ledell/data/higgs_test_5k.csv", sep=",")


# Convert R data.frames into H2O parsed data objects
training_frame <- as.h2o(train)
validation_frame <- as.h2o(test)
y <- "V1"
x <- setdiff(names(training_frame), y)
family <- "binomial"
training_frame[,c(y)] <- as.factor(training_frame[,c(y)])  #Force Binary classification
validation_frame[,c(y)] <- as.factor(validation_frame[,c(y)])

Now we will fit two models and show that the training AUC is the same both times (ie. reproducible).

fit <- h2o.deeplearning(x = x, y = y, 
                            training_frame = training_frame, 
                            reproducible = TRUE, 
                            seed = 1)
h2o.auc(fit)
#[1] 0.8715931

fit2 <- h2o.deeplearning(x = x, y = y, 
                            training_frame = training_frame, 
                            reproducible = TRUE, 
                            seed = 1)
h2o.auc(fit2)
#[1] 0.8715931
Lauren
  • 5,640
  • 1
  • 13
  • 19