1

I am trying to save model in azure databricks but getting error - "It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063."

Following is the code which I am using :-

import dill as dill
with open('model.sav', "wb") as f:
    dill.dump(model,f)

or is there any other way to download model to local without using mleap or mlflow?

I am using this code totrain model.

pipe = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler())])

pipe_xgb = make_pipeline(pipe,DataFrameScaler(),xgb.XGBClassifier(objective='binary:logistic'))
param_grid_classifier = {
                        'xgbclassifier__n_estimators':[100,200], 
                         'xgbclassifier__learning_rate' :[0.1,0.01],
                         'xgbclassifier__colsample_bytree':[0.3,0.5],
                         'xgbclassifier__max_depth': [4],
                        'xgbclassifier__reg_lambda':[0.01,0.1],
                         'xgbclassifier__n_jobs':[4]
                        }
metric = 'average_precision'
grid_search1 = GridSearchCV(pipe_xgb, param_grid=param_grid_classifier,scoring=metric, cv=2)

gridmodel1 = grid_search1.fit(X_train, y_train_label)
model = gridmodel1.best_estimator_

Thanks in advance for help.

Surbhi Jain
  • 132
  • 1
  • 6

0 Answers0