Stack Overflow
Stack Overflow

Questions tagged [joblib]

Joblib is a set of tools to provide lightweight pipelining in Python.

Joblib is a set of tools to provide lightweight pipelining in Python.

https://joblib.readthedocs.io/en/latest/

715 questions
0
votes
1 answer

Cannot load .pkl persisted model when .ipynb is in subdirectory

I created grid_search.pkl file for model persitence after fitting it in Jupyter Notebook like this: joblib.dump(grid_search, 'grid_search_rf.pkl', compress=3) after a while, I have moved my .ipynb file to subdirectory called notebooks and also I did…
PeterB
  • 2,234
  • 6
  • 24
  • 43
0
votes
0 answers

Parallelizing Python code with copy_reg and joblib

I'm new to python and even newer to parallel code. I am working on a much larger script but I tried to simplify the problem. I am using joblib and copy_reg. When executing my script I don't receive the desired output p0...p1 but instead I receive…
ta8
  • 313
  • 3
  • 12
0
votes
1 answer

Is it possible to serialize only specific classes/functions in pickle / dill python?

I have an app that want to serialize only classes/functions which are: no Python primitive data type no Numpy data type no pandas data type. So, is it possible to filter object to be saved in dill ? (filter by loop on the type) Thanks,
tensor
  • 3,088
  • 8
  • 37
  • 71
0
votes
1 answer

Error in parallelizing for loop inside flask app (TypeError: can't pickle function objects)

I have a flask app and I need to parallelize a for loop inside it. I have been using the method described in https://blog.dominodatalab.com/simple-parallelization/ till now. But when I use this method inside flask app it is failing. Below is a…
qwerty123
  • 479
  • 1
  • 3
  • 14
0
votes
1 answer

cannot load .pkl on raspberry pi

I produced the .pkl on ubuntu 14.04. ubuntu can load the .pkl file but when transferred to the raspberrypi it commit an error upon loading.below listed is the error : I used from sklearn.externals import joblib. I used joblib.dump to store the .pkl…
jude
  • 360
  • 3
  • 12
0
votes
1 answer

TypeError: can't pickle generator objects for Non Generator methods

My error is similar to lot of questions posted, but none actually solved my problem. So, I was trying to use joblib's Parallel & delayed for doing parallel processing, and the function I want to use it on, raises error when pickling. I face the…
Debasish Kanhar
  • 1,123
  • 2
  • 15
  • 27
0
votes
0 answers

Parallelization in Python using Numpy and Joblib

I want to move from MATLAB to open-source alternatives as scipy and numpy. However, I have some problems with the speed. I am aware that sometimes multi-core operations can be slower than single core due to overheading, however, I am trying to do a…
umitarabul
  • 1
  • 3
0
votes
2 answers

How to parallelise nested for loops in python

I have this function containing nested loops. I need to parallelise for faster execution of code. def euclid_distance(X,BOW_X): d3=[] d2=[] for l in range(len(X)): for n in range(l+1,len(X)): d1=[] …
ajit_gaikwad
  • 1
  • 4
0
votes
1 answer

load serialized files from django

I have some trained machine learning model saved in a django project, it is multiple files involved (x.p + x_01.npy + ... + x_0n.npy), and I am trying to load them in views.py, got file not found exception, wonder what is the correct way using…
user3453552
  • 199
  • 1
  • 2
  • 11
0
votes
1 answer

Deprecated Scikit-learn module prevents joblib from loading it

I have a Hidden Markov Model that has been pickled with joblib using the sklearn.hmm module. Apparently, in version 0.17.x this module has been deprecated and moved to hmmlearn. I am unable to load the model and I get the following…
user6858731
  • 1
  • 2
0
votes
0 answers

How to efficiently parallelize a scikit SVM classification on a 2D image in Python

I'm using the scikit-learn package to do a SVM classification on an 2D image. Each pixel has 9 features on which the classification is based. Let's assume I have a successfully trained classifier clf, and I want to use it to perform the…
Kristof
  • 144
  • 3
  • 10
0
votes
0 answers

joblib.load() in cluster cannot load files

Intro I am porting some code that run on a regular laptop to a cluster (HPC) with MPI. What I am dealing with is an embarrassingly parallel problem where I am sending different file paths to a bunch of workers. Each corresponding file contains one…
s1mc0d3
  • 523
  • 2
  • 15
0
votes
1 answer

How to parallelize this embarrassingly parallel loop with Python

I have an embarrassingly parallel loop: # Definitions def exhaustiveExplorationsWithSimilarityAll(inputFolder, outputFolder, similarityMeasure): phasesSpeedupDictFolder=parsePhasesSpeedupDictFolder(inputFolder) …
Amir
  • 1,348
  • 3
  • 21
  • 44
0
votes
0 answers

Loading AdaBoostClassifier

classifier = AdaBoostClassifier(n_estimators=100, learning_rate=1.0, algorithm='SAMME.R') try: classifier = joblib.load("final_model_Ada1.pkl") print "using trained model" except: print "building new model" classifier.fit(X_train,…
akshita007
  • 549
  • 1
  • 9
  • 15
0
votes
1 answer

Python - Parallelisation of a loop

I am trying to parallelize a loop that is very costly. Here is the code: import numpy as np class em: def __init__(self, k, x, iterations): self.k = k self.x = x self.iterations = iterations self.n =…
valentin
  • 2,596
  • 6
  • 28
  • 48
1 2 3
…
47 48