Stack Overflow
Stack Overflow

Questions tagged [joblib]

Joblib is a set of tools to provide lightweight pipelining in Python.

Joblib is a set of tools to provide lightweight pipelining in Python.

https://joblib.readthedocs.io/en/latest/

715 questions
0
votes
1 answer

Counting parallel function calls python

I have a problem where I need to call an instance function of a class in parallel and count the number of times it has been called so each call has a unique identifier (to be used to store results in a unique location). Here is a question with…
Pavel Komarov
  • 1,153
  • 12
  • 26
0
votes
1 answer

python joblib.load: Iterating over generator results in infinite loop

Below is the code I use for the generator. It is modified from Saving and loading multiple objects in pickle file?. The file was created with joblib.dump, and consists (so far) of a single machine learning model. import pickle from sklearn.externals…
JGrossman
  • 33
  • 4
0
votes
1 answer

Joblib error: TypeError: can't pickle _thread.lock objects

I am unable to run joblib using my function which takes a numpy array, list of trained Keras models and a list of strings as parameters. I tried creating the parameters as a namedtuple or even as a class with immutable properties. Any ideas ? Params…
Atti
  • 380
  • 4
  • 21
0
votes
0 answers

Joblib.load Key Error:253

I am saving a Naive Bayes classifier with Joblib.dump. Each time my machine is run, it loads the model with joblib.load(filename). This was working for a while but I started getting this error: File "test.py", line 92, in categorize subclf =…
Evan Lalo
  • 1,209
  • 1
  • 14
  • 34
0
votes
1 answer

Investigating joblib slowdown

I'm trying to use joblib to make a custom random forest implementation train in parallel. The task is embarrassingly parallel, so I assumed getting a speedup shouldn't be too hard with joblib. Here's some sample code: class RandomForest(object): …
Bar
  • 2,736
  • 3
  • 33
  • 41
0
votes
1 answer

How to reuse a selenium driver instance during parallel processing?

To scrape a pool of URLs, I am paralell processing selenium with joblib. In this context, I am facing two challenges: Challenge 1 is to speed up this process. In the moment, my code opens and closes a driver instance for every URL (ideally would be…
sudonym
  • 3,788
  • 4
  • 36
  • 61
0
votes
1 answer

Make a prediction on csv file, one line at a time

I have a large csv file that i need to take a row of data, one at a time, and score it against a model. I have tried the code below but get an error of "X has 120839 features per sample; expecting 30". I can run the model against the entire dataset…
user3046660
  • 81
  • 1
  • 1
  • 10
0
votes
1 answer

Can't load ML models in Python

I've built a model in Python and saved it with joblib from sklearn.externals package: from sklearn.externals import joblib joblib.dump(rf_Prob_F, 'Model.pkl') When I try to call the model with the following command, an error appears: from…
Charles
  • 43
  • 1
  • 6
0
votes
1 answer

joblib.Parallel processes the same set of data multiple times instead of different sets

I have a matrix array of 3D brain images which I am doing some processing for these images. The input matrix looks like M[X, Y]: where X is the brain id and Y is the data which I am reshape it later to make some enhancement for The following…
Khaled
  • 345
  • 5
  • 14
0
votes
1 answer

Multiprocessing for Python parallelization error - "function' object is not iterable"

We have NVIDIA Tesla K80 GPU accelerator computing in our data center with the following characteristics: Intel(R) Xeon(R) CPU E5-2670 v3 @2.30GHz, 48 CPU processors, 128GB RAM, 12 CPU coresrunning under Linux 64-bit. I am running the following code…
Desta Haileselassie Hagos
  • 23,140
  • 7
  • 48
  • 53
0
votes
1 answer

Pickling sklearn models and using them in web app

I have created an sklearn model that I would like to imbed into my nodejs web app. I have created the model with python3 and packaged it into a pkl file with joblib. from sklearn.externals import joblib joblib.dump(clf, 'RandomForest_jul30.pkl',…
Christopher Evans
  • 51
  • 1
  • 8
0
votes
2 answers

Installing joblib error in Python

I have been getting this error when I try to install joblib. How do I get around it? joblib/test/data/joblib_0.9.4.dev0_compressed_cache_size_pickle_py35_np19.gz_01.npy.z -> build/lib/joblib/test/data copying…
coolfun
  • 1
  • 1
  • 2
0
votes
0 answers

Parallelization with Joblib Multicore more time than Single core

I´m developing in machine learning(using Python version 2.7.13) and Im using Hyperopt to process data and get a porcentage of well processed data. I want to make crossvalidation multicore, but doing this it takes more time than doing it single core.…
alc11
  • 1
  • 2
0
votes
0 answers

Kill joblib code in python 3

I have used joblib module in python 3.6.0 to parallelize my code and ran it on my Linux machine. It has now started the process on 20 threads, as expected. But I am unable to kill all of them at once. For instance, if I kill any of the same 20…
Vembha
  • 33
  • 5
0
votes
1 answer

Joblib simple example parallel example slower than simple

from math import sqrt from joblib import Parallel, delayed import time if __name__ == '__main__': st= time.time() #[sqrt(i ** 2) for i in range(100000)] #this part in non parellel Parallel(n_jobs=2)(delayed(sqrt)(i ** 2) for i in…
Soumya
  • 87
  • 1
  • 2
  • 15
1 2 3
…
47 48