Joblib is a set of tools to provide lightweight pipelining in Python.
Joblib is a set of tools to provide lightweight pipelining in Python.
Questions tagged [joblib]
715 questions
0
votes
1 answer
Cache dynamically generated functions
I have the probability density functions func1 and func2 (including the support of each) of two random variables. Now I need the probability density function of the sum of these both random variables, which I create via:
import numpy as np
import…
Chickenmarkus
- 1,131
- 11
- 25
0
votes
1 answer
Combine tornado gen.coroutine and joblib mem.cache decorators
Imagine having a function, which handles a heavy computational job, that we wish to execute asynchronously in a Tornado application context. Moreover, we would like to lazily evaluate the function, by storing its results to the disk, and not…
Kevin Ghaboosi
- 606
- 10
- 20
0
votes
0 answers
Multiprocessing with joblib.Parallel - error when parallizing a self written algorithm
I have a class called ftrl_proximal() which fits a model on data. It is a self written classifier (not sklearn's).
The algorithm works perfect when I run using only one CPU, but once I'm trying to perform it in multiprocessing (sort of cross…
Serendipity
- 2,216
- 23
- 33
0
votes
1 answer
Nonetype not callable from joblib when requiring (but not using) cv2
This error is quite strange, whenever I use Kmeans of sklearn with n_jobs > 1 in a unittest and while requiring cv2 using setuptools results in a None being called by joblib.
A minimal failing example:
setup.py:
from setuptools import setup
setup(
…
Herbert
- 5,279
- 5
- 44
- 69
0
votes
0 answers
If starting unix pipe with Parallel command, pipe commands complain after cntrl-c
If I start a process in parallel with joblib that calls a subprocess with a unix pipe (sed | uniq | blabla) many of those processes write error messages to the terminal after having sucessfully exited the python process with Ctrl-C.
sed: couldn't…
The Unfun Cat
- 29,987
- 31
- 114
- 156
0
votes
0 answers
Python Joblib causing loop
I'm working with joblib to load a sklearn classifier. I have this in a class but it seems to keep calling the class's __init__ function. Here's an illustration:
class MyContainer(object):
FILENAME = "classifier.pkl"
def __init__(self):
…
kevin.w.johnson
- 1,684
- 3
- 18
- 37
0
votes
2 answers
python simple parallel computation with joblib
I took example from joblib tutorial. Here is how my code looks like:
from math import sqrt
from joblib import Parallel, delayed
import multiprocessing
test = Parallel(n_jobs=2)(delayed(sqrt)(i ** 2) for i in range(10))
print(test)
It produces the…
user1700890
- 7,144
- 18
- 87
- 183
0
votes
1 answer
Joblib parallel write to "shared" numpy sparse matrix
Im trying to compute number of shared neighbors for each node of a very big graph (~1m nodes). Using Joblib Im trying to run it in parallel. But Im worrying about parallel writes to sparse matrix, which supposed to keep all data. Will this piece of…
Fortyq
- 11
- 1
- 7
0
votes
0 answers
multiprocessing python code
When I try to paralellize python code I get an assertion error. Here is the code :
check = Parallel(n_jobs=ncpu)(delayed(removeident)(h) for h in splitframe)
individually, each element (h) in splitframe works well with the function removeident that…
pifcof
- 11
- 2
0
votes
1 answer
sklearn.externals.joblib does not load pkl file
I inherited some code from a coworker who left my company, and I noticed that an email open predictor he wrote hasn't run in a few weeks. So, I tried to run it. It got to this line:
clf_predict =…
jpavs
- 648
- 5
- 17
0
votes
1 answer
Why won't my nltk classification model persist with joblib?
I've noticed that others have had problems with this on different operating systems and I've also heard that there may be something within a module that disallows loading a persistent classification model. I'm using a mac with python 2.7 and…
Juan Elotes
- 9
- 3
0
votes
1 answer
joblib parallelization of 2 independent calculations on 2 cores is slower than serial
I am trying to parallelize some data expansion with numpy, and I am finding that the parallelized version takes orders of magnitude longer than the serial version, so I must be making some silly mistake.
First, some fake data to set up the…
aph
- 1,765
- 2
- 19
- 34
0
votes
1 answer
Create an empty memory mapped numpy array using joblib
I have some rather large datasets that I'm working with. Essentially, I'm running some of the tools from scikit-learn on memory-mapped numpy arrays as it seems to allow me to work with larger datasets than the memory on my computer would otherwise…
none
- 1,187
- 2
- 13
- 17
-1
votes
1 answer
A Python function did not work correctly when run on multiple cores, but runs normally on a single core" grammatically correct
This function did not work correctly when n_jobs!=1
from joblib import Parallel, delayed
import numpy as np
import pandas as pd
def foo(n_jobs):
result = {}
x = np.sin(4 * np.pi * np.arange(0, 1, 0.001))
y = np.sin(8 * np.pi *…
lllIlllllIll
- 19
- 4
-1
votes
1 answer
Access File In Another Directory (Python)
I tried to access files with with open but the files itself located not in the same folder (since I want to access files in many different folders).
import os
path = r"C:\Users\M S I\Desktop\TA MACHINE LERANING\MODEL SKENARIO UJI 1" # here contained…
Rizky Alvian
- 23
- 3