Stack Overflow
Stack Overflow

Questions tagged [joblib]

Joblib is a set of tools to provide lightweight pipelining in Python.

Joblib is a set of tools to provide lightweight pipelining in Python.

https://joblib.readthedocs.io/en/latest/

715 questions
0
votes
1 answer

How to plot the tree of a Light GBM .joblib model?

I'm very new to Machine Learning! My problem concern a model created with LighGBM. I'm not the creator of this model, so I want to see the tree that this model generates. The model is in the format .joblib, and I want to know as much information as…
Sguit
  • 35
  • 1
  • 8
0
votes
0 answers

Parallelize For Loops with Joblib Python

I was searching for a method to paralleize for loops in Python 3. I found joblib library that is convenient to use but cannot find a way to use in my problem. Is there a way use Joblib to parallelize the following for loop in python 3. def…
Klaus
  • 1,641
  • 1
  • 10
  • 22
0
votes
1 answer

Speed up a routine inside a process with high memory usage by using parallel processing in python

I have a program that uses a lot of memory (150gb of vectors indexed with nmslib) and I have trouble to parallelize the execution of the code. My machine has 40 cores and my attempts to parallelize it have been not successful so far. the programm…
Sebastian_學生
  • 345
  • 1
  • 11
0
votes
0 answers

LDA model and joblib parallel computing error

I have some code that does an LDA model on a bunch of CSV lines. lda_model = LatentDirichletAllocation( n_components=20, # Number of topics max_iter=10, # Max learning iterations learning_method='online', …
JLUser
  • 21
  • 6
0
votes
1 answer

Python: memmap list of objects become 'None' type inside joblib parallel

I am doing the following: I have a list of tensorflow DNN layers. nn.append(tf.layers.dense(...)) Each of the above list is appended to a list of np.memmap objects. nnList[i] = nn I can access the memmap list and retrieve the tensors. But when try…
Tirtha R
  • 1,148
  • 1
  • 14
  • 24
0
votes
1 answer

Precision loss in shared multiprocessing array?

I am experimenting with the multiprocessing Array type to share a writeable object between processes. I have read the many cautions about doing so, but believe it is necessary in my application due to physical memory constraints. It appears that…
ekrose
  • 94
  • 1
  • 1
  • 7
0
votes
1 answer

directly use MemmapingPool in python joblib module

the joblib module has provided a tremendously easy-to-use function Parallel to simplify coding. However, it always gathers all the results before you can access any of them. I have the need to deal with the results one by one because the results…
doraemon
  • 2,296
  • 1
  • 17
  • 36
0
votes
0 answers

Joblib doesn't work in Jupyter lab

Firstly I use the joblib.dump command in my neural network training notebook, in order to dump my pipeline that is fitted on my input data: joblib.dump(prepareinput, "prepareinput.save") Then when I try to load the data in a different notebook in…
Gijsv
  • 47
  • 7
0
votes
0 answers

Multiprocessing with JDBC connection and pooling

I would like to create a parallel process which gets Data from a Database. I am using a JDBC connector which works quite fine if i run my program not in parallel: conn = jaydebeapi.connect("com.teradata.jdbc.TeraDriver",…
Mimi Müller
  • 416
  • 8
  • 25
0
votes
2 answers

Parallelizing for loop in Python 2.7

I'm very new to Python (and coding in general) and I need help parallising the code below. I looked around and found some packages (eg. Multiprocessing & JobLib) which could be useful. However, I have trouble using it in my example. My code makes an…
Tim D
  • 111
  • 1
  • 2
  • 7
0
votes
2 answers

Python parallel computation without pickling

I have a very simple list comprehension I would like to parallelize: nlp = spacy.load(model) texts = sorted(X['text']) # TODO: Parallelize docs = [nlp(text) for text in texts] However, when I try using Pool from the multiprocessing module like…
James Ko
  • 32,215
  • 30
  • 128
  • 239
0
votes
1 answer

Joblib Unable to unpickle class properly

I am getting Attribute error while trying to unpickle my object using joblib. Should I separately pickle my classes and then unpickle them? class LengthTransformer(TransformerMixin): def transform(self, X, **transform_params): length =…
joel
  • 1,156
  • 3
  • 15
  • 42
0
votes
1 answer

Multiprocessing with complex function

I want to create a Multiprocess for a quite complex function in python: I have tested this function with a less complex code like this: from joblib import Parallel, delayed, parallel_backend from joblib import load, dump def print_hello(hallo,…
Mimi Müller
  • 416
  • 8
  • 25
0
votes
1 answer

Docker uwsgi-nginx-flask with joblib, unable to find local function, but works in standalone flask

When I tried to load a pre-trained model thru joblib inside a docker container getting following error. web_1 | 2018-02-06 15:11:50,826 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) web_1 |…
Sekar S
  • 163
  • 1
  • 10
0
votes
0 answers

Python Multiprocessing Pool Loop Read and Write Text file

I'm trying to parallelize a loop for reading and writing a significant amount of text file using multiprocessing module. This is my problem: I have a class named SWAT which includes several methods. A method called "auto" try to automate all other…
Reza Ehsani
  • 1
1 2 3
…
47 48