Stack Overflow
Stack Overflow

Questions tagged [dispy]

dispy is a Python module for distributed and parallel computing.

dispy is a module for distributed and parallel computing.

Useful links

21 questions
2
votes
0 answers

Python - dispy Port 51347 seems to be used by another program

I am trying to run the program on the dispy page (http://dispy.sourceforge.net/) and I know this question is very similar to this one once asked (Port 51347 seems to be used by another program), but please do not file as duplicate because I have…
J.Doe
  • 21
  • 3
2
votes
0 answers

Dispy, initiating a SharedJobCluster on a compute node

I am creating a compute cluster in python using dispy. One of my use cases would be very nicely solved by starting a process on a compute node that itself starts a distributed process. As such, I have implemented the SharedJobCluster on the primary…
user3536741
  • 21
  • 4
2
votes
4 answers

dispy sample program hangs

TL;DR: I can't get the most basic dispy sample code to run properly. Why not? The details: I'm trying to get into distributed processing in python, and thought the dispy library sounded interesting, due to the comprehensive feature set. However,…
Scott Mermelstein
  • 15,174
  • 4
  • 48
  • 76
2
votes
1 answer

Cluster with Python and Dispy: cannot connect node

I'm trying to create a raspberry-pi cluster using dispy for a school project. Just to try it, I'm using only one RPi as a node and my pc as master that request jobs. Below the devices and their ip(s): master (pc) --> 192.168.1.106 node (rpi) -->…
xdola
  • 562
  • 2
  • 8
  • 23
2
votes
1 answer

Why does dispynode return "ignoring ping" when contacted by dispy client?

I'm working with a cluster of Centos6.5 servers, where I have one head node and the rest are slave nodes. The nodes are connected through a switch on a local network 192.168.1.x that's not visible to the outside world. I'm trying to use Python Dispy…
Doug
  • 285
  • 2
  • 13
1
vote
0 answers

Propriate way to setup dispy and SSL

I am struggling to setup dispy and SSL. Here are steps i made to setup: To create a certificate i used command: openssl req -x509 -newkey rsa:4096 -sha256 -nodes -keyout private.key -out private.crt -days 3650 Then i merged two files: cat…
dismine
  • 575
  • 13
  • 17
1
vote
3 answers

import modules in python jobs with dispy

I'm working with a program that works in parallel execution with dispy. I'm using dispy to create tasks and then distribute it to different CPUs to execution. I have standar libraries and developed by me libraries (data and connection). The code is…
A77ak
  • 35
  • 9
1
vote
1 answer

Using dispy with port forwarding via ssh tunnel

I have dispynode running on a remote server. I'm trying to open an SSH tunnel from my computer (client) and configure dispyJobCluster to use this tunnel. But it's not working. Am I not configuring this right ? Here's how I'm doing this : ( p.s . i…
Joe S
  • 21
  • 6
1
vote
1 answer

Is there a way to pass a variable to setup() in dispy?

I need to pass a variable to the setup() method of dispy node so i can tell the node which data set to load from a config file. Else i have to write a specific script for each data set and this is going to be painful. def setup(): # executed on each…
CpILL
  • 6,169
  • 5
  • 38
  • 37
0
votes
0 answers

How to resolve python OSError 64 raised by dispy

I am having troubles tracking down a mysterious error message in Python 3.8. We're using dispy, and we randomly get the error message OSError: 64: pycos - uncaught exception in tcp_req: Traceback (most recent call last): File…
dbc
  • 677
  • 8
  • 21
0
votes
1 answer

Running dispynode.py on multiple EC2 instances using SSH appears to work but it really doesn't

I've created a cluster of EC2 instances using cfncluster and now I need to run the dispynode.py command on all the nodes. I do that by first creating a list of private IP addresses called "workers.txt" then running the following bash command for…
ApprenticeOfMathematics
  • 195
  • 1
  • 6
0
votes
0 answers

Are Python's output streams thread-safe?

I'm using Dispy/pycos to run multiple jobs across a multitude of machines. The results of computations come back to the main script, which writes them out into a set of files -- from different threads. Do I need to explicitly synchronize output…
Mikhail T.
  • 3,043
  • 3
  • 29
  • 46
0
votes
1 answer

Can the cluster_status callback submit new jobs?

I'm developing a dispy script operating on a large number of objects (duh). There are multiple operations to be performed for each object -- only one of them must happen first, the rest use the first job's results and can be done in any…
Mikhail T.
  • 3,043
  • 3
  • 29
  • 46
0
votes
1 answer

Computation Sharing Among Multiple EC2 Instances on AWS

I am working on problem where computational power required to be split across multiple EC2 instances. I am considering matrix multiplication program , for example huge matrix with thousands of rows and column requires GB's of ram and computation…
Nagendra Nigade
  • 866
  • 2
  • 12
  • 28
0
votes
1 answer

global cluster object for dispy

(1) Can I have the cluster object global across machines so that once a job is submitted, that job in turn can submit other dissimilar smaller jobs? cluster = dispy.JobCluster(compute) (2) Can the "compute" function be different each time I invoke…
Tims
  • 627
  • 7
  • 19
1
2