I am struggling to setup dispy and SSL. Here are steps i made to setup:
To create a certificate i used command:
openssl req -x509 -newkey rsa:4096 -sha256 -nodes -keyout private.key -out private.crt -days 3650
Then i merged two files:
cat private.crt private.key > private.pem
In my particular case i need to use all: scheduler (dispyscheduler), node (dispynode) and cluster (SharedJobCluster).
Here is a job i submit:
# canonical.py
# function 'compute' is distributed and executed with arguments
# supplied with 'cluster.submit' below
def compute(n):
import time, socket
time.sleep(n)
host = socket.gethostname()
return (host, n)
if __name__ == '__main__':
# executed on client only; variables created below, including modules imported,
# are not available in job computations
import dispy, random
# distribute 'compute' to nodes; in this case, 'compute' does not have
# any dependencies to run on nodes
cluster = dispy.SharedJobCluster(compute,
port=0,
certfile='C:\\project\\private.pem')
# run 'compute' with 20 random numbers on available CPUs
jobs = []
for i in range(5):
job = cluster.submit(random.randint(5, 5))
job.id = i # associate an ID to identify jobs (if needed later)
jobs.append(job)
# cluster.wait() # waits until all jobs finish
for job in jobs:
host, n = job() # waits for job to finish and returns results
print('%s executed job %s at %s with %s' % (host, job.id, job.start_time, n))
# other fields of 'job' that may be useful:
# job.stdout, job.stderr, job.exception, job.ip_addr, job.end_time
cluster.print_status() # shows which nodes executed how many jobs etc.
If i don't use an SSL connection it successfully works despite platform was used. Scheduler on Linux and node on Windows and vice versa. Any combination works just fine. But problem comes when i try to use SSL.
Here are commands i have been using to run test job:
On Linux
dispynode.py -d --certfile=/home/dismine/project/private.pem --clean
On Windows
python "C:\Program Files\Python3\Lib\site-packages\dispy\dispyscheduler.py" -d --ip_addr 127.0.0.1 --ip_addr 192.168.0.103 --node_certfile C:\project\private.pem --cluster_certfile C:\project\private.pem
python canonical.py
Such a configuration - connection between Windows and Unix doesn't work for some reasons. I constantly see errors like these:
1.
Reading standard input disabled, as multiprocessing does not seem to workwith re
ading input under Windows
2018-01-03 12:50:08 dispynode - dispynode version: 4.8.3, PID: 3288
2018-01-03 12:50:08 pycos - version 4.6.5 with IOCP I/O notifier
2018-01-03 12:50:08 dispynode - "Роман-ПК" serving 2 cpus
2018-01-03 12:50:08 dispynode - TCP server at 192.168.0.101:51348
2018-01-03 12:50:08 pycos - uncaught exception in !tcp_req/36277304:
Traceback (most recent call last):
File "C:\Program Files\Python3\lib\site-packages\pycos\__init__.py", line 3730, in _schedule
retval = task._generator.throw(*exc)
File "C:\Program Files\Python3\Lib\site-packages\dispy\dispynode.py", line 988, in tcp_req
msg = yield conn.recv_msg()
File "C:\Program Files\Python3\lib\site-packages\pycos\__init__.py", line 3732, in _schedule
retval = task._generator.send(task._value)
File "C:\Program Files\Python3\lib\site-packages\pycos\__init__.py", line 882,
in _async_recv_msg
data = yield self.recvall(n)
File "C:\Program Files\Python3\lib\site-packages\pycos\__init__.py", line 1438, in _iocp_recvall
self._read_result = win32file.AllocateReadBuffer(bufsize)
MemoryError
2.
Reading standard input disabled, as multiprocessing does not seem to workwith re
ading input under Windows
2018-01-03 12:54:12 dispynode - dispynode version: 4.8.3, PID: 3284
2018-01-03 12:54:12 pycos - version 4.6.5 with IOCP I/O notifier
2018-01-03 12:54:12 dispynode - "Роман-ПК" serving 2 cpus
2018-01-03 12:54:12 dispynode - TCP server at 192.168.0.101:51348
2018-01-03 12:54:13 pycos - uncaught exception in !tcp_req/36342840:
Traceback (most recent call last):
File "C:\Program Files\Python3\lib\site-packages\pycos\__init__.py", line 3730, in _schedule
retval = task._generator.throw(*exc)
File "C:\Program Files\Python3\Lib\site-packages\dispy\dispynode.py", line 988, in tcp_req
msg = yield conn.recv_msg()
File "C:\Program Files\Python3\lib\site-packages\pycos\__init__.py", line 3732, in _schedule
retval = task._generator.send(task._value)
File "C:\Program Files\Python3\lib\site-packages\pycos\__init__.py", line 882,
in _async_recv_msg
data = yield self.recvall(n)
File "C:\Program Files\Python3\lib\site-packages\pycos\__init__.py", line 1438, in _iocp_recvall
self._read_result = win32file.AllocateReadBuffer(bufsize)
OverflowError: Python int too large to convert to C long
3.
2018-01-03 12:59:54 dispyscheduler - dispyscheduler version 4.8.3
2018-01-03 12:59:54 pycos - version 4.6.5 with epoll I/O notifier
Enter "quit" or "exit" to terminate scheduler, anything else to get status: 2018-01-03 12:59:54 dispyscheduler - Scheduler at 192.168.0.103:51349
2018-01-03 12:59:54 dispyscheduler - TCP server at 127.0.1.1:51347
2018-01-03 12:59:54 dispyscheduler - TCP server at 192.168.0.103:51347
2018-01-03 12:59:54 dispyscheduler - Scheduler at 127.0.1.1:51349
2018-01-03 12:59:54 pycos - uncaught exception in !tcp_req/139968956623560:
Traceback (most recent call last):
File "/home/dismine/project/env/lib/python3.5/site-packages/pycos/__init__.py", line 3730, in _schedule
retval = task._generator.throw(*exc)
File "/home/dismine/project/env/bin/dispyscheduler.py", line 368, in tcp_req
msg = yield conn.recv_msg()
File "/home/dismine/project/env/lib/python3.5/site-packages/pycos/__init__.py", line 3730, in _schedule
retval = task._generator.throw(*exc)
File "/home/dismine/project/env/lib/python3.5/site-packages/pycos/__init__.py", line 868, in _async_recv_msg
data = yield self.recvall(n)
File "/home/dismine/project/env/lib/python3.5/site-packages/pycos/__init__.py", line 461, in _recvall
recvd = self._rsock.recv_into(view, len(view), *args)
File "/usr/lib/python3.5/ssl.py", line 929, in recv_into
return self.read(nbytes, buffer)
File "/usr/lib/python3.5/ssl.py", line 791, in read
return self._sslobj.read(len, buffer)
File "/usr/lib/python3.5/ssl.py", line 575, in read
v = self._sslobj.read(len, buffer)
ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1977)
Unfortunately my skills not enough to understand what is wrong with dispy or my settings. Why in case of using only Unix system or only Windows systems it works, but not together.
I also submitted issue on GitHub.
