I’m new to programming (unless some fortran coding in 1990s counts), totally new to Python and I came in via Google Colab, where I ran into crashes when using large input files. So I got a new laptop (HP Zbook), and now use Anaconda & Jupyterlab under Windows. Youtube support on specific topics have mostly increased my confusion, so I turn to StackOverflow, yet fearing I might not understand the jargon in the answers.
I managed to make some simple code work, computing unique distances between a bunch of 3D points (typically 800000000). I’m currently running cell by cell to monitor time expenditure and check where I should optimize. When progressively slicing [0::100] time was 14s, [0::50] time was 1min, [0::30] time was 2,75min, [0::20] time was 6,5min, and at [0::10] I got the error message.
However I’m getting what looks like a time out of a cell after just 12min ? I have no clue where this 12min comes from and where I can go change this.
The error I got :
MemoryError Traceback (most recent call last) File <timed exec>:12
File <timed exec>:12, in <listcomp>(.0)
MemoryError:
What I tried :
I was hoping to find in settings of the JupyterLab menu something where I would find the number "12" that I could set to a higher value, but I didn't find anything. I didn't dare venture into advanced settings.
Code :
In one cell I have:
import math
from itertools import combinations
def dist(p1, p2):
(x1, y1, z1), (x2, y2, z2) = p1, p2
return math.sqrt((x2 - x1)**2 + (y2 - y1)**2+ (z2 - z1)**2)
In the cell that times out I have:
%%time
# computed unique FSAD distances on a subsample
x = ToBePLotted_FSAD_X[0::10]
y = ToBePLotted_FSAD_Y[0::10]
z = ToBePLotted_FSAD_Z[0::10]
"""
x = ToBePLotted_FSAD_X
y = ToBePLotted_FSAD_Y
z = ToBePLotted_FSAD_Z
"""
points = list(zip(x,y,z))
FSADdistances = [dist(p1, p2) for p1, p2 in combinations(points, 2)]
