Your question is very complicated to understand. The overall feeling is that your expectations are too high. Some advanced math 101 might help (feel free to google buzzwords):
Defining a curve is an very hard problem that challenged the brightest mathematicians of the history. From the naive approach of the greeks, through the calculus of Newton and Leibniz, passing by Euler and Gauss, to the mathematical analysis of Weisstreiss, the word curve changed meaning several times. The accepted definition nowadays says that curves are continous functions in two variables, where continous is a very special word that has an exact meaning coined in the 19th century (naively is a function without jumps
from one value to another). Togheter with the notion of continuity, came the notions of connected, compact, differentiable (and so on) curves, which defined new conditions for special curves. The subject developed to what is now known as topology and mathematical analysis.
Mathematicians usually uses definitions to reproduce a class of ideas that can be brought and thought togheter. To their surprise, the definition of continuity did include really weird functions to be curves: space-filling-curves, fractals!!! They called them monsters at the time.
After this introduction, lets go back to your question. You need a geometrical object to calculate distances from a point. Lets avoid weird curves and go from continous to differentiable. Now it's better. A (conected compact) differentiable function can be expanded in Taylor series, for example, which means that all functions of this class can be written as an infinite sum of polynomial functions. In two dimensions, you need to calculate matrices involved in this expansion (Calculus in many variables is a pre-requisite). Another step further is truncating this expansion in some degree, lets say 3. Then the general curve in this case is: ax + by + cx^2 + dy^2 + ex^3 + fy^3 + gx^2y + hxy^2 + ixy + j = 0
(ab...j are free parameters). Oh! This is reasonable, you might think. Well, actually there is a name for this kind of curve: algebraic curve of deggre 3. This is an active research theme of algebraic geometry, which is a very hard field even among mathematicians. Generally speaking, there are milestone theorems about the general behavior of those curves, which involves singularities and intersection points that are allowed in the general case.
In essence, what you are looking for does not exist, and is a very hard subject. Your algorithm works with points (really cool pictures by the way) and you should baby step it into a straight line. This step already requires you to think about how to calculate distance between a point and a straight line. This is another subject that was developed in general in the 19th century, togheter with mathematical analysis: metric spaces. The straightfoward answer to this question is defining the distance between a point and a line to be the smallest distance from the point to all line points. In this case, it can be shown that the distance is the modulus of the vector that connects the point to the line in a 90 degrees angle. But this is just one definition among infinte possible ones. To be considered a distance (like the one I just described and the euclidean distance) there is a set of axioms that needs to be verified. You can have hyperbolic metrics, discrete metrics, metrics that count words, letters, LotsOfFamousPeople metric spaces... the possibilities are infinite.
So, baby steps. Do it with straight lines and euclidean minimum distance metric. Play around with other metrics you find on google. Understand the axioms and make your own!!! Going to second degree polynomials is already a big challenge, as you have to understand everything that those curves can make (they can really do weird unexpect stuff) and define a distance to it (metric space).
Well thats it! Good luck with your project. Looks really cool!