Here's what I would do. I will focus on a simplification of your problem, namely I will mark every line as "to be bent" if it overlaps with any other - Overlaps are bad. I'm not gonna make the distinctions that you did, as they are additional and not simplifying. Your questions then essentially assume the form:
Given a set of line segments in the plane, find the ones that overlap with others
If you still want to distinguish the cases you made, you will have to consider that they are ordered and do some extra checks. You could use an extra binary coordinate for their direction.
The first question is, how do we store the line segments? You seem to treat them specified as two points (x1,y1) and (x2,y2), which is why you do N^2 comparisons. I would suggest you threat them as lines with start and end point. That means, we are going to use as coordinates their slope m, y-axis intercept y, as well as x1, x2,
Finding y and m should not be hard.
Unfortunately this won't work for the y-axis itself. If you can't be sure that you have segments on the y-axis, you might need to modify this approach or think of some alternative for this specific case. Maybe give these a special key and keep the (y1,y2) values.
My suggestion now would be to store your lines in a hash table (maybe something like pythons dictionary) with key (m,y) and a list content [(x1,x2),(x1',x2'),...]. This allows you, for every segment, to only check the segments that lie on the same extended line.
You can then go through all your lines once and mark the ones that overlap with another of the same key. Don't forget to use fast overlap test to do so.
If you want to pump up efficiency even further, you might want to take a look at Interval trees. I'm not too familiar with them myself, but theoretically you could use this data structure, or some modification thereof, for your intervals in the list for a given key to optimize finding overlapping ones. This is going to be important if you have many steps on one specific line. You could mark a line as bent as soon as it has an overlap with another in this tree.
Your algorithm might look something like:
- Project line segments onto slope and y-intersect (i.e. make the hash table)
- Loop through the hash keys
- For every element in the list, check if it overlaps with another one (this is where Interval trees might help you speed up further. It's also where your case distinction might come in)
- Mark the line as "to be bent" if the check is positive
Adressing some of the comments: Using only angle from x-axis won't help. Also, remembering the direction you came from won't help, as the following example should illuminate.

Note that, what exactly you test will depend on your desired outcome. You can test for overlap or maybe you can test if an interval is fully contained in another.
It might also be worthwhile to read a bit into graph libraries and their visualizations, as your problem essentially describes a directed graph with fixed node positions.
Hope some of this is of help to you.