I made a program in C# that generates maps in a procedural way.
After generating the map, I need to somehow smooth out the edges, but when I do this via software, the output doesn't seem natural.
This is the original map:
This is the original software-smoothed map:
This is the map that I did the manual smoothing.
I would like to create an algorithm that would smooth the edges of the map in a more natural way where the expected result is something like the manual smoothing I did above.
If I'm interpreting your figures correctly, the original map is a 64x64 binary-valued image while the software-smoothed map is 256x256, and the smoothed map seems to have been generated by 4x upscaling and/or smoothing followed by thresholding.
To get smoother edges, you need stronger smoothing before the thresholding. Aka anti-aliasing, as Ian Chu hinted at in the comments.
Here is a simple approach:
Generate the 64x64 map.
Upscale the map by 4x to 256x256 by nearest neighbor, that is, replicate each pixel 4 times in each dimension.
Apply a Gaussian smoothing filter. The standard deviation or sigma parameter of the Gaussian determines the smoothing strength. A sigma of 4 pixels seems about right.
Threshold the filtered image to get a binary-valued result.
Here is what I get on the example map that you provided:
To implement the Gaussian filtering step, you might find the tips in this SO post useful.
Or if you rather implement the filtering yourself, an easy way to get something resembling Gaussian smoothing is by repeatedly applying the 5-tap smoothing filter [0.2, 0.2, 0.2, 0.2, 0.2] along each dimension. Apply the filter, then take the resulting image and apply the filter to it again, and so on. Repeat the filter application say 5 times, or more for stronger smoothing.
