Dan's answer is probably the best way to go. Since it isn't clear whether you need a minimum hull or a spline-fit to the extreme points, I'll just suggest the latter as an option. Roughly speaking, decide what a reasonable minimum radius should be, then scan from 0 to 2*pi , collecting the points of max radius (so long as that's greater than your selected minumum) at each sampled angle. The fit a spline to that set of points.
This sort of problem can get extremely complicated, as in this paper, BTW.
Edit: to answer Dan's question:
Matlab's got nothing builtin that I know of; I would choose the center to be one of the centroids: The mean x
, y
values of all points is a standard method. If you have a theory as to the underlying distribution, e.g., 2D gaussian, you could try fitting the data to a distribution function and interpolating the centroid thereof.