The general topic is mesh adaptation, more specifically isotropic mesh adaptation (size map).
If you want to keep connectivity of the cells, you're looking at r-adataption, or smoothing.
Now, I doubt you'll find any software to solve your problem, simply because mesh adaptation software always assumes a fixed boundary. These are used to produce better meshes for numerical schemes, where the domain is fixed throughout adaptation/simulation steps. This is because the domain boundary represents something of interest geometrically, like a mechanical piece or an airplane in a fluid simulation. But here, you need your quads to "push" outer edges further out, or allow them to come further in.
This also means your problem is ill-posed. Unless you have constraints you're not telling us about, there's an infinity of solutions to your problem; for instance, if you find a solution, you can just offset it and that's still a solution. This does not bode well for optimization approaches.
Still, you could try optimizing a quality function depending on the vertices. First, define the element quality function

Then, for every point P belonging to the elements
, define (for example):

I put k in there as a parameter to play with. Higher values of k (don't go crazy) will minimize more strongly the maximum value of Q_K. You can also take the maximum over the element qualities, but that wouldn't be smooth (no derivatives).
This doesn't solve the well-posedness. Any solution will be a saddle point with flat directions. This is generally not very appreciated by optimization algorithms. To address this, you could simply add a penalization term such as

to the quality at P, where P_0 is the initial (before optim) position of P. You could even set the penalization to 0 if the distance is lower than some threshold that seems reasonable to you, and start increasing from there. Better yet, only add this term to boundary points, not the interior ones who should be free to move.
Finally, you'll want to put all of that in a function, and give it to an optimization method. If you don't want to compute derivatives, you can use Nelder-Mead, the "improved simplex method", DIRECT, and many others. This mostly depends on what packages/libraries you have available.
EDIT: Forgot to mention, you can look at [Knupp, Patrick M. "Algebraic mesh quality metrics." 2001] for other possible element quality functions.