1

So I have this issue where I have to find the best distribution that, when passed through a function, matches a known surface. I have written a script that creates the distribution given some parameters and spits out a metric that compares the given surface to the known, but this script takes a non-negligible time, so I can't just run through a very large set of parameters to find the optimal set of parameters. I looked into the simplex method, and it seems to be the right path, but its not quite what I need, because I dont exactly have a set of linear equations, and dont know the constraints for the parameters, but rather one method that gives a single output (an thats all). Can anyone point me in the right direction to how to solve this problem? Thanks!

To quickly go over my process / problem again, I have a set of parameters (at this point 2 but will be expanded to more later) that defines a distribution. This distribution is used to create a surface, which is compared to a known surface, and an error metric is produced. I want to find the optimal set of parameters, but cannot run through an arbitrarily large number of parameters due to the time constraint.

Fred E
  • 131
  • 5
  • Add a `data-science` tag and more related tags so people from this field will come and help. It looks like a `ML` problem for me... – Yonlif Feb 22 '19 at 19:01
  • thanks! I added a couple more tags, including data-science. What is ML, though? I'm not familiar with the term. – Fred E Feb 22 '19 at 19:08
  • Machine learning. – Felipe Gutierrez Feb 22 '19 at 19:10
  • Oh, ok, thank you! I added that tag as well. – Fred E Feb 22 '19 at 19:11
  • Any optimization method could be appropriate, depending on how your function and error metric look like. Do you mean the [Nelder-Mead method](https://en.wikipedia.org/wiki/Nelder–Mead_method) by simplex? Have you tried it? Sounds like a good option if you don't know anything about the characteristics of the data. – Nico Schertler Feb 23 '19 at 02:42

1 Answers1

0

One situation consistent with what you have asked is a model in which you have a reasonably tractable probability distribution which generates an unknown value. This unknown value goes through a complex and not mathematically nice process and generates an observation. Your surface corresponds to the observed probability distribution on the observations. You would be happy finding the parameters that give a good least squares fit between the theoretical and real life surface distribution.

One approximation for the fitting process is that you compute a grid of values in the space output by the probability distribution. Each set of parameters gives you a probability for each point on this grid. The not nice process maps each grid point here to a nearest grid point in the space of the surface. The least squares fit is a quadratic in the probabilities calculated for the first grid, because the probabilities calculated for a grid point in the surface are the sums of the probabilities calculated for values in the first grid that map to something nearer to that point in the surface than any other point in the surface. This means that it has first (and even second) derivatives that you can calculate. If your probability distribution is nice enough you can use the chain rule to calculate derivatives for the least squares fit in the initial parameters. This means that you can use optimization methods to calculate the best fit parameters which require not just a means to calculate the function to be optimized but also its derivatives, and these are generally more efficient than optimization methods which require only function values, such as Nelder-Mead or Torczon Simplex. See e.g. http://commons.apache.org/proper/commons-math/apidocs/org/apache/commons/math4/optim/package-summary.html.

Another possible approach is via something called the EM Algorithm. Here EM stands for Expectation-Maximization. It can be used for finding maximum likelihood fits in cases where the problem would be easy if you could see some hidden state that you cannot actually see. In this case the output produced by the initial distribution might be such a hidden state. One starting point is http://www-prima.imag.fr/jlc/Courses/2002/ENSI2.RNRF/EM-tutorial.pdf.

mcdowella
  • 19,301
  • 2
  • 19
  • 25
  • I haven't looked at Torczon yet, but Nelder-Mead is already extremely promising, and looks to be almost exactly what I need! Thank you very much! – Fred E Feb 25 '19 at 15:48