0

I need to do data fitting to find the distribution of a given data.

I need to find the pdf function of the distribution.

I can use data fitting functions in matlab and python.

It looks like a truncated gamma.

But, how to find the parameters of the distribution ?

What if the data cannot fit the truncated gamma well ?

The QQ-plot (qunatile-quantile) show that it is not a good fit for truncated gamma.

How to find the distribution parameters such as alpha (shape), beta (scale) for the truncated gamma ?

If data fitting cannot work here, what other methods I can use for that ?

Any help would be appreciated.

Brian Tompsett - 汤莱恩
  • 5,753
  • 72
  • 57
  • 129
user3356689
  • 55
  • 1
  • 9
  • see `fit` function in [`sccipy.stats.gamma`](http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.gamma.html#scipy.stats.gamma). if you want to try other distributions [here](http://docs.scipy.org/doc/scipy/reference/stats.html#continuous-distributions) is a list of distributions supported by scipy and each should have a `fit` function – behzad.nouri Mar 15 '14 at 15:23

3 Answers3

1

Check out allfitdist in Matlab.

Alternatively, consider specialty packages such as ExpertFit or EasyFit. The JMP statistical software also has a fairly easy to use distribution fitting option. All of these will evaluate goodness-of-fit criteria such as Cramer-von Mises and log likelihood estimates.

Once you've picked the functional form of the distribution, parameter values are usually estimated by maximum likelihood estimators or method of moments.

If you're planning on using the results in a simulation of some sort, you might consider just bootstrapping your sample rather than distribution fitting. Yet another option if simulating would be to run a designed experiment where you vary the distribution choice and see if alternatives have a significant impact on your results before worrying too much about fitting just the right distribution.

pjs
  • 18,696
  • 4
  • 27
  • 56
  • HI pjs, i just looked over your answer above. I trying to simulate a set of process of which i have a large data sets. In a simulation, I would like to recreate data of the process in the simulation environment. I thought distribution fitting and finding the parameters would be the solutions, so that i could take sample of a distribution for each process in the simulation. Do you have any literature on why to use bootstrapping instead of distribution fitting. I am fairly new to this subject so please excuse me if the question is "stupid" – Uis234 May 03 '16 at 18:47
  • @Uis234 Not really a suitable topic for a comment thread here. Quick response would be to suggest you look up the input modeling section of any reputable simulation text book, such as Chapter 9 of [Banks, Carson, Nelson, & Nicol](https://www.pearsonhighered.com/program/Banks-Discrete-Event-System-Simulation-5th-Edition/PGM130682.html). – pjs May 03 '16 at 19:34
0

Maybe this post can help.

I presented an example of how to find the best distribution according to BIC criterium using OpenTURNS.

You define a list 'Distribution Factories' tested_distributions = [ot.WeibullMaxFactory(), ot.NormalFactory(), ot.UniformFactory()]

then you call BestModelBIC to find the best fitting best_model, best_bic = ot.FittingTest.BestModelBIC(sample, tested_distributions)

Currently, you can choose among 30 available 'Factories' in OpenTURNS (see below). TruncatedNormalFactory is available but not yet TruncatedBetaFactory

print(ot.DistributionFactory.GetContinuousUniVariateFactories())

[Out]: 
[ArcsineFactory,
BetaFactory, 
BurrFactory, 
ChiFactory, 
ChiSquareFactory, 
DirichletFactory,
ExponentialFactory,
FisherSnedecorFactory,
FrechetFactory,
GammaFactory,
GeneralizedParetoFactory,
GumbelFactory,HistogramFactory,
InverseNormalFactory,
LaplaceFactory,LogisticFactory,
LogNormalFactory,
LogUniformFactory,
MeixnerDistributionFactory,
NormalFactory,
ParetoFactory,
RayleighFactory,
RiceFactory,
StudentFactory,
TrapezoidalFactory,
TriangularFactory,
TruncatedNormalFactory,
UniformFactory,
WeibullMaxFactory,
WeibullMinFactory]
#30
Jean A.
  • 291
  • 1
  • 17
0

The distfit library can all the points you are asking for. It searches across 89 theoretical distributions to find the best with the loc, scale arg parameters. Example:

I need to do data fitting to find the distribution of a given data.

pip install distfit

from distfit import distfit
import numpy as np

# Create dataset
X = np.random.normal(0, 2, 1000)

# Default method is parametric.
dfit = distfit(distr='popular') # 'all' for all 89 or specify your own list.
# Search for best theoretical fit on your empirical data
results = dfit.fit_transform(X)

# [distfit] >INFO> fit
# [distfit] >INFO> transform
# [distfit] >INFO> [norm      ] [0.00 sec] [RSS: 0.00221125] [loc=0.002 scale=1.902]
# [distfit] >INFO> [expon     ] [0.00 sec] [RSS: 0.189094] [loc=-5.527 scale=5.529]
# [distfit] >INFO> [pareto    ] [0.00 sec] [RSS: 0.189094] [loc=-536870917.527 scale=536870912.000]
# [distfit] >INFO> [dweibull  ] [0.03 sec] [RSS: 0.00285214] [loc=0.005 scale=1.667]
# [distfit] >INFO> [t         ] [0.18 sec] [RSS: 0.00221128] [loc=0.002 scale=1.902]
# [distfit] >INFO> [genextreme] [0.08 sec] [RSS: 0.00295224] [loc=-0.699 scale=1.892]
# [distfit] >INFO> [gamma     ] [0.02 sec] [RSS: 0.00221712] [loc=-2156.094 scale=0.002]
# [distfit] >INFO> [lognorm   ] [0.13 sec] [RSS: 0.00235342] [loc=-149.018 scale=149.010]
# [distfit] >INFO> [beta      ] [0.03 sec] [RSS: 0.00208639] [loc=-11.915 scale=24.259]
# [distfit] >INFO> [uniform   ] [0.00 sec] [RSS: 0.122291] [loc=-5.527 scale=11.639]
# [distfit] >INFO> [loggamma  ] [0.06 sec] [RSS: 0.00209695] [loc=-376.616 scale=55.771]
# [distfit] >INFO> Compute confidence intervals [parametric]

fig, ax = plt.subplots(1, 2, figsize=(20, 10))
dfit.plot(chart='PDF', ax=ax[0])
dfit.plot(chart='CDF', ax=ax[1])

PDF/CDF plot

The best fit can be found in the results which will answer this question:

I need to find the pdf function of the distribution.

print(results['model'])

{'distr': <scipy.stats._continuous_distns.beta_gen at 0x155d67e35b0>,
 'stats': 'RSS',
 'params': (19.490647075756037,
  20.18413144061353,
  -11.915134641255602,
  24.25907054997436),
 'name': 'beta',
 'model': <scipy.stats._distn_infrastructure.rv_continuous_frozen at 0x15583c00760>,
 'score': 0.002086393123419647,
 'loc': -11.915134641255602,
 'scale': 24.25907054997436,
 'arg': (19.490647075756037, 20.18413144061353),
 'CII_min_alpha': -3.124111310171669,
 'CII_max_alpha': 3.141196658807374}

But, how to find the parameters of the distribution ?

results['model']['params']

What if the data cannot fit the truncated gamma well ?

# Take one of the other fitting models.
results['summary'][['distr', 'score']]

# Plot results
dfit.plot_summary()

Fit results

The QQ-plot (qunatile-quantile) show that it is not a good fit for truncated gamma.

# Make qqplot for the best fit.
dfit.qqplot(X)

QQplot

# Inspect all other fits
dfit.qqplot(X, n_top=10)

QQplot other fits

If data fitting cannot work here, what other methods I can use for that ?

You can use non-parametric methods:

# Quantile method
dfit = distfit(method='quantile')

# Percentile method
dfit = distfit(method='percentile')

Any help would be appreciated.

Disclaimer: I am also the author of this repo.

erdogant
  • 1,544
  • 14
  • 23