Gaussian Mixture Model clustering in sparklyr

Question

I'm trying to cluster my data using Gaussian Mixture Model in sparklyr:

ml_gaussian_mixture(formula= ~ var1 + var2 + var3 + var4 + var5, k = 5)

However, calling this function doesn't return a metric to evaluate the number of clusters as ml_kmeans() does (this function returns WSSSE). Is there a way to get the Silhouette score or BIC for ml_gaussian_mixture() in sparklyr ?

kangaroo_cliff · Answer 1 · 2019-10-26T02:20:42.453

0

With

gmm_model <- ml_gaussian_mixture(iris_tbl, Species ~ .)

you can get the log-likelihood as

gmm_model$summary$log_likelihood

Which you can then use to get BIC or AIC.

I'm sure there must be a way to get it directly though. But if not, you may calculate BIC as

log(n) + k-1 + k * p + k * p * (p-1) / 2 - 2 * gmm_model$summary$log_likelihood

Where n - number of samples, k - number of clusters, p- number of variables. In above, the k-1 + k * p + k * p * (p-1) / 2 is the number of free-parameters in a Gaussian mixture model (with unristricted co-variance matrices)

Example:

library(sparklyr)
sc <- spark_connect(master = "local")
iris_tbl <- sdf_copy_to(sc, iris, name = "iris_tbl", overwrite = TRUE)
gmm_model <- ml_gaussian_mixture(iris_tbl, Species ~ .)

gmm_model$summary$log_likelihood
#[1] -294.1398

edited Oct 26 '19 at 02:20

answered Oct 25 '19 at 03:26

kangaroo_cliff

6,067
3
29
42

It doesn't seem to work. I get null when calling gmm_model$summary$log_likelihood. – Egodym Oct 26 '19 at 01:33
See the example in the edit. It is possible you aren't getting a fit. – kangaroo_cliff Oct 26 '19 at 02:21
I'm curious to know how you went with this. – kangaroo_cliff Oct 26 '19 at 23:09
I'm getting a fit but still retrieving NULL when calling gmm_model$summary$log_likelihood. – Egodym Oct 29 '19 at 18:44
See if the example I have given works. If so, then it is likely the problem related to your data. – kangaroo_cliff Oct 29 '19 at 22:39
With mixture models, you are likely to get some values for model parameters even if there wasn't a good fit: difficult to calculate inverse covariance matrix, etc. If it is possible to share a version of the data you use, maybe consider asking a different question to reslove that. – kangaroo_cliff Oct 29 '19 at 22:42

Gaussian Mixture Model clustering in sparklyr

1 Answers1