1

I am wondering which technique is used to learn the Dirichlet priors in Mallet's LDA implementation.

Chapter 2 of Hanna Wallach's Ph.D. thesis gives a great overview and a valuable evaluation of existing and new techniques to learn the Dirichlet priors from the data.

Tom Minka initially provided his famous fixed-point iteration approach, however without any evaluation or recommendations.

Furthermore, Jonathan Chuang did some comparisons between previously proposed methods, including the Newton−Raphson method.

LiangJie Hong says the following in his blog:

A typical approach is to utilize Monte-Carlo EM approach where E-step is approximated by Gibbs sampling while M-step is to perform a gradient-based optimization approach to optimize Dirichlet parameters. Such approach is implemented in Mallet package.

Mallet mentions the Minka's fixed-point iterations with and without histograms.

However, the method that is actually used simply states:

Learn Dirichlet parameters using frequency histograms

Could someone provide any reference that describes the used technique?

J.Schneider
  • 384
  • 3
  • 12

1 Answers1

1

It uses the fixed point iteration. The frequency histograms method is just an efficient way to calculate it. They provide an algebraically equivalent way to do the exact same computation. The update function consists of a sum over a large number of Digamma functions. This function by itself is difficult to compute, but the difference between two Digamma functions (where the arguments differ by an integer) is relatively easy to compute, and even better, it "telescopes" so that the answer to Digamma(a + n) - Digamma(a) is one operation away from the answer to Digamma(a + n + 1) - Digamma(a). If you work through the histogram of counts from 1 to the max, adding up the number of times you saw a count of n at each step, the calculation becomes extremely fast. Initially, we were worried that hyperparameter optimization would take so long that no one would do it. With this trick it's so fast it's not really significant compared to the Gibbs sampling.

David Mimno
  • 1,836
  • 7
  • 7
  • Just to be sure: Does Mallet implement "Method 1: Using the Digamma Recurrence Relation" (Algorithm 2.2) from [Wallach's thesis, p.27](https://people.cs.umass.edu/%7Ewallach/theses/wallach_phd_thesis.pdf) with the described histogram technique and additionally introduced gamma hyperpriors (section 2.5)? – J.Schneider May 21 '21 at 15:59
  • 1
    Yes, [the code](https://github.com/mimno/Mallet/blob/4b232a85d262fc3f40af42ddf6e512036cc58ff0/src/cc/mallet/types/Dirichlet.java#L583) is using that method with a gamma(shape=1.00001, scale=1) hyper-hyper prior. – David Mimno May 22 '21 at 01:59