How to alter gpt-2 code to work with Tensorflow 2.0?

Question

I am trying to use gpt-2 for text generation. I get compatibility errors, even after running the Tensorflow 2.0 code upgrade script.

Steps I've followed:

1. Clone repo

2. From here on out, follow the directions in DEVELOPERS.md

3. Run upgrade script on files in /src

4. In terminal run: sudo docker build --tag gpt-2 -f Dockerfile.gpu .

5. After building is done, run: sudo docker run --runtime=nvidia -it gpt-2 bash

6. Enter python3 src/generate_unconditional_samples.py | tee /tmp/samples

7. Get this traceback:

   Traceback (most recent call last):
   File "src/generate_unconditional_samples.py", line 9, in <module>  
   import model, sample, encoder
   File "/gpt-2/src/model.py", line 4, in <module>
   from tensorboard.plugins.hparams.api import HParam
   ImportError: No module named 'tensorboard.plugins.hparams'
   root@f8bdde043f91:/gpt-2# python3 src/generate_unconditional_samples.py | tee 
   /tmp/samples
   Traceback (most recent call last):
   File "src/generate_unconditional_samples.py", line 9, in <module>
   import model, sample, encoder
   File "/gpt-2/src/model.py", line 4, in <module>
   from tensorboard.plugins.hparams.api import HParam
   ImportError: No module named 'tensorboard.plugins.hparams'```
   
   

It appears that HParams has been deprecated and the new version in Tensorflow 2.0 is called HParam. However, the parameters are different. In model.py, the params are instantiated as follows:

def default_hparams():
return HParams(
    n_vocab=0,
    n_ctx=1024,
    n_embd=768,
    n_head=12,
    n_layer=12,
)

There doesn't appear to be any 1:1 translation into Tensorflow 2.0. Does anyone know how to make gpt-2 work with Tensorflow 2.0?

My GPU is an NVIDIA 20xx.

Thank you.

Answer 1

If you want to take a look at my 1.x fork it's here, compiling: https://github.com/timschott/gpt-2

Answer 2

I had the same issue but got it resolved by creating a separate hparams.py file in the folder and populating it with the content from here: https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/utils/hparam.py

then in your model.py you can add the following code and swap out this:

import tensorflow as tf
from tensorflow.contrib.training import HParams

with this:

import tensorflow.compat.v1 as tf
tf.disable_v2_behavior()
from hparams import HParams

Then you will have to add "compat.v1" inbetween "tf" and the module (if that's what it's called)... For instance if it is "tf.Session" change it to "tf.compat.v1.Session" or if it is "tf.placeholder" change it to "tf.compat.v1.placeholder", etc.

I did this after trying to downgrade to tensorflow-gpu 1.13 and gpt-2 still wasn't working. Same was the case with running the environment in 3.6 version of Python.

P.S. This is my first answer, not sure if I formatted it correctly, but will learn as I go as well.

Answer 3

Automatically upgrading code rarely works out of the box. With that repo, you should use Tensorflow=1.15 maximum.

If you really want Tensorflow=2, you can look at this repo: https://github.com/akanyaani/gpt-2-tensorflow2.0

Note: that you won't get the pre-trained models (probably the most interesting part of gpt2 for the average user). Meaning no access to their 1554M or 778M models.

I know of no method to automatically upgrade pre-trained models from 1.15 to 2.3 or whatnot.