How to continue training serialized AllenNLP model using `allennlp train`?

Question

Currently training models using AllenNLP 1.2:

allennlp train -f --include-package custom-exp /usr/training_config/mock_model_config.jsonnet -s test-mock-out

The config is very standard:

"dataset_reader" : {

        "reader": "params"

    },
    "data_loader": {
        "batch_size": 3,
        "num_workers": 1,
    },
    "trainer": {
        "trainer_params": "various"
    },


    "vocabulary": {
        "type": "from_files",
        "directory": vocab_folder,
        "oov_token": "[UNK]",
        "padding_token": "[PAD]",
    },

    "model": {
        "various params": ...
    }

and serializing them to the test-mock-out directory (also have model.tar.gz).

Using the allennlp train command, is it possible to continue training? The documentation states Model.from_archive should be used, but it's unclear how the config should be adapted to use it.

http://docs.allennlp.org/v1.2.0/api/commands/train/

Could you elaborate on what you mean by "continue training"? — arjuns, Apr 28 '21 at 22:55
Hey Arjun, yes sorry for the vagueness. I mean train on the same or similar data (identical vocab, etc.). Will `--recover` parameter work out of the box for this use case? — information_interchange, Apr 29 '21 at 13:39
Yes, using `-r` or `--recover` with the same serialization directory should work! — arjuns, Apr 29 '21 at 17:30
Great thanks, I was able to get something working using `from_archive` parameter under `type` in `model` — information_interchange, Apr 30 '21 at 02:57
Hi I realize this is old, but I find that my server doesn't have enough space in the /var/tmp folder it tries to extract `my_model.tar.gz` to. Is there any way to tell it to extract the archived model to a custom location? — Russell Richie, Mar 21 '22 at 15:09
By tell 'it', I mean, is there some way we can modify the config jsonnet file to tell AllenNLP to extract the archived model to custom location? — Russell Richie, Mar 21 '22 at 15:35

information_interchange · Answer 1 · 2021-05-01T02:26:39.987

OK, so to continue the training, one solution is to load the model from_archive. Assuming you have the serialization directory, make a model.tar.gz archive of the folder. Then, you can make a new config that is identical, except for the model key which uses from_archive:

retrain_config.json:

{
### Existing params ###
 "data_loader": {
        "batch_size": 3,
        "num_workers": 1,
    },
    "trainer": {
        "trainer_params": "various"
    },
### Existing params ###
...
"model": {
       "type": "from_archive",
       "archive_file": "path/to/my_model.tar.gz"
    }

}

Then, use your original train command, pointing to this new config:

allennlp train -f --include-package custom-exp /usr/training_config/retrain_config.json -s test-mock-out

Note that your vocab and output dimensions/label space should remain consistent. Also, it seems like the global training epochs are not preserved.

Alternatively, if you just want to keep training on the exact same training data, and you just have the serialization directory, then you can avoid having to compress the directory, and just keep adding results to the same directory.

  allennlp train -f --include-package custom-exp /usr/training_config/mock_model_config.jsonnet -s test-mock-out --recover

How to continue training serialized AllenNLP model using `allennlp train`?

1 Answers1