0

I am trying to generate new prediction for the model, but I found it is not that intuitive to use fairseq. I found fairseq-interactive could help to generate with a good settings of batch_size, however, it seems that it will use 1 GPU at a time, I wonder if it is possible to use multiple GPU? Hope someone can kindly help!

Many thanks :)

xihajun
  • 37
  • 4

1 Answers1

1

You cannot do this natively within fairseq. The best way to do this is to shard your data and run fairseq-interactive on each shard in the background. Be sure to set CUDA_VISIBLE_DEVICES for each shard so you put each shard's generation on a different GPU. This advice also applies to fairseq-generate (which will be significantly faster for large inference jobs).

erip
  • 16,374
  • 11
  • 66
  • 121