How to Answer Subjective/descriptive types of lQuestions using BERT Model?

Question

I am trying to implement BERT Model for Question Answering tasks, but Its a little different from the existing Q&A models, The Model will be given some text(3-4 pages) and will be asked questions based on the text, and the expected answer may be asked in short or descriptive subjective type

I tried to implement BERT, for this task.

The Problems I am facing: The input token limit for BERT is 512. How to get the answer in long form, which can describe any instance, process, event, etc.

Answer 1

Try longformer which can have input length 0f 4096 tokens, or even 16384 tokens with gradient checkpointing. See details in https://github.com/allenai/longformer. Or on huggingface model hub https://huggingface.co/docs/transformers/model_doc/longformer.