Hugging Face Inference API returning short generated text with GPT-2 model

Question

I'm using the Hugging Face API with a GPT-2 model to generate text based on a prompt. However, I'm encountering an issue where the generated text is consistently too short, even though I'm specifying a maximum number of new tokens and using other parameters to try to generate longer text.

function GPT2()
{
const prompt = "Please give me a 100 words tell me about yourself";  

const API_TOKEN = "xxxxxxxxxxxx";
const MODEL_NAME = "gpt2";

const url = `https://api-inference.huggingface.co/models/${MODEL_NAME}`;
  const options = 
  {
    method: "post",
    headers: 
    {
      Authorization: `Bearer ${API_TOKEN}`,
      "Content-Type": "application/json",
    },
    payload: JSON.stringify
    ({
      inputs: prompt,
      options: 
      {
        contentType: "application/json",
        max_tokens: 200,
        max_length: 100, 
        num_return_sequences: 3,
        use_cache: false,
        return_full_text: true
      }
    })
  };
  const response = UrlFetchApp.fetch(url, options);
  const res = JSON.parse(response.getContentText());
  Logger.log(response);

  return res[0].generated_text.trim();
}

For example, when I input the prompt "Please give me a 100 words tell me about yourself.", the generated text returned is only a few words long, such as "I'm a writer, and I". I was expecting the generated text to be much longer.

Can anyone offer any suggestions for how I can generate longer text using the Hugging Face API with a GPT-2 model? Is there a problem with my code or input parameters that is causing the generated text to be too short?

I have also checked the API response to ensure that it is valid and contains the expected input and output values.

I've already tried setting max_tokens to 800 and are still encountering the issue of short generated text. — ma.ca, Jul 17 '23 at 12:17

score 0 · Answer 1 · answered Jul 17 '23 at 15:05

0

I believe they're slowly phasing out max_length, so I would just stick with max_tokens. There is also a min_tokens parameter available, so definitely try using that.

GPT-2 isn't very good at keeping track of how long it's responses are, so asking for a response of at least 100 characters might not work out very well. I would also try to use a larger model in the family, you will likely get outputs closer to what you're going for. Remember, the GPT-2 models are not fine-tuned for instruction, so it won't respond to "thoroughly explain how you reached your conclusion" like GPT-3 or 4 would.

answered Jul 17 '23 at 15:05

sanminchui

59
1
9

I used the "bigscience/bloom" model on Hugging Face, but I encountered same issue where the generated text was consistently too short. However, when I used the API for OpenAI GPT-3, I didn't encounter any issues. So I assume the problem lies with my coding or with Hugging Face. – ma.ca Jul 18 '23 at 04:04
Did you try using the `min_tokens` parameter? If it was significantly less than that minimum number of tokens, I'd report that as a bug. – sanminchui Jul 18 '23 at 13:41
Yes, I have tried it, but it didn't have any impact on the output result. The generated text remains consistently too short. – ma.ca Jul 19 '23 at 01:54

Hugging Face Inference API returning short generated text with GPT-2 model

1 Answers1