3

Is there a way to train a Large Language Model (LLM) to store a specific context? For example, I had a long story I want to ask questions about, but I don't want to put the whole story in every prompt. How can I make the LLM "remember the story"?

Rok Benko
  • 14,265
  • 2
  • 24
  • 49
nerdlyfe
  • 487
  • 7
  • 21
  • Please take the [tour] to learn how Stack Overflow works and read [ask] on how to improve the quality of your question. Please see: [Why is “Is it possible to…” a poorly worded question?](https://softwareengineering.meta.stackexchange.com/q/7273). Please show your attempts you have tried and the problems/error messages you get from your attempts. – Progman Feb 19 '23 at 15:40
  • there is a lot of talk on how LLMS might be doing this, i'm interested in training it. http://ai.stanford.edu/blog/understanding-incontext/#f1 – nerdlyfe Feb 19 '23 at 16:58

1 Answers1

6

Taking into account that GPT-3 models have no parameter that enables memorization of past conversations, it seems the only way at the moment to "memorize" past conversations is to include past conversations in the prompt.

If we take a look at the following example:

You are a friendly support person. The customer will ask you questions, and you will provide polite responses

Q: My phone won't start. What do I do? <-- This is a past question
A: Try plugging your phone into the charger for an hour and then turn it on. The most common cause for a phone not starting is that the battery is dead.

Q: I've tried that. What else can I try? <-- This is a past question
A: Hold the button in for 15 seconds. It may need a reset.

Q: I did that. It worked, but the screen is blank. <-- This is a current question
A:

Rule to follow:

  • Include prompt-completion pairs in the prompt, with the oldest conversations at the top.

The problem you'll face:

  • You will hit a token limit at some point (if you chat long enough). Each GPT-3 model has a maximum number of tokens you can pass to it. In the case of text-davinci-003, it is 4096 tokens. When you hit this limit, the OpenAI API will throw an error. When this happens, you need to reduce the number of past prompt-completion pairs (e.g., include only the most recent 4 past prompt-completion pairs).

Pros:

  • By including past prompt-completion pairs in the prompt, we are able to give GPT-3 models the context of the conversation.

Cons:

  • What if a user asks a question that relates to a conversation that occurred more than 4 prompt-completion pairs ago?
  • Including past prompt-completion pairs in the prompt will cost (a lot of) money!
Rok Benko
  • 14,265
  • 2
  • 24
  • 49
  • 1
    yes, i'm deep in prompt engineering, so i appreciate the information here. most interested in storing context, rather than including it in the prompt. but good advice – nerdlyfe Feb 21 '23 at 11:52
  • 1
    I'm wondering if [fine tuning](https://platform.openai.com/docs/guides/fine-tuning) might be an option. – Rubén Mar 02 '23 at 05:29
  • @Rubén no. since fine tuning is a more generic way to train your model, the model still does not know what the user will ask and what it asked (in this example) 4 questions before. – Suisse May 04 '23 at 21:00