0

This is my code for calling the gpt-4 model:

messages = [
    {"role": "system", "content": system_msg},
    {"role": "user", "content": req}
]

response = openai.ChatCompletion.create(
        engine = "******-gpt-4-32k",
        messages = messages,
        temperature=0,
        top_p=1,
        frequency_penalty=0,
        presence_penalty=0
    )

answer = response["choices"][0]["message"]["content"]

Keeping system_msg & req constant, with temperature=0, I get different answers. I got 3 different answers when I last ran this 10 times for instance. The answers are similar in concept, but differ in semantics.

Why is this happening?

Kavya Bhandari
  • 29
  • 1

2 Answers2

0

Found a solution here: https://community.openai.com/t/observing-discrepancy-in-completions-with-temperature-0/73380

TLDR; some discrepancies occur due to floating pt operations sometimes when 2 tokens have very close probabilities. Even a single token change affects the whole chain and leads to divergent generations.

Kavya Bhandari
  • 29
  • 1
0

This blogpost authored by Sherman Chann argues that:

Non-determinism in GPT-4 is caused by Sparse MoE [mixture of experts].

Franck Dernoncourt
  • 77,520
  • 72
  • 342
  • 501