Why are there different scores when chess.engine analyses a position multiple times in the same script?

Question

beginner programmer here, trying to make an app that detects blunders and wanting to learn more about the chess.engine library.

I assume that using the analyse function is a discrete, self-contained process that doesn't rely on cache or memory of previous calls from the engine or anything like that.

If this is the case, why do I get multiple different evaluations when calling analyse multiple times in a script:

import chess
import chess.engine
import os

# Loads board and engine
board = chess.Board("3r3k/pp5p/n1p2rp1/2P2p1n/1P2p3/PBNqP2P/5PP1/R2RB1K1 w - - 0 26")
engine = chess.engine.SimpleEngine.popen_uci(os.getcwd()+'/static/'+'stockfish')

#Sets engine depth constant
engine_depth = 10

# I'm assuming this is how to find the best move - arbitrary as I could have chosen any move in board.legal_moves
best_move = engine.play(board, chess.engine.Limit(depth=engine_depth)).move

# I'm assuming this is how to evaluate the score after a specific move from this position
info = engine.analyse(board, chess.engine.Limit(depth=engine_depth), root_moves=[best_move])
print(info["score"])

# Repeating the analysis call and printing the score 3 more times
info = engine.analyse(board, chess.engine.Limit(depth=engine_depth), root_moves=[best_move])
print(info["score"])

info = engine.analyse(board, chess.engine.Limit(depth=engine_depth), root_moves=[best_move])
print(info["score"])

info = engine.analyse(board, chess.engine.Limit(depth=engine_depth), root_moves=[best_move])
print(info["score"])

engine.quit()

Output:

PovScore(Cp(-21), WHITE)
PovScore(Cp(+2), WHITE)
PovScore(Cp(+19), WHITE)
PovScore(Cp(+63), WHITE)

It occurs for various engine depths above a certain level, but not when depth is low, e.g. up to 3.
It occurs with time as the limit instead of depth.
It occurs with multiple different starting positions.
It occurs with multiple different moves.
It occurs even if I have engine.quit() and engine = chess.engine.SimpleEngine.popen_uci(os.getcwd()+'/static/'+'stockfish') in between each call.

The analysis function can't have a random element because when I run the whole script again I get exactly the same scores. It's just that when it's called for the same position more than once in the same script, it gives a different score each time, as if it's using some kind of cache or it's looking even deeper each time.

So where am I going wrong in my understanding of how this works?

EDIT:

If I remove the root_moves argument (just to simplify things) and then replace:

print(info["score"])

with:

for k, v in info.items():
    print(k, v)

I get the following outputs:

string NNUE evaluation using nn-82215d0fd0df.nnue enabled
depth 10
seldepth 13
multipv 1
score PovScore(Cp(+25), WHITE)
nodes 4396
nps 439600
tbhits 0
time 0.01
pv [Move.from_uci('d1d3'), Move.from_uci('e4d3'), Move.from_uci('a1d1'), Move.from_uci('a6c7'), Move.from_uci('c3a4'), Move.from_uci('f6f8'), Move.from_uci('a4b2'), Move.from_uci('h5f6'), Move.from_uci('b2d3'), Move.from_uci('d8d7')]

string NNUE evaluation using nn-82215d0fd0df.nnue enabled
depth 10
seldepth 15
multipv 1
score PovScore(Cp(+55), WHITE)
nodes 4072
nps 290857
tbhits 0
time 0.014
pv [Move.from_uci('d1d3'), Move.from_uci('e4d3'), Move.from_uci('a1d1'), Move.from_uci('a6c7'), Move.from_uci('c3a4'), Move.from_uci('c7b5'), Move.from_uci('a4b2'), Move.from_uci('b5a3'), Move.from_uci('b2d3'), Move.from_uci('a3b5')]

string NNUE evaluation using nn-82215d0fd0df.nnue enabled
depth 10
seldepth 16
multipv 1
score PovScore(Cp(+26), WHITE)
nodes 4514
nps 282125
tbhits 0
time 0.016
pv [Move.from_uci('d1d3'), Move.from_uci('e4d3'), Move.from_uci('a1d1'), Move.from_uci('a6c7'), Move.from_uci('c3a4'), Move.from_uci('c7b5'), Move.from_uci('a4b2'), Move.from_uci('h8g7'), Move.from_uci('b2d3'), Move.from_uci('f6f8')]

string NNUE evaluation using nn-82215d0fd0df.nnue enabled
depth 10
seldepth 12
multipv 1
score PovScore(Cp(+164), WHITE)
nodes 2018
nps 252250
tbhits 0
time 0.008
pv [Move.from_uci('d1d3'), Move.from_uci('d8d3'), Move.from_uci('b3c4'), Move.from_uci('d3d8'), Move.from_uci('c3e2'), Move.from_uci('h8g7'), Move.from_uci('e1c3')]

So it looks like I'm getting different 'seldepths' each time. What is seldepth exactly? I can't find enough info about it in the documentation.

I also get different scores when I remove the 'root_moves' argument: PovScore(Cp(+25), WHITE) PovScore(Cp(+55), WHITE) PovScore(Cp(+26), WHITE) PovScore(Cp(+164), WHITE) — Vigoxin, Feb 03 '21 at 13:03
And you are sure that the board and depth is the same each time? Try to print some more info than score to see if you find what is going on. The PV-line would be interesting for example. — eligolf, Feb 03 '21 at 13:09
Ah good point. I've just edited the post to show what comes up. So first thing I notice is that I'm getting different 'seldepths'. I'm not sure what this is and I can't find enough info about it in the documentation. Also, the pv lines are different. Thank you very much for helping btw. — Vigoxin, Feb 09 '21 at 16:06
I would assume seldepth is the depth it goes for including quiescence search and other extensions. I also see you have NNUE enabled, I am not entirely sure how neural networks works actually. Can you try without them? Maybe it is that neural networks give different results? — eligolf, Feb 09 '21 at 18:48
I had no idea this means neural networks are turned on. I can't find anything about this in the documentation, would you know how to? — Vigoxin, Feb 17 '21 at 15:28

score 2 · Accepted Answer · answered Feb 17 '21 at 22:52

Stockfish is deterministic for single-threaded analysis with node and depth limits, state and options being equal.

The engine state is mostly the hashtable (and potentially loaded Syzygy tablebases). So your second analysis will take advantage of the hashtable of the first run. The third run will reuse hashtable results from the second run and so on.

Usually reusing hashtable results is desirable and improves strength, but there is a way to reset the engine state, clearing the hashtable.

In the UCI protocol this is done by sending the hint:

ucinewgame

In python-chess, this is automatically managed by using the game key.

engine.analyse(..., game="key1")
engine.analyse(..., game="key1")
engine.analyse(..., game="key2")  # will clear previous state
engine.play(..., game="key2")
engine.analyse(..., game="key1")  # will clear previous state

The key can be anything, in particular object() is not equal to any other object, so that would always clear the hashtable.

import chess
import chess.engine

board = chess.Board("3r3k/pp5p/n1p2rp1/2P2p1n/1P2p3/PBNqP2P/5PP1/R2RB1K1 w - - 0 26")
engine = chess.engine.SimpleEngine.popen_uci("stockfish")
engine_depth = 10

best_move = engine.play(board, chess.engine.Limit(depth=engine_depth), game=object()).move

info = engine.analyse(board, chess.engine.Limit(depth=engine_depth), game=object(), root_moves=[best_move])
print(info["score"])

info = engine.analyse(board, chess.engine.Limit(depth=engine_depth), game=object(), root_moves=[best_move])
print(info["score"])

info = engine.analyse(board, chess.engine.Limit(depth=engine_depth), game=object(), root_moves=[best_move])
print(info["score"])

info = engine.analyse(board, chess.engine.Limit(depth=engine_depth), game=object(), root_moves=[best_move])
print(info["score"])

engine.quit()

PovScore(Cp(+35), WHITE)
PovScore(Cp(+35), WHITE)
PovScore(Cp(+35), WHITE)
PovScore(Cp(+35), WHITE)

I'm trying to make an app that can tell which move is best, good, inaccurate, or blunder. Can you tell me how can I identify this by looking at the Cp value? — Rishabh Dhiman, Jun 13 '21 at 16:13

Why are there different scores when chess.engine analyses a position multiple times in the same script?

1 Answers1