Reducing renders in child components of a context

Question

I'm trying to delve deeper than the basics into React and am rebuilding a Tree library I had written in plain JS years back. I need to expose an API to users so they can programmatically add/remove nodes, select nodes, etc.

From what I've learned, a ref and context is a good approach. I've built a basic demo following the examples (without ref, for now) but I'm seeing every single tree node re-render when a selection is made, even though no props have changed for all but one.

I've tried a few things like memoizing my tree node component, etc but I feel like I'm failing to understand what's causing the re-render.

I'm using the react dev tools to highlight renders.

Here's a codesandbox demo.

My basic tree node component. I essentially map this for every node I need to show. On click, this calls select() from my context API. The rerender doesn't happen if that select() call is disabled.

const TreeNodeComponent = ({ id, text, children }) => {
  console.log(`rendering ${id}`);
  const { select } = useTreeContext();

  const onClick = useCallback(
    (event) => {
      event.preventDefault();
      event.stopPropagation();

      select([id]);
    },
    [select, id]
  );

  return (
    <div className="tree-node" onClick={onClick}>
      {text}
      {children ? <TreeNodesComponent nodes={children} /> : ""}
    </div>
  );
};

The important part of my context is the provider:

const TreeContextProvider = ({ children, nodes = [], selectedNodes }) => {
  const [allNodes] = useState(nodes);
  const [allSelectedNodes, setSelectedNodes] = useState(selectedNodes || []);

  const api = useMemo(
    () => ({
      selected: () => allSelectedNodes,

      select: (nodes) => {
        setSelectedNodes(Array.from(new Set(allSelectedNodes.concat(nodes))));
      }
    }),
    [allSelectedNodes]
  );

  const value = useMemo(
    () => ({
      nodes: allNodes,
      selectedNodes: allSelectedNodes,
      ...api
    }),
    [allNodes, allSelectedNodes, api]
  );

  return <TreeContext.Provider value={value}>{children}</TreeContext.Provider>;
};

Again, when the call to setSelectedNodes is disabled, the rerenders don't happen. So the entire state update is triggering the render, yet individual props to all but one tree component do not change.

Is there something I can to improve this? Imagine I have 1000 nodes, I can't rerender all of them just to mark one as selected etc.