react-window children as itemData results always to a rerender

Question

I am using react-window to render virtualized lists. I created a generic component, so it is easy to make every list virtualized.

The generic component looks like this:

const itemData = useMemo(() => React.Children.toArray(children), [children]);
const contextValue = useMemo(() => ({ setRowHeight }), [setRowHeight]);

return (
    <TableBody className={classNames(classes.root, className)} ref={rootRef} data-testid='virtualized-table-body' {...other}>
        <InfiniteLoader ref={loaderRef} itemCount={totalCount} isItemLoaded={isItemLoaded} loadMoreItems={loadMoreItems}>
            {({ onItemsRendered, ref }) => (
                <VirtualizedTableContext.Provider value={contextValue}>
                    <VariableSizeList
                        itemCount={loadedCount}
                        height={height}
                        width='100%'
                        itemSize={getRowHeight}
                        ref={ref}
                        itemData={itemData}
                        estimatedItemSize={configuration.defaultVirtualizedRowHeight}
                        overscanCount={overscanCount}
                        onItemsRendered={onItemsRendered}
                    >
                        {VirtualizedRow}
                    </VariableSizeList>
                </VirtualizedTableContext.Provider>
            )}
        </InfiniteLoader>
    </TableBody>
);

As you can see, I am passing children as itemData and the VirtualizedRow will do the following:

const VirtualizedRow = memo(function VirtualizedRow(props: VirtualizedRowProps) {
     const { index, style, data } = props;
     return (
         <div style={style}>
            <VirtualizedChildWrapper index={index}>{data[index]}</VirtualizedChildWrapper>
         </div>
     );
}, areEqual);

The problem I am currently have, is that one children changes (maybe a list item gets selected) then every other items will be rerendered as well. Since it is not possible to memorize children (see: https://gist.github.com/slikts/e224b924612d53c1b61f359cfb962c06), I really don't know how to solve that rerendering issue. Any ideas?

Here is a image of the the rerendering in the F12 Performance panel