Material-ui 5, unable to customize theme through ThemeProvider

Question

I have recently started using Material ui (v5) and I am facing some issues with theme customization. I want to change the font-family for all the components through the ThemeProvider and creating a new theme. I am able to see the theme in the react inspector with the correct values but those values don't get applied to the components (specifically fontFamily but nothing works really).

This is what my ThemeProvider looks like:

import React from "react";
import {
  createTheme,
  ThemeProvider as MUIThemeProvider,
} from "@mui/material/styles";

const theme = createTheme({
  typography: {
    fontFamily: "serif",
  },
});

type ThemeProviderProps = {
  children: React.ReactNode;
};

const ThemeProvider = ({ children }: ThemeProviderProps) => (
  <MUIThemeProvider theme={theme}>{children}</MUIThemeProvider>
);

export default ThemeProvider;

The ThemeProvider then wraps every story (using storybook):

  <div
    style={{
      height: "100%",
      width: "100%",
    }}
  >
    <ThemeProvider>
      <Story {...context} />
    </ThemeProvider>
  </div>
);

export const decorators = [withThemeProvider];```

I am able to see the theme I created if I use the useTheme hook or in the ThemeProvider in the components inspector (please ignore the different fontFamily in the screenshot below):

When I look at the actual component I see that it is still getting the default styles:

I am sure I am doing something stupid somewhere?