Remove heading options from Gutenberg core block

Question

I want to remove the heading options for the Gutenberg core block using filters.

Here is what I tried:

const { createHigherOrderComponent } = wp.compose;

const withInspectorControls = createHigherOrderComponent( ( HeadingToolbar ) => {
    return ( level, setAttributes ) => {
        return (
            <HeadingToolbar
                minLevel={ 1 }
                maxLevel={ 1 }
                selectedLevel={ level }
                onChange={ ( newLevel ) =>
                    setAttributes( { level: newLevel } )
                }
            />
        );
    };
}, 'withInspectorControl' );

wp.hooks.addFilter( 'editor.BlockEdit', 'core/heading', withInspectorControls );

However, I am just getting a blank block.

Ideally, I want the HeadingToolbar to completely disappear from the options.