I am not sure, if you have already fixed this issue but I found here a solution to prevent sharp corner when its expanded.
Her is the link: https://github.com/material-components/material-components-android/issues/1278#issuecomment-844979762
Avoid using shape shapeAppearanceOverlay and add into your layout which is used for bottomSheetDialogFragment, a drawable like below ->
<?xml version="1.0" encoding="utf-8"?>
<shape xmlns:android="http://schemas.android.com/apk/res/android">
<corners
android:topLeftRadius="24dp"
android:topRightRadius="24dp" />
<solid android:color="@color/light_white_dark_grey_02" />
</shape>
and for the style use this:
<style name="BottomSheetDialog" parent="@style/ThemeOverlay.MaterialComponents.BottomSheetDialog">
<item name="bottomSheetStyle">@style/BottomSheet</item>
<item name="android:colorBackground">@color/white</item>
</style>
<style name="BottomSheet" parent="Widget.MaterialComponents.BottomSheet.Modal">
<item name="backgroundTint">@android:color/transparent</item>
</style>
It worked for me but I recommend have a look into that link that I shared above ;)