To create custom role in Azure AD, you need to have either Azure AD Premium P1 or P2 license along with Global Admin
or Privileged Admin
roles.
I tried to reproduce the same in my environment and got below results:
I have Azure AD Premium P2
license for my Azure AD tenant like below:

To create custom role in Azure AD, you need to follow below steps:
Go to Azure Portal -> Azure Active Directory -> Roles and administrators -> All roles -> New custom role

In Basics tab, enter custom role name and select Start from scratch option -> Next:

In Permissions tab, you can select the permissions based on your requirement in the list like below:

After selecting all required permissions, you can click on Create in Review + Create
tab like below:

After that, custom role created successfully in Azure AD like below:

You can assign that custom role to Azure AD users like below:
Go to Azure Active Directory -> Roles and administrators -> All roles -> Click on your custom role -> Add assignments

I assigned that custom role to one Azure AD user like below:

You can select the type based on your need and assign role accordingly like below:

After few minutes, it assigned to the user successfully like below:

Note that, you cannot find New custom role option if your tenant doesn't have required license.
I have another tenant with Azure AD Free
license like below:

When I tried to create custom role, New custom role option is greyed out
like below:

So, make sure to have required licenses and roles before creating Azure AD custom roles.