We've struggled with this issue on and off, because the solution is not obvious, nor even sensible from a least-privilege perspective. In short:
Your application requires FullControl rights in order to access RoleAssignments.
More details
If using an application created by Azure AD, you can demonstrate this by granting and consenting to SharePoint permissions such as:
- AllSites.Read (delegated)
- MyFiles.Read (delegated)
- Sites.Selected (application)
- User.Read.All (delegated)
- User.Read.All (application)
If using SharePoint-created application, use appregnew.aspx and appinv.aspx to create the app and grant it Read rights at some reasonable scope; try http://sharepoint/content/tenant.
Then get a bearer token using a client_credentials workflow (use Postman, for example).
Try a query like:
https://your-tenant.sharepoint.com/_api/web/GetFolderByServerRelativeUrl('/Shared Documents')/Files?$select=Length,TimeLastModified,ListItemAllFields,ServerRelativeUrl
Make sure the Authorization header value is Bearer your-access-token
It will work.
Now try the same query, but with getting role assignments:
https://your-tenant.sharepoint.com/_api/web/GetFolderByServerRelativeUrl('/Shared Documents')/Files?$select=Length,TimeLastModified,ListItemAllFields,ServerRelativeUrl&$expand=ListItemAllFields/RoleAssignments/RoleDefinitionBindings/Name
It will fail as you described, with status 403 and System.UnauthorizedAccessException.
Now grant and consent to the SharePoint application permission Sites.FullControl.All, or use appinv.aspx to add FullControl rights. Get a new bearer token. (The old one encodes the old rights granted to the app in the role field of the payload.). You'll need to wait a few minutes until the permissions apparently propagate from AD to SharePoint, if you're using an Azure AD application.
Try the last query again, and it will work.
IMHO, requiring FullControl in order to resolve something like a role assignment, which is needed to capture the permissions required to access content in a SharePoint library, is unjustified. I could understand, sort of, if tenant-scope Read permission were required. However, granting AllSites.Read or tenant-scope (http://sharepoint/content/tenant in appinv.aspx XML permissions) doesn't seem to enable roleassignment lookup.