When should I commit to project?
As the project repository should only contain the subrepositories (and track their revisions), you should (and only have to) commit to project when you have checked out a different version of either the core or the common repository and you permanently want to use it for your project.
Do I create feature branches on it, or only in the subrepositories?
You have to create a "feature" repo for both the project repo and the repo where you want to implement the feature in.
If you only fork the project repo, it will still track the original core/common.
On the other hand you also need to fork the project repo to have a complete working environment containing all needed subrepos.
Another approach is that every developer has his own permanent fork of the project repo and tracks the feature repos in it that he is currently working on.
A pull request would then just be created for the changed subrepository.
What happens when a new feature requires backwards-incompatible changes in both core and common
This would imply that you would do/commit/push these changes in both repositories and update the tracked version in the project shell repo (commit the new shell repo state and push it).
This results in a working version of your project.
What will not work, of course, is using to incompatible versions of the core or common repository.
Sounds like one should strive for backwards compability for the common repo anyway as it might be used in other projects, too.