Is it possible to have Azure AD authentication for all users in an
office?
According to your description, you want to move as much of your infrastructure to the Azure, I think we can use Azure AD + on-prem AD + Azure AD domain service to manager users, in this way, we should keep your on-prem AD still on-prem, then users can use they original account to access local file server and VMs on Azure.
we can use Azure AD connect to connect Azure AD and your on-prem AD, AD connect will sync your on-prem user accounts to Azure AD, then Azure AD will sync those accounts to AAD DS. In this way, your local file server can be access via those accounts, and your VMs on Azure will join Azure AD DS, users can use those accounts to access the VMs on Azure.
Another way, we can setup a VPN between on-prem and Azure, move your on-prem AD to Azure (Azure VM), then join other Azure VMs to this AD(online). Then it works like on-prem. We can via VPN to access your local and online resource.