A few things to understand here.
Office 365 offers a few different ways you can authenticate through it.
As standard it will be Cloud Only where accounts only exist in Office 365.
Usernames and Passwords are created by the admins and maintained separately from any other identity the user might have.
Azure AD Connect allows you to sync the identities up so that the meta data (username, displayname, email) and so will match Office 365. Azure AD Connect also has the option to sync the passwords where a hashed version of your users password from AD will be synced to the Azure AD that Office 365 runs on.
This will give the users a "Same Sign-On" Experience, it it will not automatically log the user into Office 365 when they are already logged onto the AD on their computer, but the credentials are the same, so it makes it easier for the user.
Then we have the Single Sign-On options, here the concept is that the user logs in only once, when they have logged onto the machine their, the token the machine already have with the AD will be used and automatically log the user on to services like Office 365 without the need for the user to type credentials.
With ADFS, you can get Single Sign-On.
For ADFS to work with Office 365 it is absolutely required that you run Azure AD Connect (DirSync) and that your on-premises credentials matches those synced to the cloud.
You will tell Office 365 that the domain is federated, this basically says that when a user attempts to log into Office 365, it is not Azure AD it should ask, but it will redirect the user to the ADFS server and authenticate there, the ADFS server will then tell Office 365 that the user is authenticated and allow the user to log in.
If you remove ADFS but keep Azure AD Connect you can have a Same Sign-On experience (though it is possible to do Single Sign-On with the newer version of Azure AD Connect)
To remove ADFS from this setup you need to Convert your Federated domains in Office 365 to Managed Domains
I hope this helps with understanding the setup and answers your questions.
Edit
Just realised I missed part of your question.
You can easily check if Office 365 tries to federate a domain through ADFS. If you go to https://portal.office.com and type something@domain.tld and hits enter it will check to see if the domain, in this case domain.tld is federated, if it is it will redirect your login to the ADFS login page, if not it will keep it on Office 365.
If it is on Office 365 then you can go ahead and remove the ADFS servers (as least for from Office 365's point of view, if you authenticate with other providers you will need to check with them too).