For this issue , with XML file transformation, the web.xxx.config file is required. The transform will be applied when publish web app to web deploy package or publish to server directly per to the configuration.
As stated in the documentation: You need to create a Web Application package with the necessary configuration and transform files.
So if you want to removing all config transforms, as a workaround , you could add a Delete-files task in the end to remove all configuration files. For example:
Source Folder:
$(wwwRoot)\
Contents:
**\*.staging.config
Here is a case with similar issue , you can refer to it for details.