Remember that sometimes focus is placed on elements without using a keyboard, such as via script or third-party tools (like screen readers prior to page interaction). Also note that even screen reader users do not necessarily use the Tab key (for example, I can navigate the headings on a page by pressing H in most screen readers, and on mobile I can navigate a page elements without using a keyboard at all. Also note that some users with cognitive impairments benefit from focus styles even when using a mouse.
As such, if I were reviewing a site that disabled focus styles only until someone pressed the Tab key, I would still fail it for 2.4.7
Focus Visible. It might be helpful to read the Understanding SC 2.4.7 document for a little more context.
To answer your question, yes, removing it does post potential accessibility problems.
All that being said, I would instead challenge the designer to do what designers tend to do best — work within constraints. Come up with focus styles that are not distracting, with styles that are perceivable, operable, understandable, and robust (so, for example, it meets contrast minimums). It has been done before, so there is no reason it cannot be done again.