I'm creating a rhythm game in C#/Unity, and have come across a problem where on most PC's the game runs completely fine but on others the note time (which is stored as the elapsed time from the song start) gradually drifts from the audio time which ends up desyncing the gameplay from the audio. I've tried all the different time keeping values Unity has to offer (Time.time, Time.realTimeSinceStartup, Time.deltaTime, etc) and they all end up causing desyncs.
This leads me to believe that I need to smooth the song time towards the audio time, but I have no idea what's the best way to do that.
I'm not exactly sure how expensive getting the audio time is, so I assume that running it once a second and then smoothing the time values together over the next second and then repeating the process may be a decent way of doing it without causing bad amounts of overhead. Would this be a good way of doing it?