XAudio2 playing delay

Question

I'm trying to play raw-pcm data with xaudio and i have huge delays in playing (>=5ms). I'm doing this with next code:

bool Play(uint8_t *data, size_t size)
{
  _xaudio_buffer.AudioBytes = size;
  _xaudio_buffer.pAudioData = data;
  Time t1;

  if (_g_source->SubmitSourceBuffer(&_xaudio_buffer) != S_OK)
    return false;

  if(WaitForSingleObjectEx(_voice_callback.hBufferEndEvent,INFINITE,true) != WAIT_OBJECT_0)
    return false;

  Time t2;
  printf("%d\n",t2-t1);
}

Time class is just a wrapper under GetTickCount, the resulting t2-t1 will return difference in milliseconds. I've checked that my Time class doesn't produce any additional delay.
It is not so hard to calculate play time in milliseconds: play_time = size*1000 / (channels*(bits_per_sample/8) * frequency)

So for data with size 4608 bytes, 48 khz, 2 channels, 16 bit per sample it should take nearly 24ms for playing such chunk. Instead, code that i showed above needs about >= 31 ms for playing such chunk. What is producing such delay? How to deal with it, if i'm writing a video-player and i obtaining data from real-time stream (i already have synchronization function, but 5ms delay for such small sample produces not ideal sound)?

Also, I've tested this code on 2 computers with Windows 7 with different hardware. Both producing same delay.