I got this solution:
Detecting whether the dialog has been opened or not, if existing, then execute SendKeys. If not, waiting for a moment, maybe 3 sec, then detecting again.
How to:
Detecting whether the dialog has been opened or not
FindWindow FindWindowEx
Use FindWindow to find the dialog
Waiting for a period then execute the next step
Using timers in vb
Use timer to count the waiting time
Reply:
I don't have 50 reputations, so not allowed to post a comment.
To use the FindWindow, you have to use the correct parameter. You could use Spy++ (Visual Studio, Tools/Spy++) to find the parameter of this dialog window.

You could use code below:
Declare:
<DllImport("user32.dll", CharSet:=CharSet.Auto, EntryPoint:="FindWindow")>
Private Shared Function FindWindow(ByVal lpClassName As String, ByVal lpWindowName As String) As IntPtr
End Function
Function catchDialog:
Private Sub catchDialog()
Dim hwnd As IntPtr = FindWindow("Class", "Caption")
If hwnd <> IntPtr.Zero Then
yourSendKeyProcess(hwnd)
Else
' dialog not showing, waiting for 3 Sec. then detecting again.
' Use Thread.Sleep() is simple then timer.
Thread.Sleep(3000)
catchDialog()
End if
End Sub
Function yourSendKeyProcess:
Private Sub yourSendKeyProcess(ByVal window as IntPtr)
SetForegroundWindow(window)
SendKeys.SendWait("C:\TEST.mxd")
SendKeys.SendWait("{TAB 3}")
SendKeys.SendWait("{ENTER}")
End Sub
Use Spy++ to find the value of "Caption", and "Class" of the dialog.

The example above, the value of Caption of the window is "Add to Archive"
and the value of Class is "#32770(Dialog)", and the code would be:
Dim hWnd As IntPtr = FindWindow("#32770", "Add to Archive")
But even you could catch the dialog window, I am not sure that your code SendKey could work, so why you use those codes?
SendKeys.SendWait("C:\TEST.mxd")
SendKeys.SendWait("{TAB 3}")
SendKeys.SendWait("{ENTER}")
You want to input a string ("C:\TEST.mxd") in the textbox on the dialog, then press a button on the dialog?
Edit:
Set the form1 always on top but not influence operation on other window:
Add this sub to make the form1 on top in the beginning:
Private Sub Form1_Load(sender As Object, e As EventArgs) Handles MyBase.Load
Me.TopMost = True
End Sub
and Function catchDialog:
Private Sub catchDialog()
Dim hwnd As IntPtr = FindWindow("Class", "Caption")
If hwnd <> IntPtr.Zero Then
me.TopMost = false
yourSendKeyProcess(hwnd)
Else
' dialog not showing, waiting for 3 Sec. then detecting again.
' Use Thread.Sleep() is simple then timer.
Thread.Sleep(3000)
catchDialog()
End if
End Sub
and Function yourSendKeyProcess:
Private Sub yourSendKeyProcess(ByVal window as IntPtr)
SetForegroundWindow(window)
SendKeys.SendWait("C:\TEST.mxd")
SendKeys.SendWait("{TAB 3}")
SendKeys.SendWait("{ENTER}")
me.TopMost = true
End Sub