You have not mentioned what type of dialog window it is. But you can try the IWindow API of RFT to find active top window and perform the click as specifed below.
The Following code for eg can handle an alert dialog box in html by calling
handleDialogButton("Message from Webpage", "ok");
Or,to click on canel button on Notepad's Font dialog (Format>Font) by calling
handleDialogButton("font","cancel");
-------Sample code----
/*
* Activates the top window with the given caption and clicks on the child control(window) with the specified text
* @param caption- Caption of the Dialog window
* @param btnToClick- Text of the button(any other control) to click
*/
void handleDialogButton(String caption,String btnToClick)
{
IWindow[] windows = getTopWindows();
for(IWindow window: windows)
{
if(window.getText().equalsIgnoreCase(caption))
{
window.activate();
//window.close(); //we can just close it also n break.
IWindow[] children = window.getChildren(); // OR go thru the children to get the child
for(IWindow child:children)
{
if(child.getText().equalsIgnoreCase(btnToClick))
{
child.click();
break;
}
}
}
}
unregisterAll();
}