I do not know how to do this with iText, but I can tell you how it works internally in the PDF format.
Annotation objects can have a Popup attribute that customizes the popup window associated to the annotation. This Popup attribute is also an annotation object, so you can configure its possition on the page the same way you configured the parent anotation.
From the PDF Reference Document:
TABLE 8.15 Entries common to all annotation dictionaries
Key: Rect
Type: rectangle
Value: (Required) The annotation rectangle, defining the location of the annotation on the page in default user space units.
There are different types of annotations in PDF (Text, Link, FreeText, Line, Square, etc) but only markup annotations can have a popup window, and not all these types are considered as markup annotations. If you want to know which types are considered markup annotations take a look on TABLE 8.20 Annotation types of the PDF Reference document (Chapter 8 - Annotations, part 8.4.5 - Annotation types).
Looking at iText documentation, I saw there is a class called PdfAnnotation, this is probably the class you are using for creating your new annotation. This class contains a method called setPopup. Maybe you could give this method a try.