This is not trivial task, I have something similar, but not exactly this case before. First of all, I would I would create custom Layout component, e.g. by extending CssLayout in similar fashion as has been discussed here ( How to make UI receive scroll events ) in addition to reporting scroll events I would report the position of the layout on the viewport (see http://www.gwtproject.org/javadoc/latest/com/google/gwt/dom/client/Element.html ). Yes GWT and client side development is required.
I would use this layout as wrapper for the Grid, i.e. Grid would be in the layout. You could extend the Grid component as well. But I think doing the layout wrapper gives you nice tool that you can use with other components as well for which you need to determine, whether they are actually visible or not.
This way I can then calculate which portion of the layout is in the viewport. As you see, there are number of cases here, e.g. only bottom of the Grid or top of the Grid is visible. Then I need to know row-height, header height, etc. That enables me to calculate how many rows there are visible. I hope you get the idea. The outcome for the generic case will be rather lengthy piece of code when all possibilities are enumerated. This calculation you can do on server side.