It's look like the rowsPerPage value isn't pages that are "viewed" - but rather, "virtual" pages. That is, the grid only renders portions of itself at a time (in order to improve performance for very large data sets) - and the rowsPerPage value is used to determine how many rows to render at a time.
If you scroll to a position on the grid that it outside the rendered pages, it will render it on demand.
If you are displaying more than 5 rows (due to the height that you have set on your grid) - then setting rowsPerPage to 5 will just cause the viewable portion to be rendered in batches of 5 rows at a time. i.e. there will be a query called to your datastore with start=0 and count=5, and another query sent with start=5 and count=5, etc - until all the visible rows are rendered.
However you can see on this page - it is example of grid with paging. May be it helps you.