I think I may have found a work around.
My problem was that I was trying to do this when using a report model as the data source. It seems like the report builder sorted by the left hand most field in the query but when using the model this always seems to be a model generated reference field rather than the actual data held in the field you want.
To have better control of the order in which the entries appeared i did the following.
I created a new field based on the field I wanted to use.
Then put this new field rather than the original field from the data model as the left most field in the query.