I have not tried it, but I think, after the cloning you should do:
$ cd <CLONE>
$ git config core.sparseCheckout true
$ echo dir/file >.git/info/sparse-checkout
$ git reset --hard
At the last command it will automatically fetch missing blob(s). Hopefully it will get only the needed ones.
As far as I understand it, the theory is that it fetches everything except blobs - the file contents. So, by checkout or any other operation, it should know exactly list of objects it is missing, and will request only those.
PS: the specifying file for a filter with --filter=sparse:path=<path>
, as suggested in comments, could work locally, but in real-life case when server is at another computer it failed for me because server treats the argument as server file path, where it obviously did not exist.
PS2: if could be that you don't need depth. At checkout it is going to need only current blobs, not older ones. This did work for me with some early version of the feature. Though you still would have to fetch older trees. Actually, it also seems to fetch some blobs as well, but I did not play with it long enough.