I would think the use cases for this must be pretty limited; you're looking at getting a snapshot of some files, without historical context or the ability to write back changes. Well, ok...
When you use archive
with --remote
you pretty much have to pull from a ref. If you can push a tag to the remote, you can tag the version you want and then you should be able to pull from that tag. If you can't do that - and if the version you want doesn't happen to have a tag or be the current head of a branch - then you're probably out of luck.
The docs are pretty wishy-washy about even that; when it comes down to it, you're at the whim of the server whether it's going to help you out here. The git model doesn't really give much support to remote access at a sub-repository level.
Some partial solutions you might play with, depending on how exactly this repo's bloat is organized:
You could play with shallow and/or single-branch cloning. You'd still have to pull the full TREE for at least the version you want, but you'd be able to minimize (or maybe eliminate) pulling of history and unrelated versions of files.
Even though you can't clean up the original repo, if you're going to frequently read versions of this subtree, it might be worth using filter-branch
with subdirectory-filter
to create a repo with the history for just the subtree; then remove the subtree from the "next" commit in the original repo, replacing it with a submodule reference to the new repo. (But if it's just a one-time pull, this clearly wouldn't be worth the trouble.)