8

I'm using node.js (express) on Heroku, where the slug size is limited to 300MB.

In order to keep my slug small, I'd like to use git-lfs to track my express' public folder.

In that way all my assets (images, videos...) are uploaded to a lfs-store (say AWS S3) and git-lfs leaves a pointer file (with probably the S3 URL in it?).

I'd like express redirects to the remote S3 file when serving files from the public folder.

My problem is I don't kwon how to retrieve the URL from the pointer file's content...

app.use('/public/:pointerfile', function (req, res, next) {
  var file = req.params.pointerfile;
  fs.readFile('public/'+file, function (er, data) {
    if (er) return next(er);

    var url = retrieveUrl(data); // <-- HELP ME HERE with the retrieveUrl function

    res.redirect(url);
  });
});

Don't you think it will not be too expensive to make express read and parse potentially all the public/* files. Maybe I could cache the URL once parsed?

abernier
  • 27,030
  • 20
  • 83
  • 114

2 Answers2

10

Actually the pointer file doesn't contain any url information in it (as can be seen in the link you provided, or here) - it just keeps the oid(Object ID) for the blob which is just its sha256.

You can however achieve what you're looking for using the oid and the lfs api that allows you to download specific oids using the batch request.

You can tell what is the endpoint that's used to store your blobs from .git/config which can accept non-default lfsurl tags such as:

[remote "origin"]
   url = https://...
   fetch = +refs/heads/*:refs/remotes/origin/*
   lfsurl = "https://..."

or a separate

[lfs]
   url = "https://..."

If there's no lfsurl tag then you're using GitHub's endpoint (which may in turn redirect to S3):

Git remote: https://git-server.com/user/repo.git
Git LFS endpoint: https://git-server.com/user/repo.git/info/lfs

Git remote: git@git-server.com:user/repo.git
Git LFS endpoint: https://git-server.com/user/repo.git/info/lfs

But you should work against it and not S3 directly, as GitHub's redirect response will probably contain some authentication information as well.

Check the batch response doc to see the response structure - you will basically need to parse the relevant parts and make your own call to retrieve the blobs (which is what git lfs would've done in your stead during checkout).

A typical response (taken from the doc I referenced) would look something like:

{
   "_links": {
     "download": {
       "href": "https://storage-server.com/OID",
       "header": {
         "Authorization": "Basic ...",
       }
     }
  }
}

So you would GET https://storage-server.com/OID with whatever headers was returned from the batch response - the last step will be to rename the blob that was returned (it's name will typically be just the oid as git lfs uses checksum based storage) - the pointer file has the original resource's name so just rename the blob to that.

danf
  • 2,629
  • 21
  • 28
  • thx @fundeldman, could you develop a bit more about the request to make to grab the ressource URL with the batch API (without github and authorization)? many thanks – abernier May 18 '16 at 21:34
0

I've finally made a middleware for this: express-lfs with a demo here: https://expresslfs.herokuapp.com

There you can download a 400Mo file as a proof.

See usage here: https://github.com/goodenough/express-lfs#usage

PS: Thanks to @fundeldman for good advices in his answer ;)

abernier
  • 27,030
  • 20
  • 83
  • 114