-2

I understand that the question might look too specific, but I think the answers can also be useful for other areas.

The real problem I'm having: Given an arbitrary container image that has sh binary, download a file from Google Cloud Storage.

Since the downloader binary does not exist in the user image I had an idea to put the encoded binary in the command-line (a bit similar to inline scripts like python -c <code>). However there are limitation on the command-line size (~1MiB), so the binary shound be smaller that that.

I've decided to use Go due to the ease of creating statically-linked binaries and people reporting that GO binaries are small.

I've written a Go program that uses the cloud.google.com/go/storage module to create a client and read the blob storage.NewClient().Bucket(bucket).Object(object).NewReader().

Unfortunately, the binary for the simple program turns out to be 15MiB. Adding go build -ldflags=" -s -w" (as suggested in How to reduce compiled file size?) reduces that to 10MiB. That's still order of magnitude more than my size budget.

Are there binary optimizations or pruning techniques that I could use?

Should I look at some other language (e.g. C++) if I wish to have a smaller binary?

Will I have better luck if I try to use some generic REST client library? //Looks like no - the size of stripped Go program that uses http.Get is 5MiB.

Ark-kun
  • 6,358
  • 2
  • 34
  • 70
  • The *"people reporting that GO binaries are small"* were lying to you. – Jonathon Reinhart Apr 12 '20 at 23:57
  • Does this answer your question? [How to reduce compiled file size?](https://stackoverflow.com/questions/3861634/how-to-reduce-compiled-file-size) – Jonathon Reinhart Apr 12 '20 at 23:58
  • @JonathonReinhart: FWIW most answers in that question are suggesting the `ldflags` options the OP already mentioned – Eli Bendersky Apr 13 '20 at 00:06
  • Can the object in GCS or S3 be publicly visible, or do you need the user to be authorized to download it? If the former, we can probably accomplish this by using a very small HTTP library (not sure how big libcurl is but there are other options) to fetch the resource `https://storage.googleapis.com/bucket_name/object_name`. – Brandon Yarbrough Apr 13 '20 at 16:17
  • @BrandonYarbrough Authorization is needed. But this got me thinking about a two-stage solution where the GCS downloader is first downloaded from public HTTP. Unfortunately, Go's http library compiles to a pretty big binary. – Ark-kun Apr 16 '20 at 20:02

1 Answers1

2

I don't think you'd be able to bring the size of a Go binary below 1 MiB, particularly if you have to use feature-full 3rd party libraries like the GCP storage bindings.

It's rather strange that you have a 1 MiB limitation in this day and age, but if this is really the case I'd recommend:

  1. If your environment has an interpreted language runtime (like Python, Ruby etc.) use that because the size of your source code is all you ship.
  2. Otherwise, you'll probably need to use something C++. That said, if your binary has to access cloud storage and you want to use packages/libraries to do that, 1 MiB can still be challenging unless you're very careful. There's a lot of functionality involved in accessing cloud storage - security, HTTP/RPC clients, etc.
Eli Bendersky
  • 263,248
  • 89
  • 350
  • 412
  • >"It's rather strange that you have a 1 MiB limitation in this day and age". I think this is the limitation of `exec` and also the max size of a single Kubernetes object. – Ark-kun Apr 13 '20 at 00:09
  • @Ark-kun: what do you mean by exec? And what do you mean by single k8s object? Surely k8s can run binaries larger than 1 MiB... – Eli Bendersky Apr 13 '20 at 00:11
  • >"If your environment has an interpreted language runtime " - I wish I had this guarantee. I'd have just used the official `gsutil` tool written in python. Unfortunately we need to make this work for an arbitrary user-provided container image. – Ark-kun Apr 13 '20 at 00:11
  • exec means the family of Linux kernel functions used to start processes on Linux. https://stackoverflow.com/questions/14176089/system-command-line-max-length-in-perl Kubernetes has a strict limit on an object size (for example on the size of Pod object structure that specifies the container image and the command-line). I'm limited by the maximum command-line size. – Ark-kun Apr 13 '20 at 00:14