1

I know I can use Powershell to initiate and manage a BITS (Background Intelligent Transfer Service) download from my server over VPN, and I am looking to do that to stage large install resources locally while a regular user is logged on, ready for use in software installs and updates later. However, I would like to also support cloud services for the download repository, as I foresee (some) firms no longer having centralized servers and VPN connections, just cloud repositories and a distributed workforce. To that end I have tested using Copy-S3Object from the AWS Powershell tools, and that works. But it isn't throttle-able so far as I can tell. So I wonder, is there a way to configure my AWS bucket so that I can use BITS to do the download, but still constrained by AWS credentials? And if there is, is the technique valid across multiple cloud services, such as Azure and Google Cloud? I would LIKE to be cloud platform agnostic if possible.

I have found this thread, that seems to suggest that creating presigned URLs would work. But my understanding of that process is, well, non existent. I am currently creating credentials for every user. Do I basically assign those users to an AWS group and give that group some permissions, and then Powershell can be used to sign a URL with the particular user's credentials, and that URL is what BITS uses? So a user who has been removed from the group would no longer be able to create signed URLs, and so would no longer be able to access the available resources?

Alternatively, if there is a way to throttle Copy-S3Object that would work too. But so far as I can tell that is not an option.

Gordon
  • 6,257
  • 6
  • 36
  • 89

1 Answers1

1

Not sure of a way to throttle the copy-s3 object but you can definitely BITS a pre-signed s3 URL.

For example, if you have your AWS group with users a/b/c in there, and the group has a policy attached that allows the relevant access to your bucket - those users a/b/c will be able to create pre-signed URLs for objects in that bucket. For example, the following create a pre-signed url for an object called 'BITS-test.txt':

aws s3 presign s3://youbucketnamehere/BITS-test.txt

That will generate a pre-signed URL that can be passed into an Invoke-WebRequest command.

This URL is not restricted to only those users though, anybody with this URL will be able to download the object - but only users a/b/c (or anyone else with access to that bucket) will be able to create these URLs. If you don't want users a/b/c to be able to create these URLs anymore, then you can just remove them from the AWS group like you mentioned.

You can also add an expiry param to the presign command for example --expires-in 60 which keeps the link valid for only that period of time (in this case for 1 hour - expiry param is set in minutes).

blinu
  • 68
  • 8
  • Ah, @blinu, that's interesting. I had been looking for the permissions I needed to allow creation of a presigned URL. Access to the target of the URL makes sense, and explains why I wasn't finding info looking for allow presigned URL creation permissions. I suspect I will need to either limit the URL to an hour or two, or even test download performance and set the timeout appropriately. I guess this week I'll try to get this working with AWS S3, then see if I can apply a similar approach to Azure and Google Cloud. Thanks! – Gordon Feb 28 '21 at 19:17
  • @Gordon Np at all! You can definitely do something similar in Azure using Az PowerShell and the New-AzStorageBlobSASToken command. I have no experience with GCP but I'm sure there will be an equivalent there too. Good luck! – blinu Feb 28 '21 at 21:21