0

When I launch a new instance from an AMI (this AMI opens steamVR after logon), steamVR runs into an error. From what I have been reading it is because when an instance is launched based on an AMI, the EBS volume attached to the ec2 instance needs to get all the data from the snapshot that the AMI is based off of. i/o operations are slow for the first read and therefore when SteamVR is launched on login, it runs into a critical error. more details are here: https://forums.developer.nvidia.com/t/failed-watchdog-timeout-in-thread-server-main-in-cloudxrremotehmd-after-10-440015-seconds-aborting/180742/10

In this post, the final reply is the following:

"We did figure out a workaround last week. It turns out this issue is related specifically to AWS machines with EBS. When you start an EBS-backed machine in AWS, the EBS volume needs to be hydrated from the snapshot that it is based on. The snapshot data is backed by S3, and therefore is very slow for the first read of data from the volume. We are now using a little PowerShell script to basically cat {files} > /dev/null for all the files in the CloudXR and SteamVR directories before we start SteamVR. This hydrates those files into the volume so that they are fast, and we haven’t had issues crashing since then. More information on this is here: https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/ebs-initialize.html#ebs-initialize-windows. Hope this helps."

How do I initialize the ebs volume for only the files in the steamvr and cloudxr directories? If I did use fio/dd to initialize the entire volume (200gb), how long would that take?

Any help would be greatly appreciated!

ugexe
  • 5,297
  • 1
  • 28
  • 49
Anish Kumar
  • 3
  • 1
  • This might interest you (extra charges apply, so beware!): [Amazon EBS fast snapshot restore - Amazon Elastic Compute Cloud](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-fast-snapshot-restore.html) – John Rotenstein Mar 15 '23 at 06:49
  • Based on the article you quoted, simply `cat` the files in those directories. – John Rotenstein Mar 15 '23 at 06:50
  • I did look into fast snapshot restore but it did not work every time. Plus the extra charges would really add up. Additionally, I am not sure how to cat the files into those directories. Like, would I cat files from the snapshot into the ebs volume? If so, how would I go about doing that? – Anish Kumar Mar 15 '23 at 21:54

1 Answers1

0

When an Amazon EBS volume is created from an AMI, AWS does not actually copy an blocks to the Amazon EBS volume.

Instead, when a block on the EBS volume is first accessed, the EBS service checks to see whether the block is empty (unused) or whether it is a block that was stored in the AMI. If it was stored in the AMI, then the EBS service copies the block from the AMI snapshot to the EBS volume. This is why there is a slight delay.

If you wish to avoid the delay, then it is necessary to access the block to force the "copy from snapshot to volume" operation to occur. One way to do this is to cat a file, which causes the underlying storage block(s) that hold that file to be copied to the volume.

Therefore, to speed up access to the steamvr and cloudxr directories, you could simply initiate a command that reads the files in those directories.

Based on Concatenating thousands of files: > vs >>, you could use something like:

find /steamvr /cloudxr  -print0 | xargs -0 cat >/dev/null

(Adjust directories names as required.)

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
  • Thank you for answering! Since I am on powershell, I get the error The term 'xargs' is not recognized as the name of a cmdlet, function, script file, or operable program. I try running a "powershell equivalent" of the command (for steamvr): Get-ChildItem -Path "C:\Program Files (x86)\Steam\steamapps\common\SteamVR" -Recurse | ForEach-Object { Get-Content $_.FullName | Out-Null } And I get permission errors for certain folders even though I am running powershell as administrator and administrator has full control access when I look at the properties of those folders. – Anish Kumar Mar 15 '23 at 23:17