0

I am attempting to use the library s3fs in a tornado application. The easiest approach would be to make a small function that fetches some data from s3 (using s3fs) and wrap that in run_in_executor().

Is there any advantage to writing a custom library to fetch from s3 using tornado's AsyncHTTPClient instead?

(Note: I am aware of botoaio and others. For my particular use-case I require the output to be in the format provided by s3fs.)

Cody Gray - on strike
  • 239,200
  • 50
  • 490
  • 574
chaimp
  • 16,897
  • 16
  • 53
  • 86
  • 2
    `AsyncHTTPClient` will be more performant than `run_in_executor`. But for low to medium traffic applications, I don't think there will be much difference. You can still run multiple tornado processes to compensate for that. Probably, start with `run_in_executor` and if and when performance becomes an issue, invest your time in writing an asynchronous client. – xyres Jul 26 '19 at 05:00

1 Answers1

1

AsyncHTTPClient is more efficient than running a synchronous library with run_in_executor. The difference is mainly the memory needed for thread stacks at high levels of concurrency. Performance will generally be similar as long as you have enough threads and memory for your level of concurrency. So unless you're doing a lot of s3 operations in parallel, there's little reason to invest a lot of work in converting everything to async interfaces.

Ben Darnell
  • 21,844
  • 3
  • 29
  • 50