I have an application that takes files from one place and moves them to another place - pretty much all this application does is checks if files are in s3 and downloads ones that are not to another s3. Currently application uses very low amounts of provided CPU. From this post, my understanding that it is to be expected (seeing as my app is pretty much I/O and nothing more).
My initial idea was to lower the number of CPUs provided to the app. However, providing less and less negatively impacted the speed at which my app performs its duties (which according to this article kind of make sense - less CPU means less total clock speed). This is not an option as it needs to run somewhat fast.
I am using kafka messages to start my app. So another idea of mine was to increase the number of partitions in my topic from which my app consumes the messages (so that I can increase the no. of threads that can run concurrently). That allowed me to reduce the number of CPU that I provide to my app (while maintaining the desired processing speed) but my app still uses very low amounts of CPUs.
My app runs in kubernates whose cluster is deployed to EC2s, if that is of any difference. My app is springBoot java. I tried to only give it a minimum number of CPUs, while maxing out the no. of concurrent threads in my app, but again I can see a lot of wasted CPU there.
My question is then as follows: is it possible to somehow make an application to use all available CPU (thus making it more efficient) in this scenario? Is there a config or a method or something that does that? Or for an app that checks data is present and downloads data somewhere else, this is an expected behavior - increasing the number of available resource would improve speed at which my app runs but as a cons of that, there will be waster CPU? (so I am in the classic "good comes with the bad" sort of situation here?)
