I have been looking at parallelizing options and found ray and modin. After some tests I got slightly lost in what benefits from using modin. Two examples:
df = pd.read_csv() for 180 MB file
pandas 5.2s vs. modin.pandas 2.7s
but df.groupby()
pandas 0.59s vs. modin.pandas 5.46s
What kind of applications will benefit from using modin? It there a general rule here or everything has to be tested separately?
From https://modin.readthedocs.io/en/latest/
Modin uses Ray or Dask to provide an effortless way to speed up your pandas notebooks, scripts, and libraries. Unlike other distributed DataFrame libraries, Modin provides seamless integration and compatibility with existing pandas code. Even using the DataFrame constructor is identical.
Two main features that stand out are:
In pandas, you are only able to use one core at a time when you are doing computation of any kind. With Modin, you are able to use all of the CPU cores on your machine.
With Modin, because of its light-weight, robust, and scalable nature, you get a fast DataFrame at 1MB and 1TB+
Specifically for the slow group_by part of the question, there is a github discussion that points out that regular old pandas works better than modin.pandas: https://github.com/modin-project/modin/issues/895
Modin is still under active development, the README.md from their github repo(https://github.com/modin-project/modin) tabulates panda API coverage mentioning these functions:
As a rule of thumb any transformations that you would be doing on the columns like the aggregate functions(groupby(), sum(), count()) will always be faster in modin.
The very simple reason is that Modin will be using the multiple cores of your machine and hence all these will be faster than what pandas would do.
typically if you are using .transform() or .apply() on any of columns, Modin will be able to do it faster.
However, there are a few cases in which modin WILL BE SLOWER than PANDAS. example :
.append()
thanks!
