0

Recently, I came upon this useful library called swifter to speed up the processing of pandas series in python. I am sure it does a lot of vectorized processing and optimization under the hood, but I am curious how it manages to introduce a new attribute to a pandas series or data frame object sheerly by virtue of being imported. Take this minimal code.

#!/usr/bin/env python3
# encoding: utf-8
import pandas as pd # Version 1.4.1
import numpy as np # Version 1.22.3
samples:int=2**20
colname:str='Value'
frame=pd.DataFrame(data={colname:np.random.randint(low=0, high=45353, size=samples)})

import swifter # Version 1.22.3 The following line throws an attribute error without this import
frame[colname].swifter # With swifter imported, this is swifter.SeriesAccessor object

This really seems like magic, because I thought an import statement can introduce new classes, functions, etc. but cannot alter the API (methods available) of objects that already exist in the namespace. That just somehow conflicts with my mental model of how objects work and interact in an OOP paradigm. Any guidance on how it is done, or whether it uses some deeper advanced feature of the python language would be great.

Manoj Agrawal
  • 429
  • 3
  • 10
Della
  • 1,264
  • 2
  • 15
  • 32

1 Answers1

2

When you import a module or a package, Python loads and execute the code inside it.

When you import swifter, Python loads the swifter.__init__ file which contains:

if "modin.pandas" in sys.modules:
    register_modin()

As the condition is true, the register_modin is executed. Generally, when you import a module or a package, the code has no side effect except register some variables, functions or classes in the local scope.

To avoid magic, the good practice would be something like:

from swifter import register_modin
register_modin()

Take a look to pandas.api.extensions.register_dataframe_accessor

Corralien
  • 109,409
  • 8
  • 28
  • 52
  • Even your code snippet to ```register_modin()``` does not seem to touch the pandas classes directly. So you mean there is something inside ```register_modin()``` that can modify pandas without explicitly touching it at user facing code? – Della Jul 29 '22 at 09:03