Select columns from dataframe on condition they exist

Question

I have a pandas DataFrame with multiple columns (columns names are numbers; 1, 2, ...) and I want to copy some of them if they do exist.

For example df1 = df[[1,2,3,4]] But it might happen that some columns do not exist in df, eg df might only have columns 1, 2, and 4 or columns 1, and 2 etc

Eric O. Lebigot · Answer 1 · 2020-10-03T20:27:29.570

35

It is simpler to directly calculate the set of common columns and ask for them:

df[df.columns & [1, 2, 3, 4]]

(The & operator is the (set) intersection operator.)

edited Oct 03 '20 at 20:27

answered Jun 12 '19 at 13:27

Eric O. Lebigot

91,433
48
218
260

1

I prefer this solution over the original accepted one, as this is simpler to be implemented in one line – Daniel Ortega Oct 02 '20 at 09:23
4

I get a `FutureWarning` stating that this method is deprecated as of 1.2.4, and that `index.intersection(other)` should be used instead. – zmike May 26 '21 at 23:32

score 34 · Accepted Answer · answered Apr 21 '17 at 08:06

34

Use isin with loc to filter, this will handle non-existent columns:

In [97]:
df = pd.DataFrame(columns=[1,2,4])
df.loc[:,df.columns.isin([1,2,3,4,])]

Out[97]:
Empty DataFrame
Columns: [1, 2, 4]
Index: []

answered Apr 21 '17 at 08:06

EdChum

376,765
198
813
562

score 4 · Answer 3 · answered Mar 18 '21 at 03:56

One possible way:

df[df.columns.intersection(set(['list', 'of', 'cols']))]

For example:

$ ipython
Python 3.8.5 (default, Sep  3 2020, 21:29:08) [MSC v.1916 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 7.20.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]:
import pandas as pd
df = pd.DataFrame(columns=[1,2,3,4])
df
Out[1]:
Empty DataFrame
Columns: [1, 2, 3, 4]
Index: []

In [2]:
df[df.columns.intersection(set([1, 2, 2, 5]))]
Out[2]:
Empty DataFrame
Columns: [1, 2]
Index: []

In [3]:
pd.__version__
Out[3]:
'1.2.1'

This is the modern, non-deprecated way. – Nermin Mar 06 '23 at 09:34 — Nermin, Mar 06 '23 at 09:34

Select columns from dataframe on condition they exist

3 Answers3