24

Here's an MWE of some code I'm using. I slowly whittle down an initial dataframe via slicing and some conditions until I have only the rows that I need. Each block of five rows actually represents a different object so that, as I whittle things down, if any one row in each block of five meets the criteria, I want to keep it -- this is what the loop over keep.index accomplishes. No matter what, when I'm done I can see that the final indices I want exist, but I get an error message saying "IndexError: positional indexers are out-of-bounds." What is happening here?

import pandas as pd
import numpy as np

temp = np.random.rand(100,5)

df = pd.DataFrame(temp, columns=['First', 'Second', 'Third', 'Fourth', 'Fifth'])

df_cut = df.iloc[10:]

keep = df_cut.loc[(df_cut['First'] < 0.5) & (df_cut['Second'] <= 0.6)]

new_indices_to_use = []
for item in keep.index:
    remainder = (item % 5)
    add = np.arange(0-remainder,5-remainder,1)
    inds_to_use = item + add
    new_indices_to_use.append(inds_to_use)

new_indices_to_use = [ind for sublist in new_indices_to_use for ind in sublist]
final_indices_to_use = []
for item in new_indices_to_use:
    if item not in final_indices_to_use:
        final_indices_to_use.append(item)

final = df_cut.iloc[final_indices_to_use]
Arnold
  • 863
  • 5
  • 13
  • 21
  • 1
    Have you tried printing `final_indices_to_use` and verifying it is what you think it is? On your example I get `[10, 11, ..., 98, 99]` and `len(df_cut)` gives `90`. – TemporalWolf May 22 '17 at 22:34
  • 1
    I'm not sure whether to vote to close as a typo or not, but it looks to me like you should be using `.loc` instead of `.iloc` in your last line -- `.iloc` is for positional access, but you want label-based access. – DSM May 22 '17 at 22:40
  • 1
    Hmmmm in my MWE, changing it to .loc does solve the problem. I'll have to check if it works in my actual code -- I could swear I tried that already. Why is .iloc not right here? These are indices, and the description of .iloc says it accepts a list of indices. – Arnold May 22 '17 at 22:49
  • Also, if I use an opposite condition and then do final = df_cut.drop(df_cut.index[final_indices_to_use]), I get the same error. – Arnold May 22 '17 at 22:55
  • 1
    Because `df_cut` has a length of 90; indices above `89` will give an `IndexError`. – TemporalWolf May 22 '17 at 22:59
  • That is not true. I didn't reset the indices on the dataframe so everything from 10 to 99 is still there. – Arnold May 22 '17 at 23:00
  • (You have to print the actual dataframe to see this, not the length of it) – Arnold May 22 '17 at 23:00
  • not for `iloc`: `print df_cut.iloc[99]` gives `IndexError: single positional indexer is out-of-bounds` – TemporalWolf May 22 '17 at 23:05

1 Answers1

52

From Pandas documentation on .iloc (emphasis mine):

Pandas provides a suite of methods in order to get purely integer based indexing. The semantics follow closely python and numpy slicing. These are 0-based indexing.

You're trying to use it by label, which means you need .loc

From your example:

>>>print df_cut.iloc[89]
...
Name: 99, dtype: float64

>>>print df_cut.loc[89]
...
Name: 89, dtype: float64
TemporalWolf
  • 7,727
  • 1
  • 30
  • 50