Remove not representative values from a list

Question

I'm wondering how to remove not representative samples from the population.

Let's assume that I have following numbers as a list:

pop = [1, 2, 3, 4, 1, 3, 3, 4, 5, 7, 2, 1, 1, 1, 1, 1, 1002]

Is there any way to inspect the list and remove value 1002 as it is completely not representative and can disturb calculations?

To illustrate this let make the following computation:

>>> pop = [1, 2, 3, 4, 1, 3, 3, 4, 5, 7, 2, 1, 1, 1, 1, 1, 1002]
>>> mean(pop)
61.294117647058826
>>> pop1 = [1, 2, 3, 4, 1, 3, 3, 4, 5, 7, 2, 1, 1, 1, 1, 1]      
>>> mean(pop1)
2.5

you should look at the outliers. You could see this topic. https://stackoverflow.com/q/70849524/14823310 — pacdev, Mar 04 '22 at 13:30
Does this answer your question? [Python: Find outliers inside a list](https://stackoverflow.com/questions/62802061/python-find-outliers-inside-a-list) — mkrieger1, Mar 04 '22 at 13:57

pacdev · Accepted Answer · 2022-03-04T13:53:54.123

data = {"pop": [1, 2, 3, 4, 1,3,3,4,5,7,2,1,1,1,1,1,1002]}

df = pd.DataFrame(data)

q3 = np.quantile(df["pop"], 0.75)
q1 = np.quantile(df["pop"], 0.25)

iqr = q3 - q1
upper_bound = q3 + 1.5 * iqr
lower_bound = q1 - 1.5 * iqr

df_wo_outliers = df[(df["pop"] >= lower_bound) & (df["pop"] <= upper_bound)]

But be aware that it could also remove 7 for instance as your upper bound is 5.5

df_wo_outliers["pop"].mean()
# 2.5

Remove not representative values from a list

1 Answers1