Left merging dask dataframes results to empty dataframe

Question

I have the following code:

raw_data = pd.DataFrame({'username':list('ab')*10, 'user_agent': list('cdef')*5, 'method':['POST'] * 20, 'dst_port':[80]*20, 'dst':['1.1.1.1']*20})
past = pd.DataFrame({'user_agent':list('cde'), 'percent':[0.3, 0.3, 0.4]})
past = past.set_index('user_agent')
import dask.dataframe as dd
dask_raw = dd.from_pandas(raw_data, npartitions=4)
dask_past = dd.from_pandas(past, npartitions=4)
merged_raw = dask_raw.merge(dask_past, how='left', left_on='user_agent', right_index=True)

Compute on merged_raw gives me the dataframe of this form:

Out[20]: 
    dst  dst_port method user_agent username  percent
12  1.1.1.1        80   POST          c        a      0.3
16  1.1.1.1        80   POST          c        a      0.3
8   1.1.1.1        80   POST          c        a      0.3
0   1.1.1.1        80   POST          c        a      0.3
4   1.1.1.1        80   POST          c        a      0.3
10  1.1.1.1        80   POST          e        a      0.4
11  1.1.1.1        80   POST          f        b      NaN
13  1.1.1.1        80   POST          d        b      0.3
14  1.1.1.1        80   POST          e        a      0.4
15  1.1.1.1        80   POST          f        b      NaN
17  1.1.1.1        80   POST          d        b      0.3
18  1.1.1.1        80   POST          e        a      0.4
19  1.1.1.1        80   POST          f        b      NaN
5   1.1.1.1        80   POST          d        b      0.3
6   1.1.1.1        80   POST          e        a      0.4
7   1.1.1.1        80   POST          f        b      NaN
9   1.1.1.1        80   POST          d        b      0.3
1   1.1.1.1        80   POST          d        b      0.3
2   1.1.1.1        80   POST          e        a      0.4
3   1.1.1.1        80   POST          f        b      NaN

Computing the features:

grouped_by_df = merged_raw.groupby(['username', 'dst', 'dst_port'])
feature_one = grouped_by_df.apply(lambda x: 'POST' in x.values).to_frame(name='feature_one')
feature_two = grouped_by_df.percent.min()
feature_two = feature_two.fillna(0)
feature_two = feature_two.to_frame(name='feature_two')
features_three = grouped_by_df.method.apply(lambda x: 'CONNECT' in x.values).to_frame(name='feature_three')
features = feature_one.merge(feature_two, left_index=True, right_index=True, how='left')
features.compute()
                       feature_one  feature_two
username dst     dst_port                          
a        1.1.1.1 80               True          0.3
b        1.1.1.1 80               True          0.3

features_full = features.merge(features_three, how='left', right_index=True, left_index=True)
features_full.compute()

Result is:

Out[53]: 
Empty DataFrame
Columns: [feature_one, feature_two, feature_three]
Index: []

But features_three has values and are the same index as features

feature_three.compute()
username dst     dst_port               
a        1.1.1.1 80                False
b        1.1.1.1 80                False

Why does dask return an empty datframe?

How is one to understand your first merge operation? Can you outline what you expect merged_raw to look like. If I do the equivalent in pandas I get percent and user_agent_y columns filled with NaNs. On the other hand, raw_past = raw_data.merge(past, how='left', left_on='user_agent', right_on='user_agent') returns as expected merged df with a percent column with NaNs only where user_agent is f. — KRKirov, Mar 26 '18 at 13:25
@KRKirov The first merge merges raw data with the percent of the user_agents in the past data. If a user_agent in the raw does not exist in the past it must add NaN value. So the result of merged data is correct. — Apostolos, Mar 26 '18 at 17:38
Looking at this I wonder whether what you get is a result of a bug as well https://github.com/dask/dask/issues/2237 — KRKirov, Mar 26 '18 at 18:53
It seems worth it. In the worst case, the issue will get closed and you may still get an idea why you get such results. — KRKirov, Mar 26 '18 at 19:19

KRKirov · Answer 1 · 2018-03-26T18:35:18.480

This does not fully solve of your problem, however here is what I get if I compute the merged_raw data frame first. If I comment the merged_raw.compute() command I get an error message. I wonder whether you could use pandas all the way and do the parallel computation using dask delayed functions.

import dask.dataframe as dd
import pandas as pd

raw_data = pd.DataFrame({'username':list('ab')*10, 'user_agent': list('cdef')*5, 'method':['POST'] * 20, 'dst_port':[80]*20, 'dst':['1.1.1.1']*20})
past = pd.DataFrame({'user_agent':list('cde'), 'percent':[0.3, 0.3, 0.4]})
past = past.set_index('user_agent')

dask_raw = dd.from_pandas(raw_data, npartitions=4)
dask_past = dd.from_pandas(past, npartitions=4)
merged_raw = dask_raw.merge(dask_past, how='left', left_on='user_agent', right_index=True)

merged_raw = merged_raw.compute()

grouped_by_df = merged_raw.groupby(['username', 'dst', 'dst_port'])
feature_one = grouped_by_df.apply(lambda x: 'POST' in x.values).to_frame(name='feature_one')
feature_two = grouped_by_df.percent.min()
feature_two = feature_two.fillna(0)
feature_two = feature_two.to_frame(name='feature_two')
features_three = grouped_by_df.method.apply(lambda x: 'CONNECT' in x.values).to_frame(name='feature_three')

features = feature_one.merge(feature_two, left_index=True, right_index=True, how='left')

features_full = features.merge(features_three, how='left', right_index=True, left_index=True)

features_full
Out[85]: 
                           feature_one  feature_two  feature_three
username dst     dst_port                                         
a        1.1.1.1 80               True          0.3          False
b        1.1.1.1 80               True          0.3          False


# Error message when the merged_raw.compute() command is commented:
C:/CODE/apostolos.py:23: UserWarning: `meta` is not specified, inferred from partial data. Please provide `meta` if the result is unexpected.
  Before: .apply(func)
  After:  .apply(func, meta={'x': 'f8', 'y': 'f8'}) for dataframe result
  or:     .apply(func, meta=('x', 'f8'))            for series result
  feature_one = grouped_by_df.apply(lambda x: 'POST' in x.values).to_frame(name='feature_one')
C:/CODE/apostolos.py:31: UserWarning: `meta` is not specified, inferred from partial data. Please provide `meta` if the result is unexpected.
  Before: .apply(func)
  After:  .apply(func, meta={'x': 'f8', 'y': 'f8'}) for dataframe result
  or:     .apply(func, meta=('x', 'f8'))            for series result
  features_three = grouped_by_df.method.apply(lambda x: 'CONNECT' in x.values).to_frame(name='feature_three')

Left merging dask dataframes results to empty dataframe

1 Answers1