Quickest way to combine 2 strings, interleaving string from second column into the first by row throughout the dataframe

Question

I have written a function (incorporating bits and pieces scavenged from stack overflow) that will move throughout a data frame by row, interleaving strings from col-x to col-y, for all two column x,y pair in all rows.

I have a working solution. The problem is that it is slow going on large data frames.

Is there a quicker way?

I have tried the following setup:

# Import modules
import pandas as pd
from itertools import chain, zip_longest

def interleave_strings(string1, string2):
    tuples = zip_longest(string1, string2, fillvalue='')
    string_list = [''.join(item) for item in tuples]
    return ''.join(string_list)

# Create the pandas DataFrame 
data = [['timy', 'toma', 'tama', 'tima', 'tomy', 'tome'], ['nicka', 'nacka', 'nucka', 'necka', 'nomy', 'nome'], ['julia', 'Julia', 'jalia', 'jilia', 'jomy', 'jome']] 
df = pd.DataFrame(data, columns = ['A', 'B', 'C', 'D', 'E', 'F']) 

df

This gets us...

    timy    toma    tama    tima    tomy    tome
    nicka   nacka   nucka   necka   nomy    nome
    julia   Julia   jalia   jilia   jomy    jome

And this works, but slowly...

# new_df

il_df = pd.DataFrame()
for i in range (int(len(df.columns)/2)):
    selection = df.iloc[:,2*i:2*i+2]
    L = []
    for j in range (len(df.index)):
        res = interleave_strings(selection.iloc[j,0], selection.iloc[j,1])

        L.append(res)
        S = pd.Series(L)
    #il_df = pd.concat(D, ignore_index=True)   
    il_df = il_df.append(S, ignore_index=True)

And with

il_df.transpose()

The correct output is:

    0           1           2
0   ttiommya    ttaimmaa    ttoommye
1   nniacckkaa  nnuecckkaa  nnoommye
2   jJuulliiaa  jjailliiaa  jjoommye

Are the columns shown in the "correct output" partial? I was expecting to see 6P2 columns. — Karthik V, Jul 03 '19 at 21:12
Are the words in the combined columns always the same length? For example: `timy, toma` and `nicka, nacka`? — Erfan, Jul 03 '19 at 21:55

score 0 · Answer 1 · answered Jul 03 '19 at 22:25

We can use groupby over each pairs of two columns over axis=1 (read: column axis).

Just like your own solution, we use interleave:

from toolz import interleave

m = [x//2 for x in range(len(df.columns))]

df = df.groupby(m, axis=1).apply(lambda x: [''.join(interleave(t)) for t in zip(x.iloc[:, 0], x.iloc[:, 1])])

df = pd.DataFrame(df.to_numpy().tolist(), columns = df.index).T

output

            0           1         2
0    ttiommya    ttaimmaa  ttoommye
1  nniacckkaa  nnuecckkaa  nnoommye
2  jJuulliiaa  jjailliiaa  jjoommye

Notice If your pandas version is < 0.24 use .values instead of .to_numpy

df = pd.DataFrame(df.values.tolist(), columns = df.index).T

score 0 · Answer 2 · answered Jul 03 '19 at 22:27

We can do this in two steps. First create a new frame containing all the permutations of (x, y), and then apply a function to interleave the strings of element of the new frame.

  >>>import pandas as pd
  >>>import itertools
  >>>df
  Out[61]: 
         A      B      C      D     E     F
  0   timy   toma   tama   tima  tomy  tome
  1  nicka  nacka  nucka  necka  nomy  nome
  2  julia  Julia  jalia  jilia  jomy  jome

  >>>df_permute = df.apply(lambda x: pd.Series(list(itertools.permutations(x, 2))), axis=1)
  >>>df_permute
  Out[66]: 
                 0               1       ...                  28            29
  0    (timy, toma)    (timy, tama)      ...        (tome, tima)  (tome, tomy)
  1  (nicka, nacka)  (nicka, nucka)      ...       (nome, necka)  (nome, nomy)
  2  (julia, Julia)  (julia, jalia)      ...       (jome, jilia)  (jome, jomy)
  [3 rows x 30 columns]

  >>>def foo(x, y):
  ...  """Interleave string x, and y"""
  ...  return ''.join(p for p in itertools.chain(*izip_longest(x, y)) if p)
  ...

  >>> df_permute.applymap(lambda x: foo(*x))
  Out[68]: 
             0           1           2     ...            27         28        29
  0    ttiommya    ttiammya    ttiimmya    ...      ttoammea   ttoimmea  ttoommey
  1  nniacckkaa  nniucckkaa  nniecckkaa    ...     nnoumceka  nnoemceka  nnoommey
  2  jJuulliiaa  jjualliiaa  jjuilliiaa    ...     jjoamleia  jjoimleia  jjoommey
  [3 rows x 30 columns]

score 0 · Answer 3 · answered Aug 30 '19 at 19:44

Thanks for your responses! They are appreciated. I originally asked, "Is there a quicker way to do this." So if you are interested, it appears that Erfan's method is quicker than mine by half while Karthik's is a bit slower than mine.

Here are the results from %%timeit ran in jupyterlab for the actual interleaving. Those ms would add up if you had larger dataframes.

Erfan   - 3.46 ms ± 150 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
greg    - 6.81 ms ± 113 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Karthik - 10.6 ms ± 98.9 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Cheers!

Quickest way to combine 2 strings, interleaving string from second column into the first by row throughout the dataframe

3 Answers3