NetworkX find_cliques error using PySpark

Question

I'm trying to calculate find_cliques functionality to locate the maximal cliques for each subgroup.

I'm using this implementation using pandas_udf grouped by each connected component.

def pd_create_subgroups(pdf):

    index = pdf.component.unique()[0]

    try:
        # building the graph
        gnx = nx.from_pandas_edgelist(pdf, "src", "dst")

        bic = list(find_cliques(gnx))

        if len(bic) <= 1:
            return pd.DataFrame(data={"cliques": [[f"issue_{index}"]]})

        bic_sorted = sorted(map(sorted, bic))
        bic_sorted = [b for b in bic_sorted if len(b) >= 3]

        if len(bic_sorted) == 0:
            return pd.DataFrame(data={"cliques": [[f"issue_{index}"]]})

        return pd.DataFrame([bic_sorted]).transpose().rename(columns={0: "cliques"})
    except:
        return pd.DataFrame(data={"cliques": [[f"issue_{index}"]]})

pdf is a pandas dataframe containing the fields src, dst, component it has around 200M-300M undirected edges

and returns the following error -

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 12.0 failed 4 times, most recent failure: Lost task 0.3 in stage 12.0 (TID 331) (executor 9): java.lang.IndexOutOfBoundsException: index: 2147483628, length: 36 (expected: range(0, 2147483648))

When running on smaller graphs it works properly.