Pyspark string comparison with == for RDkit functions throws error

Question

I've a Pyspark UDF defined as below -

from rdkit import Chem

input_smile = 'CCOC(=O)c1cc2cc(ccc2[nH]1)C(=O)O'
converted_smile_in = Chem.MolToSmiles(Chem.MolFromSmiles(input_smile)

def convertSmile(smile):
        return (Chem.MolToSmiles(Chem.MolFromSmiles(smile)))
applyconvertSmileUdf = udf(convertSmile)

data_converted = data_converted.withColumn("converted_smile", applyconvertSmileUdf(data_filtered.smiles))

if __name__ == "__main__": 
        # using the new approach
        data_converted.filter(data_converted.converted_smile == converted_smile_in ).select("id","smiles").show()
else:
        print("Cannot convert!")

Comparison between data_converted.converted_smile and converted_smile_in throws error. I've printed some 20 values for converted_smile and it looks good. Can't we do string comparison this way?

Boost.Python.ArgumentError: Python argument types in rdkit.Chem.rdmolfiles.MolToSmiles(NoneType) did not match C++ signature: MolToSmiles(RDKit::ROMol mol, bool isomericSmiles=True, bool kekuleSmiles=False, int rootedAtAtom=-1, bool canonical=True, bool allBondsExplicit=False, bool allHsExplicit=False, bool doRandom=False)

score 0 · Answer 1 · answered Feb 04 '20 at 10:50

replace

data_converted.filter(data_converted.converted_smile == converted_smile_in ).select("id","smiles").show()

with

from pyspark.sql.functions import lit

data_converted.filter(data_converted.converted_smile == lit(converted_smile_in) ).select("id","smiles").show()

Pyspark string comparison with == for RDkit functions throws error

1 Answers1