I tried to pass a string parameter into a Spark NoteBook written entirely in .NET Spark C# No matter what I tried it did not work. What finally did work was to
- Define the notebook as PySharp
- Define the parameter - PySharp
- Put the parameter value in a temporary table - PySharp
- Then from C# I could pull the value from the temporary table and perform my logic.
See the example code at the bottom but does anyone have a way to make this work in C# without all of the jumping around?
--- parameter cell ---
sourcefilepath = "test"
--- cell 1 ---
from pyspark.sql.types import StructType,StructField, StringType
schema = StructType([ StructField("sourcefilepath",StringType(),True)])
df = spark.createDataFrame([[sourcefilepath]],schema)
df.createOrReplaceTempView("sourcefilepathTable") ;
--- cell 2 ---
%%csharp
using System;
using System.Collections.Generic;
using Microsoft.Spark.Sql;
using Microsoft.Spark.Sql.Types;
using System.Diagnostics;
using System.IO ;
using System.Text.Json;
using System.IO.Compression ;
var dfSql = spark.Sql("Select sourcefilepath from sourcefilepathTable");
string sourcefilepath = dfSql.First().GetAs<string>("sourcefilepath");
-- remainder of my code goes here