2

I tried to pass a string parameter into a Spark NoteBook written entirely in .NET Spark C# No matter what I tried it did not work. What finally did work was to

  1. Define the notebook as PySharp
  2. Define the parameter - PySharp
  3. Put the parameter value in a temporary table - PySharp
  4. Then from C# I could pull the value from the temporary table and perform my logic.

See the example code at the bottom but does anyone have a way to make this work in C# without all of the jumping around?

--- parameter cell ---
sourcefilepath = "test" 
--- cell 1 ---
from pyspark.sql.types import StructType,StructField, StringType

schema = StructType([ StructField("sourcefilepath",StringType(),True)])
df = spark.createDataFrame([[sourcefilepath]],schema)
df.createOrReplaceTempView("sourcefilepathTable") ;

--- cell 2 ---
%%csharp
using System;
using System.Collections.Generic;
using Microsoft.Spark.Sql;
using Microsoft.Spark.Sql.Types;
using System.Diagnostics;
using System.IO ;
using System.Text.Json;

using System.IO.Compression ;
var dfSql = spark.Sql("Select sourcefilepath from sourcefilepathTable");

string sourcefilepath = dfSql.First().GetAs<string>("sourcefilepath");

-- remainder of my code goes here


bmukes
  • 119
  • 2
  • 9

0 Answers0