Submit a U-SQL job using .net SDK

Question

I referred the site https://github.com/toddkitta/azure-content/blob/master/articles/data-lake-analytics/data-lake-analytics-get-started-net-sdk.md for submitting a U-SQL job but the above GitHub sample showing error in the console application in the Authentication AcquireToken method.

Please Suggest me a sample for submitting a U-SQL job using.Net SDK

Edited: PlatformParameters is showing error in my Visual Studio. I think, forgot include that class.

Answer 1

As Rick van den Bosch mentioned that we need to use AcquireTokenAsync method, I also following the document to do a demo. And change some code in my side, then it works correctly on my side. The following is my detail steps.

preparation：

1.Registry an native AD application and Add Windows Azure Service Management API permission, more details please refer to Azure official tutorials. After that we can get tenantId, appId, Redirect URI from the Azure Portal.

2.Create Data Lake Analytics and Data Lake Store account. I assgin permission to to folder or file in the data lake store, more detail please refer to another SO thread.

3.Prepare a script file in the local folder,I get the script from your mentioned document.

C:\Tom\SampleUSQLScript.txt

4.Upload the SearchLog.tsv to the Azure storage account and set it as public

Steps:

1.Create a console project and reference to corresponding SDK,details please refer to the packages.config section.

2.Add how to get TokenCredentials function

   public static TokenCredentials AuthenticateUser(string tenantId, string resource, string appClientId, Uri appRedirectUri, string userId = "")
        {
            var authContext = new AuthenticationContext("https://login.microsoftonline.com/" + tenantId);

            var tokenAuthResult = authContext.AcquireTokenAsync(resource, appClientId, appRedirectUri, new PlatformParameters(PromptBehavior.Auto),
                 UserIdentifier.AnyUser).Result;

            return new TokenCredentials(tokenAuthResult.AccessToken);
        }

Test the TokenCredentials functon

3.Add SetupClients function

 public static void SetupClients(TokenCredentials tokenCreds, string subscriptionId)
    {
        _adlaClient = new DataLakeAnalyticsAccountManagementClient(tokenCreds) {SubscriptionId = subscriptionId};

        _adlaJobClient = new DataLakeAnalyticsJobManagementClient(tokenCreds);

        _adlaCatalogClient = new DataLakeAnalyticsCatalogManagementClient(tokenCreds);

        _adlsClient = new DataLakeStoreAccountManagementClient(tokenCreds) {SubscriptionId = subscriptionId};

        _adlsFileSystemClient = new DataLakeStoreFileSystemManagementClient(tokenCreds);
    }

4.Add SubmitJobByPath function

 public static string SubmitJobByPath(string scriptPath, string jobName)
        {
            var script = File.ReadAllText(scriptPath);
            var jobId = Guid.NewGuid();
            var properties = new USqlJobProperties(script);
            var parameters = new JobInformation(jobName, JobType.USql, properties, priority: 1000, degreeOfParallelism: 1);
            var jobInfo = _adlaJobClient.Job.Create(_adlaAccountName,jobId, parameters);
            return jobId.ToString();
        }

5.Add other related functions

 public static void UploadFile(string srcFilePath, string destFilePath, bool force = true)
        {
            var parameters = new UploadParameters(srcFilePath, destFilePath, _adlsAccountName, isOverwrite: force);
            var frontend = new DataLakeStoreFrontEndAdapter(_adlsAccountName, _adlsFileSystemClient);
            var uploader = new DataLakeStoreUploader(parameters, frontend);
            uploader.Execute();
        }

        // Download file
  public static void DownloadFile(string srcPath, string destPath)
        {
            var stream = _adlsFileSystemClient.FileSystem.Open(srcPath, _adlsAccountName);
            var fileStream = new FileStream(destPath, FileMode.Create);

            stream.CopyTo(fileStream);
            fileStream.Close();
            stream.Close();
        }


 public static JobResult WaitForJob(string jobId)
        {
            var jobInfo = _adlaJobClient.Job.Get(_adlaAccountName,Guid.Parse(jobId));
            while (jobInfo.State != JobState.Ended)
            {
                jobInfo = _adlaJobClient.Job.Get(_adlaAccountName, Guid.Parse(jobId)); 
            }
            return jobInfo.Result.Value;
        }

public static void WaitForNewline(string reason, string nextAction = "")
        {
            if (!String.IsNullOrWhiteSpace(nextAction))
            {
                Console.WriteLine(reason + "\r\nPress ENTER to continue...");
                Console.ReadLine();
                Console.WriteLine(nextAction);
            }
            else
            {
                Console.WriteLine(reason + "\r\nPress ENTER to continue...");
                Console.ReadLine();
            }

6.Add the test submitting job code.

 private static void Main(string[] args)
    {
        _adlsAccountName = "data lake store account"; // TODO: Replace this value with the name for a created Store account.
        _adlaAccountName = "data lake analytics"; // TODO: Replace this value with the name for a created Analytics account.
        string localFolderPath = @"C:\tom\"; // TODO: Make sure this exists and contains the U-SQL script.

        // Authenticate the user
        // For more information about applications and instructions on how to get a client ID, see: 
        // https://azure.microsoft.com/en-us/documentation/articles/resource-group-create-service-principal-portal/
        var tokenCreds = AuthenticateUser("common", "https://management.core.windows.net/",
            "application id", new Uri("http://localhost")); // TODO: Replace applicaion id and redirect url values.

        SetupClients(tokenCreds, "subscription id"); // TODO: Replace subscription value.

        // Run sample scenarios

        // Transfer the source file from a public Azure Blob container to Data Lake Store.
        CloudBlockBlob blob = new CloudBlockBlob(new Uri("https://tomnew.blob.core.windows.net/adls-sample-data/SearchLog.tsv"));
        blob.DownloadToFile(localFolderPath + "SearchLog.tsv", FileMode.Create); // from WASB
        UploadFile(localFolderPath + "SearchLog.tsv", "/mytempdir/SearchLog.tsv"); // to ADLS
        WaitForNewline("Source data file prepared.", "Submitting a job.");

        // Submit the job
        string jobId = SubmitJobByPath(localFolderPath + "SampleUSQLScript.txt", "My First ADLA Job");
        WaitForNewline("Job submitted.", "Waiting for job completion.");

        // Wait for job completion
        WaitForJob(jobId);
        WaitForNewline("Job completed.", "Downloading job output.");


    }

7.Debug from the local and check the result from azure portal.

packages.config

<?xml version="1.0" encoding="utf-8"?>
<packages>
  <package id="Microsoft.Azure.KeyVault.Core" version="1.0.0" targetFramework="net46" />
  <package id="Microsoft.Azure.Management.DataLake.Analytics" version="3.0.0" targetFramework="net46" />
  <package id="Microsoft.Azure.Management.DataLake.Store" version="1.0.4" targetFramework="net46" />
  <package id="Microsoft.Azure.Management.DataLake.StoreUploader" version="1.0.1-preview" targetFramework="net46" />
  <package id="Microsoft.Data.Edm" version="5.8.2" targetFramework="net46" />
  <package id="Microsoft.Data.OData" version="5.8.2" targetFramework="net46" />
  <package id="Microsoft.Data.Services.Client" version="5.8.2" targetFramework="net46" />
  <package id="Microsoft.IdentityModel.Clients.ActiveDirectory" version="3.14.1" targetFramework="net46" />
  <package id="Microsoft.Rest.ClientRuntime" version="2.3.8" targetFramework="net46" />
  <package id="Microsoft.Rest.ClientRuntime.Azure" version="3.3.7" targetFramework="net46" />
  <package id="Newtonsoft.Json" version="6.0.8" targetFramework="net46" />
  <package id="System.ComponentModel.EventBasedAsync" version="4.0.11" targetFramework="net46" />
  <package id="System.Dynamic.Runtime" version="4.0.0" targetFramework="net46" />
  <package id="System.Linq.Queryable" version="4.0.0" targetFramework="net46" />
  <package id="System.Net.Requests" version="4.0.11" targetFramework="net46" />
  <package id="System.Spatial" version="5.8.2" targetFramework="net46" />
  <package id="WindowsAzure.Storage" version="8.1.4" targetFramework="net46" />
</packages>

Update:

If we want to use silently login, we could use following code to get tokenCreds var tokenCreds = AuthenticateSlientUser("https://management.core.windows.net/", tenantId, applicationId, secretKey)

public static TokenCredentials AuthenticateSlientUser(string resource,string tenantId, string appClientId, string secretKey)
        {
            var authContext = new AuthenticationContext("https://login.microsoftonline.com/" + tenantId);

            var tokenAuthResult = authContext.AcquireTokenAsync(resource, new ClientCredential(appClientId, secretKey)).Result;

            return new TokenCredentials(tokenAuthResult.AccessToken);
        }

Answer 2

The sample from msdn not having the AcquireTokenAsync method - Arron

Sometimes documentation lags behind. They probably updated the package but haven't updated the documentation yet. Use the Async method, you'll be fine.

You could just as easily use AcquireTokenAsync method. Call it like stated below. This will await the async method and return the result, even in synchronous methods.

var token = authContext.AcquireTokenAsync(parameters).Result;

For the parameters to be used, have a look at the documentation: Authentication​Context.​Acquire​Token​Async Method

You can find multiple examples of working code online. Here's one example.