How to move processed file to another directory using U-SQL?

Question

I am writing a U-SQL query that extract information from a file (say query.txt) that is stored in a directory (say A). Now I want to move the file query.txt to another directory (say processed) after i am done processing the file query.txt and have outputted some resultset to other files in different directory. How I can move my source file(query.txt) to Processed directory after i am done with my U-SQL job?

Answer 1

You can't use U-SQL for File System operations on the Data Lake Store.

You can read the file and write the output into a processed directory, but that would just create a copy of that file in the new directory, the original file would not be deleted.

There is no way to move, delete, or copy using U-SQL operations. A good way is to use the ADLS SDK, a write a simple program that does it for you. The SDK provides functions for such operations.

Answer 2

U-SQL does not provide you with file system operations. I would look into one of the SDK ADLS commands (e.g., in the Azure Powershell) that offers the standard WebHDFS file system commands to interact with the file system and orchestrate the job and the file movement with either Powershell or Azure Data Factory.

Answer 3

Using PowerShell

# Log in to your Azure account
Login-AzureRmAccount

$DataLakeStoreAccount = "<yourAccountNameHere>";
$source = "/Before/query.txt";
$Destination = "/After/query.txt";

Move-AzureRmDataLakeStoreItem -AccountName $DataLakeStoreAccount -Path $source -Destination $Destination;