0

Currently i'm storing my data in 4 diffrent file (that handle data from specified day,month,week,year). The day size of file is 1440 (that handle data which is send reads every minute and the date of reads is also stored per line) In this way if i for example need data from 1 May To 3 May i just download data from weekend file and filter downloaded records.

Is there any better way to fast accessing to files (or just download part of file or download splitted one file from specified files). I just thinking about u-sql executing beforce downloading but it take too long until u-sql end his job.

dawcza94
  • 327
  • 2
  • 10
  • Which tool or SDK are you currently using to upload or download your data? Are you trying to upload to ADLS or download from ADLS? What is the typical size of the files, and how many files are there in total? You can use Azure PowerShell, Azure CLI 2.0, .NET SDK, or Python SDK to download and upload a lot of data with high throughput using their build-in functionality, or any other option if you want to handle the parallelization manually. If you provide some details about your scenario, it would help StackOverflow users give a more precise answer. – Matt H May 31 '17 at 18:22
  • I'm storing data by date in separate file for each device. For example i have data from 31 May 2017 in path 2017\5\31\device1.csv , 2017\5\31\device2.csv and now what i currently want have in possible fast way in .NetSdk to get data from date 1 January 2017 to 31 May 2017, which i need to perform some actions on my asp.net application. – dawcza94 May 31 '17 at 19:38
  • If you want to use the Data Lake Store .NET SDK to download the data, then you could use either the `DownloadFile` or `DownloadFolder` methods to download data from Data Lake Store to the local machine where your application is running. Alternatively, you could use the `Open` method to read the data to memory, but depending on the size of the file, this may be slower. [Here's a useful article to get started with the .NET SDK.](https://learn.microsoft.com/en-us/azure/data-lake-store/data-lake-store-get-started-net-sdk) – Matt H May 31 '17 at 20:18

0 Answers0