Can I add DeviceID in path while storing data from Azure Stream Analytics to Blob storage

Question

I have Data incoming from Different devices to IoT hub from there using Stream Analytics to process it and store it in blob storage. I know we can add {date}{time} we add in the path according to needed format, in that path can we add deviceId too.

example : For 2018/10/30/01 ( Date/month/day/hour) Can add /deviceId in that path while storing to blob

This Device means 2018/10/30/01/device1 2018/10/30/01/device2 i want store each deviceId data into that file — Amjath Khan, Oct 30 '18 at 17:03
[Not yet](https://feedback.azure.com/forums/270577-stream-analytics/suggestions/17738683-allow-variable-names-in-output-paths) — Peter Bons, Oct 30 '18 at 18:25
It's not supported. As a workaround can be used a HttpTrigger function with an output blob binding. — Roman Kiss, Oct 30 '18 at 23:28

score 1 · Answer 1 · edited Jul 16 '19 at 22:35

1

I know we can add {date}{time} we add in the path according to needed format, in that path can we add deviceId too.'

As @Peter Bons mentioned in the comment, variable names in output is not supported so far.

As workaround, you could use Blob Trigger Azure Function. You need to pass the deviceId in the output columns then get it in the blob trigger function. Then use blob sdk to create /deviceId directory to copy blob into it and delete the previous blob.

edited Jul 16 '19 at 22:35

halfer

19,824
17
99
186

answered Nov 01 '18 at 08:21

Jay Gong

23,163
2
27
32

From i can do that by using blob SDK . But when i get the next message it needs to add to that blob . for example i have added month/day/deviceId/abc.txt when azure function triggered next time it will replace the contents of abc.txt, I cannot append to to it. is there a way to append to blob ? especially suppose multiple functions triggered at same time and all trying to append to same blob. – Amjath Khan Nov 02 '18 at 10:28
As you can see in https://azure.github.io/azure-storage-node/BlobService.html#appendBlockFromText__anchor If multiple writes are happening at same time then that will not sure will be appended. in Options we have to mention this absorbConditionalErrorsOnRetry bool Specifies whether to absorb the conditional error – Amjath Khan Nov 02 '18 at 13:08

score 1 · Accepted Answer · answered Nov 01 '18 at 12:20

The following is an example of workaround for your case. It's based on the using an azure function (HttpTrigger) for output ASA job to append a data to the specific blob storage in the push manner. Note, that the following workaround using the Max batch count for delivering events to the azure function value 1 (one telemetry data at the time).

ASA job query:

SELECT
  System.Timestamp as [time], * 
INTO outAF
FROM 
  iot TIMESTAMP BY time

Azure Function (HttpTrigger):

run.csx

#r "Newtonsoft.Json"
#r "Microsoft.WindowsAzure.Storage"

using System.Net;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Primitives;
using Microsoft.WindowsAzure.Storage.Blob;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;

public static async Task<IActionResult> Run(string body, CloudBlobContainer blobContainer, ILogger log)
{
    log.LogInformation($"{body}");

    var jtoken = JToken.Parse(body);
    var jobject = jtoken is JArray ? jtoken.SingleOrDefault<JToken>() : jtoken;
    if(jobject != null)
    {
        var jtext = jobject.ToString(Formatting.None);
        var data = JsonConvert.DeserializeAnonymousType(jtext, new {IoTHub = new { ConnectionDeviceId = ""}});        
        var blobName = $"{DateTime.UtcNow.ToString("yyyy/MM/dd/hh")}/{data.IoTHub.ConnectionDeviceId}";  
        var blob = blobContainer.GetAppendBlobReference(blobName);
        if(!await blob.ExistsAsync())
        {
            await blob.CreateOrReplaceAsync();
        }
        await blob.AppendTextAsync(jtext + "\r\n");
    }
return new NoContentResult();

}

function.json

    {
      "bindings": [
       {
           "authLevel": "function",
           "name": "body",
           "type": "httpTrigger",
           "direction": "in",
           "methods": [
             "get",
             "post"
             ]
      },
      {
          "name": "blobContainer",
          "type": "blob",
          "path": "myContainer",
          "connection": "mySTORAGE",
          "direction": "out"
      },
      {
          "name": "$return",
          "type": "http",
          "direction": "out"
      }
      ]
 }

Thank you . Is Azure functions scalable enough to when we are going to scale for 10000 devices getting messages and triggering this functions every second. do you have node js code for it ? — Amjath Khan, Nov 02 '18 at 10:43
As you can see in https://azure.github.io/azure-storage-node/BlobService.html#appendBlockFromText__anchor If multiple writes are happening at same time then that will not sure will be appended. in Options we have to mention this absorbConditionalErrorsOnRetry bool Specifies whether to absorb the conditional error — Amjath Khan, Nov 02 '18 at 11:42
Note that using a function might result in events received out of order by the destination. If your business logic depends on the order you might need extra logic to enforce that, or to resolve this situation (e.g. discarding delayed telemetry). — Devis L., Nov 06 '18 at 01:44

Can I add DeviceID in path while storing data from Azure Stream Analytics to Blob storage

2 Answers2