2

I'd like to set the prefix based on some of the data coming from event hub.

My data is something like: {"id":"1234",...}

I'd like to write a blob prefix that is something like: foo/{id}/guid....

Ultimately I'd like to have one blob for each id. This will help how it gets consumed downstream by a couple of things.

What I don't see is a way to create prefixes that aren't related to date and time. In theory I can write another job to pull from blobs and break it up after the stream analytics step. However, it feels like SA should allow me to break it up immediately.

Any ideas?

Chris
  • 169
  • 1
  • 2
  • 14

2 Answers2

2

{date} , {time} and {partition} are the only ones supported in blob output prefix. {partition} is a number.

Using a column value in blob prefix is currently not supported.

If you have a limited number of such {id}s then you could workaround by writing multiple "select --" statements with different filters writing to different outputs and hardcode the prefix in the output. Otherwise it is not possible with just ASA.

Vignesh Chandramohan
  • 1,306
  • 10
  • 15
  • In addition to the hard "no", Alexander offered some good practical workarounds when I had a similar question: http://stackoverflow.com/questions/41233928/stream-analytics-dynamic-output-path-based-on-message-payload – Pete M Jan 11 '17 at 14:19
1

It should be noted that now you actually can do this. Not sure when it was implemented but you can now use a single property from your message as a custom partition key and the syntax is exactly as the OP has asked for: foo/{id}/something/else

More details are documented here: https://learn.microsoft.com/en-us/azure/stream-analytics/stream-analytics-custom-path-patterns-blob-storage-output

Key points:

  • Only one custom property allowed
  • Must be a direct reference to an existing message property (i.e. no concatenations like {prop1+prop2})
  • If the custom property results in too many partitions (more than 8,000) then an arbitrary number of blobs may be created for the same parition
ChrisWue
  • 18,612
  • 4
  • 58
  • 83