8

The Problem

Upon a trigger, generate 3 XML files and once complete, ftp them to a site.

The Current Approach

I have a HTTP Trigger Azure function that when run will construct 3 XML files and save these to an Azure Storage Blob container. Because of the multiple outputs, and the need to control the output path/filenames, I use the imperative binding approach and use the IBinder outputBinder in my function. This all works just fine. An example of the output path in the blob storage is export/2017-03-22/file-2017-03-22T12.03.02.54.xml. The file is in a folder with the date, and each filename has the time stamp to ensure uniqueness.

When all 3 files are generated, I want to trigger another function that will sFTP these to a site. Now I initially thought that I should use a blob trigger, but I couldn't figure how how to trigger on inputs that whose filenames and paths were dynamic. I coudldn't find such an example in the blob trigger documentation.

So then I thought I could have my HTTP Trigger output to a declarative binding and also output the XML files into an outgoing container in my blob storage which my blob trigger could be looking at. This also works however because my function is on the consumption plan, there can be up to a 10-minute day in processing new blobs.

So the documented alternative is to use a queue trigger. I can output to my queue and have the queue trigger just fine, but how do I also pass the 3 XML streams to my QueueTrigger function?

I suppose as a fall back, I can post an object that can contain the Azure Storage paths of the constructed XMLs and then use the Storage SDK to fetch the streams and use that to post to the FTP, but would it be more efficient to also pass those Storage Blob streams as an input to my QueueTrigger?

Community
  • 1
  • 1
flyte
  • 1,242
  • 11
  • 18

1 Answers1

8

I think your approach with Queue Trigger makes sense. I would construct a message like this

public class QueueItem
{
    public string FirstBlobPath { get; set; }
    public string SecondBlobPath { get; set; }
    public string ThirdBlobPath { get; set; }
}

and then use declarative binding in the queue processing function, something like 

{
  "bindings": [
    {
      "type": "queueTrigger",
      "name": "item",
      "direction": "in",
      "queueName": "myqueue",
      "connection":"...",    
    },
    {
      "type": "blob",
      "name": "file1",
      "path": "mycontainer/{FirstBlobPath}",
      "connection": "...",
      "direction": "in"
    },
    {
      "type": "blob",
      "name": "file2",
      "path": "mycontainer/{SecondBlobPath}",
      "connection": "...",
      "direction": "in"
    },
    {
      "type": "blob",
      "name": "file3",
      "path": "mycontainer/{ThirdBlobPath}",
      "connection": "...",
      "direction": "in"
    }
  ],
  "disabled": false
}

and the function

public static void Run(QueueItem item, Stream file1, Stream file2, Stream file3)
{
}
Mikhail Shilkov
  • 34,128
  • 3
  • 68
  • 107
  • When you have `{FirstBlobPath}` in the blob input, is it automatically referring to the path contained within the QueueItem ? – flyte Mar 24 '17 at 15:37
  • This didn't work. No matter what I tried, I couldn't get the streams. The error output was `QueueTrigger: Microsoft.Azure.WebJobs.Host: Error indexing method 'Functions.TransmitQueueTrigger'. Microsoft.Azure.WebJobs.Host: Blob identifiers must be in the format 'container/blob'` upon startup. How is it possible to obtain streams to the imperative inputs to this trigger? – flyte Mar 24 '17 at 15:52
  • @flyte Indeed, forgot the container name in binding path. See updated version. And yes, it is referring to property of QueueItem. – Mikhail Shilkov Mar 24 '17 at 16:03
  • hrm.. I think we're getting closer. My container has a subfolder for the current day, and that seems to be causing problems? `A ScriptHost error has occurred Exception while executing function: Functions.TransmitQueueTrigger. Microsoft.Azure.WebJobs.Host: Exception binding parameter 'file1'. Microsoft.Azure.WebJobs.Host: Invalid container name: export/2017-03-24.` Happens even with or without the trailing slash..? – flyte Mar 24 '17 at 18:04
  • your answer works if the container provided doesn't contain a subfolder. Incredible! Is this technique documented anywhere? That is, being able to drill into the provided queue item to be used in an input binding? – flyte Mar 24 '17 at 18:30
  • 1
    Good question about subfolders - never tried dynamic folders in the path. Let me know if you figure out a way to make it work with subfolders. See Parameter binding at https://learn.microsoft.com/en-us/azure/azure-functions/functions-triggers-bindings for docs – Mikhail Shilkov Mar 24 '17 at 18:56
  • For subfolders you can use containername/{year}/{name} in the configuration of the binding. – GitteTitter May 12 '17 at 06:53