Receiving JSON (Cosmos DB Documents) as input from Azure Service Bus in Azure Functions

Question

I have an Azure Function (f1) take Cosmos DB Documents as input and I want to set up a path to redirect any Documents to an Azure Service Bus Queue in case of some kind of failure that might happen downstream (SQL statement to a SQL Server) possibly. I want to then run an Azure Function (f2) on a timer that will serve as "cleanup" for any of these redirected documents and attempt to do the same thing basically that f1 tried to do.

I have been able to send a single Document or multiple documents to a ServiceBus Queue but I could only successfully process input from a Queue in f2 when it was a single Document. Note that f2 is on a trigger and I am still not sure how to even read a queue from an Azure function? How would I accomplish iterating through the entire Queue of messages in 1 function? The bindings show that ServiceBus is not a valid "Input" for Azure Functions though it is valid as a trigger. https://learn.microsoft.com/en-us/azure/azure-functions/functions-triggers-bindings

What I am working with below so far:

f1 with a single output

#r "Microsoft.Azure.Documents.Client"
#r "Microsoft.ServiceBus"
using Microsoft.ServiceBus.Messaging;
using Microsoft.Azure.Documents;
using System.Collections.Generic;
using System;

public static void Run(IReadOnlyList<Document> inputFeed, TraceWriter log,
    out Document outputSbQueue)
{
  //f1 with a single output to a Queue
}

f1 with an ICollector output

#r "Microsoft.Azure.Documents.Client"
#r "Microsoft.ServiceBus"
using Microsoft.ServiceBus.Messaging;
using Microsoft.Azure.Documents;
using System.Collections.Generic;
using System;

public static void Run(IReadOnlyList<Document> inputFeed, TraceWriter log,
    ICollector<Document> outputSbQueue)
{
  //f1 with an ICollector output
}    

f2 with a single input from a ServiceBus trigger (works)

#r "Microsoft.Azure.Documents.Client"
#r "Newtonsoft.Json"
using Microsoft.Azure.Documents;
using System.Collections.Generic;
using System;
using System.Threading.Tasks;
using Newtonsoft.Json;
public static void Run(Document myQueueItem, TraceWriter log)
{
    //f2 with a working single input -- myQueueItem is an SB trigger
}

f2 with an ICollector or any kind of IEnumerable input (doesn't work...throws Exception about JSon serialization at runtime)

#r "Microsoft.Azure.Documents.Client"
#r "Newtonsoft.Json"
using Microsoft.Azure.Documents;
using System.Collections.Generic;
using System;
using System.Threading.Tasks;
using Newtonsoft.Json;

public static void Run(IReadOnlyList<Document> myQueueItem, TraceWriter log)
{
    //f2 with an input of more than one document
}

throws this exception:

2018-05-15T20:48:21.535 [Error] Exception while executing function: Functions.ServiceBusQueueTriggerCSharp1. Microsoft.Azure.WebJobs.Host: Exception binding parameter 'myQueueItem'. Microsoft.Azure.WebJobs.ServiceBus: Binding parameters to complex objects (such as 'ICollector1') uses Json.NET serialization. 1. Bind the parameter type as 'string' instead of 'ICollector1' to get the raw values and avoid JSON deserialization, or 2. Change the queue payload to be valid json. The JSON parser failed: Cannot deserialize the current JSON array (e.g. [1,2,3]) into type 'Microsoft.Azure.WebJobs.ICollector`1[Microsoft.Azure.Documents.Document]' because the type requires a JSON object (e.g. {"name":"value"}) to deserialize correctly. To fix this error either change the JSON to a JSON object (e.g. {"name":"value"}) or change the deserialized type to an array or a type that implements a collection interface (e.g. ICollection, IList) like List that can be deserialized from a JSON array. JsonArrayAttribute can also be added to the type to force it to deserialize from a JSON array. Path '', line 1, position 1.

I am a complete newb to dealing with JSON, serialization and deserialization. I just want to parse it, read stuff from it and get a SQL statement ready to send downstream.

Answer 1

At the moment, Service Bus Trigger doesn't support batches (multiple items as input). See Service Bus Batch Trigger github issue.

If I understand your scenario correctly, you don't necessarily have to accept a batch in f2 function, nor do you need to "iterate through the entire Queue of messages".

Iteration will be handled by Functions. Your f2 function will get called as many times as many messages f1 sends to the Service Bus queue. If you "clean up" the documents one-by-one as you get them, you will eventually clean up all of the documents.