0

I am currently using azure eventhub to send/receive data. I want to connect my IoT device to NoSQL database from Azure in a way that JSON message produced from IoT data be stored in NoSQL database(maybe cosmosDB).

I was trying to using function app (classic) to process data stream, but it seems not working.

So, the pipeline maybe need to be something like this:

IoT -> Event hub -> CosmosDB(MongoDB?) -> Azure databricks

I need to establish the first three steps and I preferred to use python language. Can you guys recommend any tools for this?

A short example code would be really appreciated.

Ivan Glasenberg
  • 29,865
  • 2
  • 44
  • 60
Brian Lee
  • 173
  • 3
  • 14

2 Answers2

0

If you are looking to stay using Python, not sure if Event Hub is the correct choice. A couple possible solutions would be IoT Device->IoT Hub->Service Bus->App Function->Cosmos DB->Azure Databricks. The function here could be written in Python, this may not be the most effective solution though as it various how many messages are being sent to the service bus and then can vary what the pricing tier is of the Function.

Another setup that might be more ideal would be IoT Device->IoT Hub->Stream Analytics->CosmosDb->Azure Databricks. This pattern wouldn't use Python but could use JavaScript or Steam Analytics Querying to lightly transform the data.

Now CosmosDb->Databricks can be it's own challenges to depending how the connection will be established, i.e. the connection string to CosmosDb. That information most likely should be stored in an Azure Key Vault which the integration between Databricks and Key Vault is still in public preview.

DreadedFrost
  • 2,602
  • 1
  • 11
  • 29
0

The Cosmos DB Team has created a reference application for this scenario that should be useful to you. The solution has an IOT Hub that is read by an Azure Function that writes to Cosmos DB. You can see the architecture diagram and get the source code for this application here, https://github.com/AzureCosmosDB/scenario-based-labs/blob/master/IoT/Demo%20step-by%20step%20-%20Cosmos%20DB%20scenario-based%20labs%20-%20IoT.md#solution-architecture.

This app is written in C# and uses the Cosmos Core (SQL) API. You can use the Python SDK if you want (https://github.com/Azure/azure-cosmos-python/) but not sure why you'd use the MongoDB API. If this is a new app then easier to use the Core API.

Hope this is helpful.

Mark Brown
  • 8,113
  • 2
  • 17
  • 21