Azure Eventhub Multiple Columns

Question

I am working through the Azure tutorials on streaming tweets through an eventhub into Databricks and through cognitive services to do sentiment analysis (link to that). That is working fine but I would like to do some analysis on the location of the tweet. The tutorial only sends the text of the tweet (I believe) to the eventhub. Notice the status.getText() in the Scala code below.

while (!finished) {
        val result = twitter.search(query)
        val statuses = result.getTweets()
        var lowestStatusId = Long.MaxValue
        for (status <- statuses.asScala) {
          if(!status.isRetweet()){
            sendEvent(status.getText(), 5000)
          }
          lowestStatusId = Math.min(status.getId(), lowestStatusId)
        }
        query.setMaxId(lowestStatusId - 1)
      }

sendEvent is here:

def sendEvent(message: String, delay: Long) = {
      sleep(delay)
      val messageData = EventData.create(message.getBytes("UTF-8"))
      eventHubClient.get().send(messageData)
      System.out.println("Sent event: " + message + "\n")
    }

Can I ALSO call geoLocation() and send that to the eventhub so that I can pull the geoLocation and/or the text for a given tweet from the eventhub later?

Mainly wondering if EventData.create() can somehow send two "columns" (properties) for the same "row" (single event).

score 0 · Accepted Answer · answered Jan 30 '20 at 16:52

The EventData object only transports binary data. What you put in there is completely up to you. Very often people will create a JSON structure that is being saved as a string and then converted to bytes. So instead of

sendEvent(status.getText(), 5000)

you could do something like (bit of pseudo code)

sendEvent('{"text": "' + status.getText() + '", "location":"' + yourLocation + '"}', 5000)

score 0 · Answer 2 · answered Jan 30 '20 at 19:55

0

EventData object supports bag of application properties that you can stuff small data for quick loookup on the receiver side depending on your app and business logic.

var eventData = new EventData(Encoding.UTF8.GetBytes("Hello EventHub!"));

eventData.Properties["ContosoEventType"] = "some value here";

answered Jan 30 '20 at 19:55

Serkant Karaca

1,896
9
8

Very interesting! How will I retrieve this when it is time to read it? – user12814878 Jan 31 '20 at 02:24
Property will be available in the received EventData.Properties dictionary. var propValue = eventData.Properties["ContosoEventType"]; – Serkant Karaca Jan 31 '20 at 02:56
Ok thanks. I marked the other answer as correct since that's what I used (and it worked) but this is another interesting approach. I'll definitely look into it. THANK YOU! – user12814878 Jan 31 '20 at 20:55

Azure Eventhub Multiple Columns

2 Answers2