Python Iterate and Send Data Over Socket (Logstash/Elasticsearch)

Question

I am attempting to stream JSON data and feed each individual JSON event (delimited by '\n') into a TCP socket that communicates with LogStash (localhost 5000). However, I can only seem to populate the last json object in Elasticsearch via LogStash. I can validate the API call works and should contain about 70 individual events. Only the final one ends up in Elasticsearch via this LogStash entry point. I have also validated the stream and delimiter in the requests call is working and returns a single JSON event. Each iteration of iter_lines will print a single JSON object as expected. I want to send each json object over this socket individually, but it not working. Any ideas? Do I need to tear down and build up the socket for each json object?

HOST = 'localhost'
PORT = 5000

try:
  sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
except socket.error, msg:
  sys.stderr.write("[ERROR] %s\n" % msg[1])
  sys.exit(1)

try:
  sock.connect((HOST, PORT))
except socket.error, msg:
  sys.stderr.write("[ERROR] %s\n" % msg[1])
  sys.exit(2)

#make web request
AUTH = {
    "Key" : key,
    "Email" : email
}
URL = "https://LOGAPIENDPOINT/{}/logs/requests?start={}".format(zone, start)
counter=0
r = requests.get(URL, headers=AUTH, stream=True)
for d in r.iter_lines(delimiter="\n"):
    print d
    sock.send()

print "Closing Socket"
sock.close()
sys.exit(0)

score 0 · Answer 1 · answered May 24 '17 at 21:19

So turns out by delimiting on the "\n", I was getting rid of what LogStash needs to see to know a log event is terminated.

By adding:

r = requests.get(URL, headers=AUTH, stream=True)
for d in r.iter_lines(delimiter="\n"):
    d += "\n"
    sock.send(d)

Everything works! Logstash basically didn't know how to recognize one event ended and one event started.

Python Iterate and Send Data Over Socket (Logstash/Elasticsearch)

1 Answers1