how to speed up a splunk export?

Question

I am using the python 3 splunk API to export some massive logs. My code essentially follows the splunk API guidelines:

import splunklib.client as client
import splunklib.results as results
import pandas as pd

kwargs_export = {"earliest_time": "2019-08-19T12:00:00.000-00:00",
                 "latest_time": "2019-08-19T14:00:00.000-00:00",
                 "search_mode": "normal"}

exportsearch_results = service.jobs.export(mysearchquery, **kwargs_export)

reader = results.ResultsReader(exportsearch_results)    

df = pd.DataFrame(list(reader))

But this is extremely slow...

Ultimately I want to store the output of the search as a csv to disk. Is there any way to speed the export?

Thanks!

Answer 1

Check this as it works

kwargs_export = {"earliest_time": "-1d",
                 "latest_time": "now",
                 "search_mode": "normal"}

service = client.connect(**args)

job = service.jobs.create(query, **kwargs_export)

with open(filename, 'wb') as out_f:

    try:

        job_results = job.results(output_mode="csv", count=0)

        for result in job_results:

             out_f.write(result)

    except :

        print("Session timed out. Reauthenticating")