0

I have an app that I wrote for Splunk that has dependencies on boto3 and pyOpenSSL libraries. I haven't found a good way to get app dependencies into the apps bin folder other than drag/drop, which isn't working for boto3 and pyOpenSSL.

To this point in time, every time we needed to make a python module available to a single app in Splunk, we would drag and drop the python modules into $SPLUNK_HOME/etc/apps/APP_NAME/bin/MODULE . This has worked until we needed the pyOpenSSL and boto3 libraries which have lots of cryptography and single script dependencies that don't come over correctly.

What I've tried:

1| python3 -m venv $SPLUNK_HOME/etc/apps/APP_NAME/
2| python3 -m pip install (pyOpenSSL, boxsdk, pyJWT, boto3) < base dependencies
3| move $SPLUNK_HOME/etc/apps/APP_NAME/lib/python3.7/site-packages/ > $/SPLUNK_HOME/etc/apps/APP_NAME/bin
4| Put all my app scripts in $/SPLUNK_HOME/etc/apps/APP_NAME/bin alongside all the modules I just installed to that folder using venv
5| Start Splunk
6| search | search_command arg=0

At this point, Splunk tells me that the enum34, ipaddress, chainmap, cryptography (_constant_time module buried in here somewhere doesn't exist where it should) modules don't exist.

I then shut down Splunk, redid steps 1-6 but also installing all those missing modules on step 2. The error I'm getting now is this:

External search command 'boxfiles' returned error code 1. First 1000 (of 1456) bytes of script output: "No module named constant_time ERROR "Error 'No module named constant_time'. Traceback (most recent call last): File ""/Applications/Splunk/etc/apps/TA-box-connector/bin/box_connector/init.py"", line 3, in from box_connector import BoxConnector File ""/Applications/Splunk/etc/apps/TA-box-connector/bin/box_connector/box_connector.py"", line 10, in from OpenSSL import crypto File ""/Applications/Splunk/etc/apps/TA-box-connector/bin/OpenSSL/init.py"", line 8, in from OpenSSL import crypto, SSL File ""/Applications/Splunk/etc/apps/TA-box-connector/bin/OpenSSL/crypto.py"", line 12, in from cryptography import x509 File ""/Applications/Splunk/etc/apps/TA-box-connector/bin/cryptography/x509/init.py"", line 8, in from cryptography.x509.base import ( File ""/Applications/Splunk/etc/apps/TA-box-connector/bin/cryptography/x509/base.py"", line 18, in from cryptography.x509.extensions import Exte".

I'd like to solve this error, but I've been working through this dependency issue for some time now, so if there's a better solution to getting these packages on here, I would love to hear about it.

Cdhippen
  • 615
  • 1
  • 10
  • 32

1 Answers1

1

You can use the following commands to install the dependencies into the appropriate directory. Make sure $SPLUNK_HOME is set properly, and replace APP_NAME with your new app's name.

pip install pyOpenSSL -t $SPLUNK_HOME/etc/apps/APP_NAME/bin
pip install boxsdk -t $SPLUNK_HOME/etc/apps/APP_NAME/bin
pip install pyJWT -t $SPLUNK_HOME/etc/apps/APP_NAME/bin
pip install boto3 -t $SPLUNK_HOME/etc/apps/APP_NAME/bin

You may also need to look at following what the Splunk Add-On for AWS does, and deliberately add the required module's to Python's path. (Splunk_TA_aws/bin/aws_bootstrap_env.py).

'''
Add common libs to sys.path
'''


import os
import os.path
import re
import sys
import logging


def setup_python_path():
    # Exclude folder beneath other apps, Fix bug for rest_handler.py
    ta_name = os.path.basename(os.path.dirname(os.path.dirname(__file__)))
    pattern = re.compile(r"[\\/]etc[\\/]apps[\\/][^\\/]+[\\/]bin[\\/]?$")
    new_paths = [path for path in sys.path if not pattern.search(path) or ta_name in path]
    new_paths.insert(0, os.path.dirname(__file__))
    sys.path = new_paths

    bindir = os.path.dirname(os.path.abspath(__file__))
    # We sort the precedence in a decending order since sys.path.insert(0, ...)
    # do the reversing.
    # Insert library folder
    sharedpath = os.path.join(bindir, '3rdparty', 'python3')
    sys.path.insert(0, sharedpath)


# preventing splunklib initialize an unexpected root handler
def setup_null_handler():
    logging.root.addHandler(logging.NullHandler())


def run_module(name):
    instance = __import__(name, fromlist=['main'])
    instance.main()

setup_python_path()
setup_null_handler()

References

Simon Duff
  • 2,631
  • 2
  • 7
  • 15
  • Also tried using the same method to install the missing dependencies, and I still get stuck on "_constant_time" – Cdhippen Jan 27 '20 at 17:52
  • I tried the python path addition via this command: `bin/splunk cmd python3 etc/apps/APP_NAME/bin/aws_bootstrap_env.py`, it failed because of the enum34 library: https://github.com/iterative/dvc/issues/1582 so I uninstalled the enum34 library, ran it again, it was successful, so I loaded Splunk, it said I was missing enum, so I installed it again, and now I'm back to the original "_constant_time" exception – Cdhippen Jan 27 '20 at 21:44
  • My apologies, I didn't mean to run that command. I meant that you should add something like that code to your app. That code will add all the libraries installed in your app to python's search path, which should then be usable in the app – Simon Duff Jan 28 '20 at 01:10
  • I added that code to my app and it ran successfully, that's what I was saying. It didn't work. The code ran, but I'm still missing dependencies. – Cdhippen Jan 28 '20 at 18:13
  • I've built an example that seems to work fine for me using that `bootstrap_env.py` approach. https://github.com/sduff/splunk_app_with_libs code and instructions supplied – Simon Duff Jan 29 '20 at 10:46
  • I took your app skeleton and refactored my scripts a bit, and then when I went to pip install the libraries needed I noticed this: WARNING: Target directory /Applications/Splunk/etc/apps/TA_box_connector/bin/bin already exists. I realized that when I was installing these libraries, the /bin/bin folder wasn't getting added to after the first install, so I backed up the bin folder after each install and moved the old bin items into the new one, and now it's working! I appreciate all your help. – Cdhippen Jan 30 '20 at 02:08