0

I am looking at this example to implement the data processing of incoming raw data for a sagemaker endpoint prior to model inference/scoring. This is all great but I have 2 questions:

  • How can one debug this (e.g can I invoke endpoint without it being exposed as restful API and then use Sagemaker debugger)
  • Sagemaker can be used "remotely" - e.g. via VSC. Can such a script be uploaded programatically?

Thanks.

cs0815
  • 16,751
  • 45
  • 136
  • 299

1 Answers1

1

Sagemaker Debugger is only to monitor the training jobs.

https://docs.aws.amazon.com/sagemaker/latest/dg/train-debugger.html

I dont think you can use it on Endpoints.

The script that you have provided is used both for training and inference. The container used by the estimator will take care of what functions to run. So it is not possible to debug the script directly. But what are you debugging in the code ? Training part or the inference part ?

While creating the estimator we need to give either the entry_point or the source directory. If you are using the "entry_point" then the value should be relative path to the file, if you are using "source_dir" then you should be able to give an S3 path. So before running the estimator, you can programmatically tar the files and upload it to S3 and then use the S3 path in the estimator.

CrzyFella
  • 191
  • 4
  • thanks. I am wondering how to debug the inference/scoring part. The sklearn_abalone_featurizer.py does the job of data preprocessing of data send to the endpoint before it passed to the model for prediction(s) .... – cs0815 Apr 29 '22 at 07:17
  • It is hard to debug the inference part without the endpoint. You can spin up the endpoint in your machine using local mode and then you can test the inference part. – CrzyFella Apr 29 '22 at 07:56
  • sorry no let us assume there is an endpoint. – cs0815 Apr 29 '22 at 07:58
  • If there is an endpoint then you can pass the data to the endpoint and see how it behaves. The endpoints requests are logged in the CloudWatch. – CrzyFella Apr 29 '22 at 08:10
  • yes but can you see how input are transformed? Maybe one can log stuff? – cs0815 Apr 29 '22 at 08:42
  • 1
    Yes. You use print statements but this involves creating the container every time there is a change in the script. So I would suggest to see what is the data that is coming to different functions and mimic those without endpoints. – CrzyFella Apr 29 '22 at 23:41