Debugging Sagemaker Pipeline with custom cloudwatch log messages

Question

I have scoured the docs and I cannot figure out how to create custom log messages that are more descriptive. We are using sagemaker to run pipelines and they are failing, but the logs I see in cloudwatch are too generic. Is there any way to just console log ("hello world") in a sagemaker step?

Answer 1

SageMaker Pipelines are used to launch specific steps. Steps such as TrainingStep, ProcessingStep, etc...

Each one of these steps would kick of a resource. If your step is a Processing Job then there would be a specific script and container that is run. If you want to add extra logging you can add print statements and/or extra logging to the script. All stdout and stderr will be sent to CloudWatch logs automatically.

While I am not sure what or where your error is coming from, it is important to triage the issue and confirm if the failure is coming from the Job as a result of the script/container or is the Job not even getting kicked off?