Is there a way to separate AWS Batch with Fargate logs when running different jobs?

Question

I would like to find the success and fail rate of the tasks that run on my Batch with Fargate compute environment, but can't seem to find a way to do that. I see documentation around sending batch logs to a single custom log group(https://aws.amazon.com/blogs/compute/custom-logging-with-aws-batch/) but nothing that would let me separate out my jobs into separate log groups or otherwise differentiate them. I'd also be open to using workarounds.

My Compute Environment definition looks like follows(Some sensitive items removed):

RequestProcessingComputeEnvironment:
    Type: AWS::Batch::ComputeEnvironment
    Version: 1.0
    Properties:
      ComputeEnvironmentName: !Sub ${AWS::StackName}-request-processing-compute-environment
      ComputeResources:
        MaxvCpus: 64
        SecurityGroupIds:
        - !GetAtt BatchVpcSecurityGroup.GroupId
        Subnets: !Ref VpcSubnets
        Type: FARGATE
      ServiceRole: !Sub arn:aws:iam::${AWS::AccountId}:role/${Namespace}${Component}-${SecretEnvironment}-batch-service-role
      State: ENABLED
      Type: MANAGED

  RequestProcessingJobQueue:
    Type: AWS::Batch::JobQueue
    Properties:
      ComputeEnvironmentOrder:
      - ComputeEnvironment: !Sub arn:aws:batch:${AWS::Region}:${AWS::AccountId}:compute-environment/${AWS::StackName}-request-processing-compute-environment
        Order: 1
      JobQueueName: !Sub ${AWS::StackName}-request-processing-job-queue
      Priority: 10
      State: ENABLED
    DependsOn:
      - RequestProcessingComputeEnvironment

  RequestProcessingJobDefinition:
    Type: AWS::Batch::JobDefinition
    Properties:
      ContainerProperties:
        Command:
          - "ruby"
          - "./batch_entrypoint/batch_entrypoint.rb"
          - #other refs
        Environment:
          # Values
        ExecutionRoleArn: !Sub ExecutionRoleArn
        FargatePlatformConfiguration:
          PlatformVersion: LATEST
        Image: !Sub ${LatestDockerUri}
        JobRoleArn: !Sub JobRoleArn
        NetworkConfiguration:
          AssignPublicIp: ENABLED
        ResourceRequirements:
          - Type: MEMORY
            Value: 1024
          - Type: VCPU
            Value: 0.5
      JobDefinitionName: !Sub JobDefinitionName
      PlatformCapabilities:
        - FARGATE
      PropagateTags: true
      RetryStrategy:
        Attempts: 3
      Type: container

And I am sending jobs in the step function definition like follows (There are about 5 different tasks I am sending to batch with fargate):

"Some Task Job": {
              "Type": "Task",
              "Resource": "arn:aws:states:::batch:submitJob.sync",
              "ResultPath": "$.batch_output",
              "Parameters": {
                "JobName": "SomeTaskJob",
                "JobQueue": "prefix-api-request-processing-job-queue",
                "JobDefinition": "prefix-api-request-processing-job-definition",
                "Parameters": {
                  "file_path": "${FilePath}",
                  "method_name": "process",
                  "kind": "None",
                  "domain_name.$": "$.domain_name",
                  "client_id.$": "$.client_id",
                  "request.$": "States.JsonToString($)"
                }
              },

Is there any way to differentiate these tasks so I can monitor each of their performance?

Is there a way to separate AWS Batch with Fargate logs when running different jobs?

0 Answers0