Have an AWS MediaConvert job template output to the same directory

Question

I'm trying to set up a job in Amazon's MediaConvert service which will take any video I upload to an S3 bucket and convert it to a streaming-friendly format. For the most part, this seems to be working - my app calls the API to run the job each time I upload a video.

However, for convenience (to make it easier to find the outputs for a specific file), I would like to put the output files into the same directory that the input file was found in.

Is there a way to set a job template's output to a path relative to the input? Alternatively, is it possible to override the output directory without having to define the entire job from scratch in the Java code?

This is how I'm setting the input directory in the Java API:

CreateJobRequest request = CreateJobRequest.builder()
    .role(roleName)
    .jobTemplate("Process uploaded videos") // Matches a custom template I made in AWS
    .settings(JobSettings.builder()
        .inputs(Input.builder()
            .fileInput("s3://" + mConfig.bucket + "/" + mFolder + filename)
            .build())
        .build())
    .build();

Updated version with output path specified:

CreateJobRequest request = CreateJobRequest.builder()
    .role(MEDIACONVERT_ROLE)
    .jobTemplate("Process uploaded videos") // Matches a custom template I made in AWS
    .settings(JobSettings.builder()
        .inputs(Input.builder()
            .fileInput("s3://" + mConfig.bucket + "/" + mFolder + filename)
            .build())
        .outputGroups(OutputGroup.builder()
            .name("Streaming video") // Matches the name of the output group in the template
            .outputGroupSettings(OutputGroupSettings.builder()
                .type(OutputGroupType.FILE_GROUP_SETTINGS)
                .fileGroupSettings(FileGroupSettings.builder()
                    .destination("s3://" + mConfig.bucket + "/" + mFolder) // Put output in the same folder as the input
                    .build())
                .build())
            .build())
        .build())
    .build();

JeffH-AWS · Accepted Answer · 2021-01-27T22:41:46.363

There is no native method in AWS Elemental MediaConvert for setting a relative path for the output based on the location of the input, however you are right in that you can override parameters from a template without needing to define the entire job from scratch.

For instance, here's an AWS CLI (boto3) JSON example of submitting job using a template that does not reference presets.

Notes: This template contains an input array object (notice there is not a FileURI parameter, you will add this on job create)

{
  "Queue": "arn:aws:mediaconvert:us-west-2:111122223333:queues/Default",
  "Name": "Test_MP4", 
  "Description":"Test MP4"
  "Settings": {
    "OutputGroups": [
      {
        "Name": "File Group",
        "Outputs": [
          {
            "ContainerSettings": {
              "Container": "MP4",
              "Mp4Settings": {
                "CslgAtom": "EXCLUDE",
                "FreeSpaceBox": "EXCLUDE",
                "MoovPlacement": "NORMAL"
              }
            },
            "VideoDescription": {
              "Width": 3840,
              "ScalingBehavior": "DEFAULT",
              "Height": 2160,
              "VideoPreprocessors": {
                "Deinterlacer": {
                  "Algorithm": "INTERPOLATE",
                  "Mode": "DEINTERLACE",
                  "Control": "NORMAL"
                }
              },
              "TimecodeInsertion": "DISABLED",
              "AntiAlias": "ENABLED",
              "Sharpness": 50,
              "CodecSettings": {
                "Codec": "H_265",
                "H265Settings": {
                  "InterlaceMode": "PROGRESSIVE",
                  "ParNumerator": 1,
                  "NumberReferenceFrames": 3,
                  "FramerateDenominator": 1001,
                  "GopClosedCadence": 1,
                  "AlternateTransferFunctionSei": "DISABLED",
                  "HrdBufferInitialFillPercentage": 90,
                  "GopSize": 48,
                  "Slices": 4,
                  "GopBReference": "ENABLED",
                  "HrdBufferSize": 20000000,
                  "SlowPal": "DISABLED",
                  "ParDenominator": 1,
                  "SpatialAdaptiveQuantization": "ENABLED",
                  "TemporalAdaptiveQuantization": "ENABLED",
                  "FlickerAdaptiveQuantization": "DISABLED",
                  "Bitrate": 10000000,
                  "FramerateControl": "SPECIFIED",
                  "RateControlMode": "CBR",
                  "CodecProfile": "MAIN_MAIN",
                  "Tiles": "ENABLED",
                  "Telecine": "NONE",
                  "FramerateNumerator": 24000,
                  "MinIInterval": 0,
                  "AdaptiveQuantization": "HIGH",
                  "CodecLevel": "LEVEL_5",
                  "SceneChangeDetect": "ENABLED",
                  "QualityTuningLevel": "MULTI_PASS_HQ",
                  "FramerateConversionAlgorithm": "DUPLICATE_DROP",
                  "UnregisteredSeiTimecode": "DISABLED",
                  "GopSizeUnits": "FRAMES",
                  "ParControl": "SPECIFIED",
                  "NumberBFramesBetweenReferenceFrames": 3,
                  "TemporalIds": "DISABLED",
                  "SampleAdaptiveOffsetFilterMode": "ADAPTIVE"
                }
              },
              "AfdSignaling": "NONE",
              "DropFrameTimecode": "ENABLED",
              "RespondToAfd": "NONE",
              "ColorMetadata": "INSERT"
            },
            "AudioDescriptions": [
              {
                "AudioTypeControl": "FOLLOW_INPUT",
                "CodecSettings": {
                  "Codec": "AAC",
                  "AacSettings": {
                    "AudioDescriptionBroadcasterMix": "NORMAL",
                    "Bitrate": 160000,
                    "RateControlMode": "CBR",
                    "CodecProfile": "LC",
                    "CodingMode": "CODING_MODE_2_0",
                    "RawFormat": "NONE",
                    "SampleRate": 48000,
                    "Specification": "MPEG4"
                  }
                },
                "LanguageCodeControl": "FOLLOW_INPUT",
                "AudioType": 0
              }
            ],
            "NameModifier": "_1"
          }
        ],
        "OutputGroupSettings": {
          "Type": "FILE_GROUP_SETTINGS",
          "FileGroupSettings": {
            "Destination": "s3://myawsbucket/out/"
          }
        }
      }
    ],
    "AdAvailOffset": 0,
    "Inputs": [
      {
        "AudioSelectors": {
          "Audio Selector 1": {
            "Offset": 0,
            "DefaultSelection": "DEFAULT",
            "ProgramSelection": 1
          }
        },
        "VideoSelector": {
          "ColorSpace": "FOLLOW"
        },
        "FilterEnable": "AUTO",
        "PsiControl": "USE_PSI",
        "FilterStrength": 0,
        "DeblockFilter": "DISABLED",
        "DenoiseFilter": "DISABLED",
        "TimecodeSource": "EMBEDDED"
      }
    ]
  }
}

The following JSON payload adds an input and a caption selector on input as well as a caption track to the output, and it changes the destination location for the output file.

Note: You will need to make sure to include you job template name as well as the role you want the service to assume (i.e. what you set up in Setting up using IAM)

{
  "Settings": {
    "OutputGroups": [
      {
        "Name": "File Group",
        "Outputs": [
          {
          "CaptionDescriptions": [
              {
                "DestinationSettings": {
                  "DestinationType": "EMBEDDED"
                },
                "CaptionSelectorName": "Captions Selector 1",
                "LanguageCode": "ENG"
              }
            ]
          }
        ],
        "OutputGroupSettings": {
          "Type": "FILE_GROUP_SETTINGS",
          "FileGroupSettings": {
            "Destination": "s3://myawsbucket/newfolder/out/"
          }
        }
      }
    ],
    "AdAvailOffset": 0,
    "Inputs": [
      {
        "AudioSelectors": {
          "Audio Selector 1": {
            "Offset": 0,
            "DefaultSelection": "DEFAULT",
            "ProgramSelection": 1
          }
        },
        "VideoSelector": {
          "ColorSpace": "FOLLOW"
        },
        "FilterEnable": "AUTO",
        "PsiControl": "USE_PSI",
        "FilterStrength": 0,
        "DeblockFilter": "DISABLED",
        "DenoiseFilter": "DISABLED",
        "TimecodeSource": "EMBEDDED",
        "FileInput": "s3://myawsbucket/input/test.mp4",
        "CaptionSelectors": {
          "Captions Selector 1": {
            "SourceSettings": {
              "SourceType": "SCC",
              "FileSourceSettings": {
                "SourceFile": "s3://myawsbucket/input/Captions/SCC/test.scc"
              }
            }
          }
        }
      }
    ]
  },
  "JobTemplate": "Test_MP4",
  "Role": "arn:aws:iam::111122223333:role/MediaConvertRole"
}

To only change the destination, you would use this JSON:

{
  "Settings": {
    "OutputGroups": [
      {
        "OutputGroupSettings": {
          "Type": "FILE_GROUP_SETTINGS",
          "FileGroupSettings": {
            "Destination": "s3://myawsbucket/newfolder/out/"
          }
        }
      }
    ],
    "Inputs": [
      {
        "FileInput": "s3://myawsbucket/input/test.mp4"
      }
    ]
  },
  "JobTemplate": "Test_MP4",
  "Role": "arn:aws:iam::111122223333:role/MediaConvertRole"
}

Nice. So if I understand correctly, the matching name "File Group" will cause the request to merge the per-job JSON into the template's definition? — John Brink, Jan 27 '21 at 16:03
It's the JobTemplate value that must match the job create JSON to the pre-existing template. The other parameters are then merged and override settings in the template. — JeffH-AWS, Jan 27 '21 at 17:18
I tried copying the OutputGroups -> OutputGroupSettings block out of your example, and now I'm getting an error about "/outputGroups/0/outputGroupSettings: Should have at most 2 properties". It doesn't look like any of the other parts of your sample JSON are essential to add to my code - am I missing something, or is something else to blame here? — John Brink, Jan 27 '21 at 19:30
You would want to use something like this: `{ "Settings": { "OutputGroups": [ { "OutputGroupSettings": { "Type": "FILE_GROUP_SETTINGS", "FileGroupSettings": { "Destination": "s3://myawsbucket/newfolder/out/" } } } ], "Inputs": [ { "FileInput": "s3://myawsbucket/input/test.mp4", } ] }, "JobTemplate": "Test_MP4", "Role": "arn:aws:iam::111122223333:role/MediaConvertRole" }` Alright that looks awful - adding that bit to the answer. — JeffH-AWS, Jan 27 '21 at 22:28
If you're still having trouble after trying the updated JSON format example, could you please log out your createJobRequest so I can take a look at the JSON you're sending? I just tested it on my end to make sure I didn't leave any typos. Other than the comma I accidentally left in my previous comment at the end of this line: `"FileInput": "s3://myawsbucket/input/test.mp4", ` — JeffH-AWS, Jan 27 '21 at 22:48
Unfortunately due to the way the AWS API 2.0 for Java works, I can't build a job from, or output its contents as JSON. The best I can do is a `.toString()`: ```CreateJobRequest(JobTemplate=Process uploaded videos, Role=arn:aws:iam::ID:role/service-role/MediaConvert_Default_Role, Settings=JobSettings(Inputs=[Input(FileInput=s3://BUCKET_NAME/PATH/file_example_MP4_1280_10MG.mp4)], OutputGroups=[OutputGroup(Name=Streaming video, OutputGroupSettings=OutputGroupSettings(FileGroupSettings=FileGroupSettings(Destination=s3://BUCKET_NAME/PATH/), Type=FILE_GROUP_SETTINGS))]))``` — John Brink, Jan 28 '21 at 19:52
The updated source code I used to generate the above has been added to my initial question above. — John Brink, Jan 28 '21 at 19:53
Well, I'm still struggling with this issue, but your answer seems to be correct as far as I can tell so I'll mark it. — John Brink, Feb 01 '21 at 20:59
I finally found out the cause of the issue - I checked the JSON for my template, and it had OutputGroupSettings.Type = "CMAF_GROUP_SETTINGS". I changed my request to match, and the job was started successfully. Thanks for your help. — John Brink, Feb 03 '21 at 15:58
Hey John sorry about leaving you hanging there I had some personal things come up that kept me away for a bit. I'm so glad you were able to get that up and running! — JeffH-AWS, Feb 04 '21 at 16:03

Have an AWS MediaConvert job template output to the same directory

1 Answers1