We are using EMR. Before submitting a spark job, we are trying to pull config parameters from AWS Param Store (we have written a java program) and create a file and use it in spark-submit.
We are using aws-java-sdk-ssm API to pull the parameters from param store.
When we try to run this program on the master node, it expect's a token (.aws/configure/credentials) to connect to aws ssm to pull the parameters.
We are able to access S3 buckets without any credentials, but not the param store.
We are spinning up the EMR cluster from concourse pipeline.
The below is the method that we have written to pull the parameters from the paramstore:
private static List<Parameter> getParametersFromAWSParamStore(String path, String awsRegion){
List<Parameter> paramList = new ArrayList<>();
AWSSimpleSystemsManagement client = AWSSimpleSystemsManagementClientBuilder.standard().withRegion(awsRegion).build();
GetParametersByPathRequest request = new GetParametersByPathRequest();
request.withRecursive(true);
request.withPath(path);
request.setWithDecryption(true);
GetParametersByPathResult result = null;
do {
result = client.getParametersByPath(request);
paramList.addAll(result.getParameters());
}while( result.getNextToken() !=null && !result.getNextToken().isEmpty());
return paramList;
}
