0

I use spring cloud dataflow deployed to pivotal cloud foundry, to run spring batch jobs as spring cloud tasks, and the jobs require aws credentials to access an s3 bucket.

I've tried passing the aws credentials as task properties, but the credentials are showing up in the task's log files as arguments or properties. (https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#spring-cloud-dataflow-global-properties)

For now, I am manually setting the credentials as env variables in pcf after each deployment, but I'm trying to automate this. The tasks aren't deployed until the tasks are actually launched, so on a deployment I have to launch the task, then wait for it to fail due to missing credentials, then set the env variable credentials with the cf cli. How so I provide these credentials, without them showing in the pcf app's logs?

I've also explored using vault and spring cloud config, but again, I would need to pass credentials to the task to access spring cloud config.

Thanks!

Rick Suggs
  • 1,582
  • 1
  • 15
  • 30

3 Answers3

1

Here's a Task/Batch-Job example.

This App uses spring-cloud-starter-aws. And this starter already provides the Boot autoconfiguration and the ability to override AWS creds as Boot properties.

You'd override the properties while launching from SCDF like:

task launch --name S3LoaderJob --arguments "--cloud.aws.credentials.accessKey= --cloud.aws.credentials.secretKey= --cloud.aws.region.static= --cloud.aws.region.auto=false"

You can decide to control the log-level of the Task, so it doesn't log them in plain text.

Sabby Anandan
  • 5,636
  • 2
  • 12
  • 21
1

Secure credentials for tasks should be configured either via environment variables in your task definition or by using something like Spring Cloud Config Server to provide them (and store them encrypted at rest). Spring Cloud Task stores all command line arguments in the database in clear text which is why they should not be passed that way.

Michael Minella
  • 20,843
  • 4
  • 55
  • 67
  • I've explored using spring cloud config server, but then I would need to pass the credentials to that, defeating the purpose. Would you mind pointing out where to include environment variables in my task definitions. Would it be something like `task create mytask --definition "ingest-from-s3 --cloud.aws.credentials.accessKey="foo"` – Rick Suggs May 25 '18 at 15:42
  • What do you mean by "defeating the purpose"? And yes, that's what your DSL would look like (although i don't believe the quotes are required around foo). – Michael Minella May 25 '18 at 15:57
  • By "defeating the purpose" I mean that if I pass spring cloud config server credentials as arguments, I am still passing credentials that would appear in the logs, just different credentials. – Rick Suggs May 25 '18 at 16:01
  • No they won't appear in the logs. Only command line args should appear in the db and in the logs. – Michael Minella May 25 '18 at 16:02
1

After considering the approaches included in the provided answers, I continued testing and researching and concluded that the best approach is to use a Cloud Foundry "User Provided Service" to supply AWS credentials to the task.

https://docs.cloudfoundry.org/devguide/services/user-provided.html

Spring Boot auto-processes the VCAP_SERVICES environment variable included in each app's container.

http://engineering.pivotal.io/post/spring-boot-injecting-credentials/

I then used properties placeholders in the application-cloud.properties to map the processed properties into spring-cloud-aws properties:

cloud.aws.credentials.accessKey=${vcap.services.aws-s3.credentials.aws_access_key_id}
cloud.aws.credentials.secretKey=${vcap.services.aws-s3.credentials.aws_secret_access_key}
Rick Suggs
  • 1,582
  • 1
  • 15
  • 30