2

For development, I'm using the local Spring Cloud Dataflow server on my Mac, though we plan to deploy to a Kubernetes cluster for integration testing and production. The SCDF docs say you can use environment variables to configure various things, like database configuration. I'd like my registered app to use these env variables, but they don't seem to be able to see them. That is, I start the SCDF server by running its jar from a terminal window, which can see a set of environment variables. I then configure a stream using some Spring Cloud stream starter apps and one custom Spring Boot app. I have the custom app logging System.getenv() and it's not showing the env variables I need. I set them in my ~/.bashrc file, which I also source from ~/.bash_profile. That works for my terminal windows and most other things that need environment, but not here. Where should I be defining them?

To the points in the first answer and comments, they sound good, but nothing works for me. I have an SQS Source that get's its connection via:

return AmazonSQSAsyncClientBuilder.standard()
    .withRegion(Regions.US_WEST_2.getName()))
    .build();

When I deploy to a Minikube environment, I edit the sqs app's deployment and set the AWS credentials in the env section. Then it works. For a local deployment, I've now tried:

stream deploy --name greg1 --properties "deployer.sqs.AWS_ACCESS_KEY_ID=<id>,deployer.sqs.AWS_SECRET_ACCESS_KEY=<secret>"

stream deploy --name greg1 --properties "deployer.sqs.aws_access_key_id=<id>,deployer.sqs.aws_secret_access_key=<secret>"

stream deploy --name greg1 --properties "app.sqs.AWS_ACCESS_KEY_ID=<id>,app.sqs.AWS_SECRET_ACCESS_KEY=<secret>"

stream deploy --name greg1 --properties "app.sqs.aws_access_key_id=<id>,app.sqs.aws_secret_access_key=<secret>"

All fail with the error message I get when credentials are wrong, which is, "The specified queue does not exist for this wsdl version." I've read the links, and don't really see anything else to try. Where am I going wrong?

Greg Charles
  • 1,880
  • 4
  • 20
  • 39

1 Answers1

3

You can pass environment variables to the apps that are deployed via SCDF using application properties or deployment properties. Check the docs for a description of each type.

For example:

dataflow:> stream deploy --name ticktock --properties "deployer.time.local.javaOpts=-Xmx2048m -Dtest=foo"
Vinicius Carvalho
  • 3,994
  • 4
  • 23
  • 29
  • That's not exactly what I'm looking for. Deployer (or App) properties may be available to the applications, but not as environment variables. In some cases, that's where they need to be. The Spring Framework, in particular, looks for things like DB or AWS credentials in well-known environment variables. When we deploy to Kubernetes, we can configure these into the Pod, or change them in the Deployment. For the local SCDF I'm using for dev work, I'm stuck. – Greg Charles Jan 10 '18 at 18:00
  • Couple of things, Spring DOES not look for things in environment variables, it looks on properties sources which get merged, take a look into the boot documentation that explains how it works. What makes possible for us to configure boot apps is this flexibility. So it won't look for DB creds on env vars, it will look for it on env vars, arguments, jndi and there's a resolution for all that. We pass those as arguments to the boot app that is started (application properties) and deployment properties are to configured how to pass deployer specific configs, memory in local is a sample – Vinicius Carvalho Jan 10 '18 at 19:30
  • To pass property overrides (e.g., jdbc or aws creds), you'd supply them via the DSL at "each app level" - we have a few [samples](https://docs.spring.io/spring-cloud-dataflow-samples/docs/current/reference/htmlsingle/#_using_the_local_server) on this subject. – Sabby Anandan Jan 11 '18 at 14:37
  • If the intent is to do runtime env-var override, you'd that via `deployer. ..` as recommended previously. There's also global override option, which will make the application properties available to all the deployed apps - see [here](https://docs.spring.io/spring-cloud-dataflow/docs/1.2.3.RELEASE/reference/htmlsingle/#spring-cloud-dataflow-global-properties). – Sabby Anandan Jan 11 '18 at 14:40
  • OK, it took some trial and error, but Vinicius was exactly correct about what was needed. The form for specifying AWS credentials as Java system properties is just slightly different from supplying them from environment variables. This finally worked for me: stream deploy greg1 --properties "deployer.sqs.local.javaOpts=-Daws.accessKeyId= -Daws.secretKey=" – Greg Charles Jan 16 '18 at 01:50