Scala: How to load environment variables into Application.conf using scala?

Question

I have a ConfigLoader which I'm pretty sure works fine. I suspect I am not using application.conf file correctly or perhaps my directory containing the prd.env and dev.env is not in the correct place.

What I expect to happen:

I when I enter sbt run the ConfigLoader reads the application.conf file, sees there are variables within that file. Then checks if prd.env or dev.env depending on which environment it is in and then finally loads the variables. I want this so I can have a different database for dev and prd.

I have not idea how it would find the Meta directory which contains the prd.env and dev.env (see image).
I have no idea how it would be able to tell whether if the environment is in dev or prd.
The goal is to now look towards deploying this app, which is why I need these environment variables to work.

I really appreciate all the help but please try and detail your answers because I am really stuck and short answers often assume I know more than I do. Thanks :)

Tech stack incase relevant:

HTTP4S,
CATS,
Doobie,
PostgreSQL

application.conf file below:

appone {
    environment = ${ENV}

    server-config {
      url = ${?BASE_URL}
      host = "0.0.0.0"
      port = 8080
    }

    db-config {
        driver = "org.postgresql.Driver"
        url = ${?DATABASE_URL}
        user = ${?APPONE_POSTGRES_USER}
        password = ${?PASSWORD}
        connection-threads = 4
    }
}

Meta/dev.env file (I also have a Meta/prd.env but have shown it here due to contents)

ENV=dev

BASE_URL=http://localhost:8080/
DATABASE_URL=jdbc:postgresql://localhost:5400/bookswapdb
APPONE_POSTGRES_USER=su
PASSWORD=password

LoadConfig file below:

package com.fullstackryan.appone.config

import cats.ApplicativeError
import cats.implicits._
import pureconfig.error.ConfigReaderException
import pureconfig.{ConfigReader, ConfigSource, Derivation}
import shapeless.the

trait LoadConfig[F[_], TConfig] {
  def load: F[TConfig]
}

object LoadConfig {
  def load[F[_], TConfig](implicit loadConfig: LoadConfig[F, TConfig]): F[TConfig] =
    the[LoadConfig[F, TConfig]].load

  def apply[F[_], TConfig](
                            implicit reader: Derivation[ConfigReader[TConfig]], ae: ApplicativeError[F, Throwable]
                          ): LoadConfig[F, TConfig] =
    new LoadConfig[F, TConfig] {
      def load: F[TConfig] = ApplicativeError[F, Throwable].fromEither {
        ConfigSource.default
          .at("appone")
          .load[TConfig]
          .leftMap(ConfigReaderException(_))
      }
    }

}

error

pureconfig.error.ConfigReaderException: Cannot convert configuration to a scala.runtime.Nothing$. Failures are:
  at 'appone.db-config':
    - (application.conf @ jar:file:/Users/ryanmcavoy/fullStackRyan/appone/target/bg-jobs/sbt_3cc4b1f5/job-11/target/419ddc2c/5befcb57/appone_2.13-0.0.1-SNAPSHOT.jar!/application.conf: 10) Key not found: 'url'.
    - (application.conf @ jar:file:/Users/ryanmcavoy/fullStackRyan/appone/target/bg-jobs/sbt_3cc4b1f5/job-11/target/419ddc2c/5befcb57/appone_2.13-0.0.1-SNAPSHOT.jar!/application.conf: 10) Key not found: 'username'.
    - (application.conf @ jar:file:/Users/ryanmcavoy/fullStackRyan/appone/target/bg-jobs/sbt_3cc4b1f5/job-11/target/419ddc2c/5befcb57/appone_2.13-0.0.1-SNAPSHOT.jar!/application.conf: 10) Key not found: 'pool-size'.

        at com.fullstackryan.appone.config.LoadConfig$$anon$1.$anonfun$load$1(LoadConfig.scala:25)
        at cats.syntax.EitherOps$.leftMap$extension(either.scala:172)
        at com.fullstackryan.appone.config.LoadConfig$$anon$1.load(LoadConfig.scala:25)
        at com.fullstackryan.appone.server.ApponeServer$.$anonfun$stream$1(ApponeServer.scala:32)
        at com.fullstackryan.appone.server.ApponeServer$.$anonfun$stream$1$adapted(ApponeServer.scala:31)
        at fs2.Stream$.$anonfun$flatMap$1(Stream.scala:1188)
        at fs2.internal.FreeC$.go$2(Algebra.scala:609)
        at fs2.internal.FreeC$.$anonfun$flatMapOutput$1(Algebra.scala:616)
        at fs2.internal.FreeC$$anon$1.cont(Algebra.scala:53)
        at fs2.internal.FreeC$ViewL$$anon$9$$anon$10.cont(Algebra.scala:242)
        at fs2.internal.FreeC$ViewL$.mk(Algebra.scala:231)
        at fs2.internal.FreeC$ViewL$.apply(Algebra.scala:220)
        at fs2.internal.FreeC.viewL(Algebra.scala:106)
        at fs2.internal.FreeC$.go$1(Algebra.scala:414)
        at fs2.internal.FreeC$.$anonfun$compile$8(Algebra.scala:464)
        at fs2.internal.FreeC$.$anonfun$compile$1(Algebra.scala:430)

addSbtPlugin("io.github.davidgregory084" % "sbt-tpolecat" % "0.1.14")
addSbtPlugin("io.spray" % "sbt-revolver" % "0.9.1")
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.10")

addSbtPlugin("au.com.onegeek" % "sbt-dotenv" % "2.1.204")

// deploy heroku
addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "1.7.6")
addSbtPlugin("com.heroku" % "sbt-heroku" % "2.1.0")

It's to complicated. Can you provide the simplest project, that leads to a problem? — Mikhail Ionkin, Jan 16 '21 at 14:17
Here is the github repo if that helps? https://github.com/fullStackRyan/appone — Ry2254, Jan 16 '21 at 14:19
No. Repo is also a big (and than, a complex) project. The solution to the problem is usually achieved by reducing its complexity. Localizing a problem is usually the first step towards solving it. — Mikhail Ionkin, Jan 16 '21 at 14:35
Frist, something that is not your app should create those envvars so **pureconfig** could read it, this is usually work of your deployment tooling not of your application. Specially because those are secrets and must not be part of your code nor of your repo, those should be on an encrypted vault and the automatic deployment should read them and inject them into your runtime environment before running the app. — Luis Miguel Mejía Suárez, Jan 16 '21 at 15:00

score 2 · Answer 1 · answered Jan 16 '21 at 17:45

See where the error says application.conf: Key not found: 'username'? That means it's looking for some username in that application.conf file and not finding it. If you look at your application.conf file yourself, you can see that indeed there is no setting in there called username. There is one called user, maybe that's what you had intended?

We also see Key not found: 'pool-size', looks like there's no setting for that either.

Finally, it says Key not found: 'url'. There are two url keys in your application.conf, which means that the code isn't looking in the right part of the config file. Maybe it's looking for appone.url when you want it to be looking for appone.db-config.url.

score 1 · Accepted Answer · answered Jan 16 '21 at 16:34

When running via sbt run, the environment is set by the shell in which you're running sbt. Defining a dev.env file by itself does nothing.

The mechanism for setting the environment in which sbt is running will vary depending on your shell.

For example if bash is your shell (this is worth trying in other Bourne-compatible shells), prefixing the environment variables with export, e.g.:

export ENV=dev

Then you incorporate the environment variables you've exported from dev.env into your bash environment with

source meta/dev.env

Those environment variables will then be set and thus incorporated into your config for the duration of your shell session (i.e. they'll continue across multiple sbt runs until you exit the shell). If you change dev.env, they won't be available until you source meta/dev.env again.

OP is using this sbt plugin https://github.com/mefellows/sbt-dotenv . So, unless OP did not correctly configure their build.sbt, it should work using `sbt run`. Assuming that OP is using `sbt run` to run the project. — sarveshseri, Jan 16 '21 at 17:54
Just checked the project on github, I did not see any configuation related to `dotenv`. — sarveshseri, Jan 16 '21 at 17:56

Scala: How to load environment variables into Application.conf using scala?

2 Answers2