Inject variables into JSON config file

Question

We have an AWS Lambda function in place that dynamically builds test environments for our engineers to test their code through GitHub pull requests.

This Lambda function is called using GitHub webhooks, whereby Github POSTs across all the information needed to configure the test environment for that specific product.

At the moment, the AWS Lambda function is hard coded to parse the POST data and build the test environment. However, as the range of products being tested increases and the range of test environments become more diverse, we are wanting to move away from the hard coded approach; for manageability sake too.

What I want to be able to do, is load a configuration file, preferably in JSON and apply some of the data in the GitHub POST data to the configuration file, essentially injecting the data into the JSON config.

I don't know how to approach this. I've seen a question being asked on here wanting to do something similar, but in Java: Inject dynamically generated data into JSON

Originally our Lambda function was written in Node.js, but we've started to move across to Go - mainly because it was an interesting new challenge. If there is a solution to this in Node.js I'll take that, but if there is a solution available in Go, that would be preferable.

Edit:

The config file and the GitHub POST data have two different structures.

GitHub POST data (heavily stripped down):

{
 "action": "opened",
  "number": 89,
  "pull_request": {
    "url": "https://api.github.com/repos/Owner/ExampleRepository/pulls/89",
    "head": {
      "repo": {
        "id": 123454678,
         "name": "ExampleRepository"
        }
      }
  }
}

Example Config File (also stripped down):

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Description": [GitHub Post data].pull_request.url,
  "Resources": {
      "ElasticBeanstalk": {
         "Type": "AWS::ElasticBeanstalk::Environment",
         "Properties": {
             "ApplicationName": [GitHub Post data].pull_request.head.repo + [GitHub Post data].number,
             "Description": [GitHub Post data].pull_request.url
          }
     }
  }
}

Both the config file and the GitHub POST data is much more complex than this. Also, the config file will reference various parts of the POST data multiple times and sometimes would require the concatenation of multiple values.

Do you want to just treat the json as text, and template it, or do you want to manipulate more complex data structures? Either way the solutions seems pretty straightforward, what have you tried? — JimB, Jan 25 '18 at 14:05
Yeah I would prefer to treat the JSON as text. I don't want to do anything complex other than be able to essentially add in the JSON config file some kind of 'link' to reference the JSON POST data coming from GitHub. In Node.js I've tried to loop over the JSON config file and doing a find and replace approach, or dynamically loading the JSON config file as a module and passing across the POST data from GitHub; both of which are cumbersome and the latter overly convoluted. Not really tried anything in Go as I am entirely new to it and don't really know what the language can do. — JSmith, Jan 25 '18 at 14:11

icza · Answer 1 · 2018-01-25T14:27:31.933

2

Attempting to text-manipulate JSON-text can be error-prone (think about escapes and unicode sequences, e.g. this is valid JSON: {"a":"\"b"}). It's much easier and safer to unmarshal the JSON config into a Go value, perform the modifications on it, then marshal to JSON.

If the config structure is known, create matching Go structs to model it exactly.

If the config structure is unknown or you don't want to hassle with it, you may unmarshal into a value of type interface{}, and perform the modification on it. This will be more verbose as you have to perform type assertions for each property and indices.

To ease the pain of the latter, you may use 3rd party libs such as github.com/icza/dyno (disclosure: I'm the author).

For "merging" your existing config and data from the POST request, check out this answer: "Merge" fields two structs of same type

edited Jan 25 '18 at 14:27

answered Jan 25 '18 at 14:16

icza

389,944
63
907
827

The JSON structure is known, so that isn't too much of an issue. How can I use structs to apply the data from the GitHub POST data to the JSON template - assuming the JSON config isn't text based if that makes it easier. – JSmith Jan 25 '18 at 14:21
@JSmith For that check out this answer: ["Merge" fields two structs of same type](https://stackoverflow.com/questions/47395430/merge-fields-two-structs-of-same-type/47396406#47396406) – icza Jan 25 '18 at 14:29
That's assuming the structure of both the POST data and the config is the same, which it's not. I've tried adding an example of the data I'm dealing with but I'm struggling with the formatting on StackOverflow. – JSmith Jan 25 '18 at 14:53
@JSmith just add an empty line between code and normal text for proper formatting. – mkopriva Jan 25 '18 at 15:34

score 0 · Answer 2 · answered Jan 25 '18 at 17:09

Why not use the standard packages? The os package has ExpandEnv to expand ${VARNAME} and $VARNAME style substrings in a given string with the environment variable values. Just pass the values through as environment values, read the file and do:

replacedVars := os.ExpandEnv(fContents)

Check os.ExpandEnv docs

Basically, I'd write something like:

package main

import (
    "fmt"
    "io/ioutil"
    "os"
)

func main() {
    fname = "foobar.json" // wherever this comes from
    buf, err := ioutil.ReadFile(fname)
    if err != nil {
        fmt.Printf("Failed to read file %s - %+v", fname, err)
        return
    }
    conf := os.ExpandEnv(string(buf))
    // do whatever you need to do with conf string...
}

score 0 · Answer 3 · answered Jan 29 '18 at 09:56

0

I appreciate all the responses, each of which has helped me further increase my knowledge of Go.

The best way of solving this issue for me was to use the text/template library and create the JSON configuration as a text template, which allowed me to easily apply the variables from the GitHub post request.

https://golang.org/pkg/text/template/

answered Jan 29 '18 at 09:56

JSmith

143
1
8

Please check the help section for this site. You're asked not to leave thank you comments as an answer to your own question. The way to show appreciation would be to vote up/down, and perhaps leave a comment. – Elias Van Ootegem Apr 12 '18 at 16:51

Inject variables into JSON config file

3 Answers3