Saltstack - File Searialization - Keep Empty lines, File content sequence and Comments intact

Question

I am trying to use salt file serialization, but it is removing all empty lines, sorting the complete file in alphabetical order and removing all comments. I dont want these to happen. Tried to check here and here, but could not find anything.

{% import_yaml "app-filebeat.yml" as config %}

manage_file:
  file.managed:
    - name: /etc/filebeat/filebeat.yml
    - source: salt://filebeat.yml.tmpl
    - template: jinja

conf_file:
  file.serialize:
    - name: /etc/filebeat/filebeat.yml
    - dataset: {{ config }}
    - formatter: yaml
    - merge_if_exists: true

app_filebeat.yml:

output.logstash:
  hosts: ['myhost.example.com:5158']

filebeat.yml.tmpl:

# Filebeat prospector configuration
filebeat:
  config:
    inputs:
      enabled: true
      path: /opt/filebeat/prospector-conf/*.yml
      reload.enabled: true
      reload.period: 300s

# Logstash configuration
output.logstash:
  hosts:
  worker: 1
  compression_level: 3
  loadbalance: true
  ssl:
    certificate: /usr/share/filebeat/file_beat.crt
    key: /usr/share/filebeat/file_beat.key
    verification_mode: none

# Filebeat Logging Configuration
logging:
  level: debug
  to_files: true
  files:
    path: /var/tellme/log/filebeat/
    name: filebeat.log
    rotateeverybytes: 10485760
    keepfiles: 7

Ouput:

filebeat:
  config:
    inputs:
      enabled: true
      path: /opt/filebeat/prospector-conf/*.yml
      reload.enabled: true
      reload.period: 300s
logging:
  files:
    keepfiles: 7
    name: filebeat.log
    path: /var/tellme/log/filebeat/
    rotateeverybytes: 10485760
  level: debug
  to_files: true
output.logstash:
  compression_level: 3
  hosts:
  - myhost.example.com:5158
  loadbalance: true
  ssl:
    certificate: /usr/share/filebeat/file_beat.crt
    key: /usr/share/filebeat/file_beat.key
    verification_mode: none
  worker: 1

Salt State output:

----------
          ID: manage_file
    Function: file.managed
        Name: /etc/filebeat/filebeat.yml
      Result: True
     Comment: File /etc/filebeat/filebeat.yml updated
     Started: 04:53:53.678745
    Duration: 118.489 ms
     Changes:
              ----------
              diff:
                  ---
                  +++
                  @@ -0,0 +1,29 @@
                  +# Filebeat prospector configuration
                  +filebeat:
                  +  config:
                  +    inputs:
                  +      enabled: true
                  +      path: /opt/filebeat/prospector-conf/*.yml
                  +      reload.enabled: true
                  +      reload.period: 300s
                  +
                  +# Logstash configuration
                  +output.logstash:
                  +  hosts:
                  +  worker: 1
                  +  compression_level: 3
                  +  loadbalance: true
                  +  ssl:
                  +    certificate: /usr/share/filebeat/file_beat.crt
                  +    key: /usr/share/filebeat/file_beat.key
                  +    verification_mode: none
                  +
                  +# Filebeat Logging Configuration
                  +logging:
                  +  level: debug
                  +  to_files: true
                  +  files:
                  +    path: /var/tellme/log/filebeat/
                  +    name: filebeat.log
                  +    rotateeverybytes: 10485760
                  +    keepfiles: 7
----------
          ID: conf_file
    Function: file.serialize
        Name: /etc/filebeat/filebeat.yml
      Result: True
     Comment: File /etc/filebeat/filebeat.yml updated
     Started: 04:53:53.797737
    Duration: 14.986 ms
     Changes:
              ----------
              diff:
                  ---
                  +++
                  @@ -1,4 +1,3 @@
                  -# Filebeat prospector configuration
                   filebeat:
                     config:
                       inputs:
                  @@ -6,24 +5,21 @@
                         path: /opt/filebeat/prospector-conf/*.yml
                         reload.enabled: true
                         reload.period: 300s
                  -
                  -# Logstash configuration
                  +logging:
                  +  files:
                  +    keepfiles: 7
                  +    name: filebeat.log
                  +    path: /var/tellme/log/filebeat/
                  +    rotateeverybytes: 10485760
                  +  level: debug
                  +  to_files: true
                   output.logstash:
                  -  hosts:
                  -  worker: 1
                     compression_level: 3
                  +  hosts:
                  +  - myhost.example.com:5158
                     loadbalance: true
                     ssl:
                       certificate: /usr/share/filebeat/file_beat.crt
                       key: /usr/share/filebeat/file_beat.key
                       verification_mode: none
                  -
                  -# Filebeat Logging Configuration
                  -logging:
                  -  level: debug
                  -  to_files: true
                  -  files:
                  -    path: /var/tellme/log/filebeat/
                  -    name: filebeat.log
                  -    rotateeverybytes: 10485760
                  -    keepfiles: 7
                  +  worker: 1

Please help.

Only the `hosts` section seems to have additional value extra in rendered output. Looking at this, its hard to say what is the expected output? What is in `app-filebeat.yml`? — seshadri_c, Jan 21 '22 at 12:36

seshadri_c · Answer 1 · 2022-01-22T02:40:55.170

2

The file.serialize module does not look like the right choice for this requirement.

You already have a template for filebeat.yml, and the requirement is to modify this file in-place and update only hosts under output.logstash: section.

For this the file.accumulated might be better with changes as below:

filebeat.yml.tmpl:

# Logstash configuration
output.logstash:
  hosts:
  {%- for host in accumulator['extra_config'] %}
    - {{ host }}
  {%- endfor %}

app-filebeat.yml:

hosts: ['myhost.example.com:5158']

Then the below states:

{% import_yaml "app-filebeat.yml" as config %}

extra_config:
 file.accumulated:
    - filename: /etc/filebeat/filebeat.yml
    - text: {{ config.hosts }}
    - require_in:
        - file: manage_file

manage_file:
  file.managed:
    - name: /etc/filebeat/filebeat.yml
    - source: salt://filebeat.yml.tmpl
    - template: jinja

edited Jan 22 '22 at 02:40

answered Jan 21 '22 at 14:20

seshadri_c

6,906
2
10
24

I am using file.serialize because my requirement is to override any value in the template. Replacing the host is just a small example that I took. Correct requirement is to override any key in the yaml hierarchy. Also I should be able to add new keys on the fly while overriding. – Sujeet Padhi Jan 21 '22 at 14:37
May be you can update the question with this requirement so its more obvious. Also, the template are meant to have many such conditions/loops to dynamically update configuration on-the-fly. – seshadri_c Jan 21 '22 at 15:48
Serialize actually works perfect, but it removes the comments and empty lines as well as it is sorting the yaml keys in alphabetical order. Sorting is not an issue, but I would like to keep the comments and empty lines for better readability. :( – Sujeet Padhi Jan 21 '22 at 16:02
3

but that is the point. `file.serialize` will not keep comments. nor any kind of presentation data. it is a serialization component not a text editor. it takes machine data and translates it into other machine language. the comments and presentation data is lost before it even gets to file.serialize. importing the yaml into a data structure removes them because it is a data structure not a text structure. – whytewolf Jan 21 '22 at 20:08

OrangeDog · Answer 2 · 2022-01-21T13:30:45.827

0

import_yaml and merge_if_exists load YAML as a python data structure. All formatting, ordering, and comments are lost. This is intentional, because they are irrelevant to the data structure.

If you want to manage the file as a plain text file, then use file.managed. If you want to manage it as structured data, then you don't get to choose the presentation details. Don't try to do both to the same file.

edited Jan 21 '22 at 13:30

answered Jan 21 '22 at 12:48

OrangeDog

36,653
12
122
207

I can see Serializer doing that from the output. If you see my code "import_yaml" is just importing a yaml file having 2 lines, it does not have any comment – Sujeet Padhi Jan 21 '22 at 12:53

Saltstack - File Searialization - Keep Empty lines, File content sequence and Comments intact

2 Answers2