0

I have a fluentd configuration that pulls data from the file and pushes to the SQL server however there is a primary key with the auto-incremented column, so, in my fluentd configuration if I don't mention that column it throws an error saying that the field is missing and if I include the column in the configuration it gives identity error, in below configuration "Id" is the primary and auto-incremented column, also let me know if adapter "sqlserver" is the right thing to use.

<filter record.**>
      @type record_transformer
      enable_ruby  true
      <record>
        Id ${id}
      </record>
      <record>
        timestamp ${time}
      </record>
    </filter>

    <filter record.**>
      @type stdout
    </filter>

    <match record.**> 
        
        @type sql
        host myhost
        username myuser
        password mypassword
        database mydb
        adapter sqlserver

        <table>
          table simple_table
          column_mapping 'Id:Id,timestamp:timestamp'
        </table>
        
        flush_interval 1s
        # disable_retry_limit
        # num_threads 8
        # slow_flush_log_threshold 40.0
    </match>
Mysterious288
  • 365
  • 5
  • 24
  • Please add the error messages for both cases in the question. Also, there's no mention of such a case in the docs of [fluent-plugin-sql](https://github.com/fluent/fluent-plugin-sql) output plugin (https://github.com/fluent/fluent-plugin-sql#output-how-it-works). Nor is there any special handling for this case while parsing `column_mapping` (https://github.com/fluent/fluent-plugin-sql/blob/8c3c59322a51b2bce8d9fcebdf6e9aa0598e71b3/lib/fluent/plugin/out_sql.rb#L140-L149). There is no such issue reported so far. See [issues](https://github.com/fluent/fluent-plugin-sql/issues), just to be sure. – Azeem Jul 04 '21 at 07:19
  • You should [open an issue](https://github.com/fluent/fluent-plugin-sql/issues/new/choose) and report this. – Azeem Jul 04 '21 at 07:21
  • @Azeem It's an issue however I figured it out what was causing the error, opening an issue in GitHub doesn't seem like someone will reply soon. – Mysterious288 Jul 04 '21 at 15:46

1 Answers1

0

Well, I figured this out, it's mandatory to send the column name in the column_mapping even though if its primary key and auto-incremented, if you login with some other SQL credential it will give you an error, however, if you login with the same details used at the time of table creation it works.

Mysterious288
  • 365
  • 5
  • 24