This is a new install of foreman/puppet using a postgresql backend. When attempting to add a new host (or update an existing one using our imported previous DB hosts) the following error is seen in the foreman web ui.
Unable to save
Create Reverse IPv4 DNS record for raul-cubito.ncct.global task failed with the following error: ERF12-2357 [ProxyAPI::ProxyException]: Unable to set DNS entry ([RestClient::BadRequest]: 400 Bad Request) for proxy https://factory-7.ncct.global:8443/dns
We also receive the following error inside of our named log (raul-cubito.ncct.global is the random name foreman created).
25-Jan-2017 19:30:31.408 general: debug 1: zone_settimer: zone 105.100.IN-ADDR.ARPA/IN: enter
25-Jan-2017 19:30:31.408 general: debug 1: zone_timer: zone 112.100.IN-ADDR.ARPA/IN: enter
25-Jan-2017 19:30:31.408 general: debug 1: zone_maintenance: zone 112.100.IN-ADDR.ARPA/IN: enter
25-Jan-2017 19:30:31.408 general: debug 1: zone_settimer: zone 112.100.IN-ADDR.ARPA/IN: enter
25-Jan-2017 19:30:31.408 general: debug 1: zone_timer: zone 127.100.IN-ADDR.ARPA/IN: enter
25-Jan-2017 19:30:31.408 general: debug 1: zone_maintenance: zone 127.100.IN-ADDR.ARPA/IN: enter
25-Jan-2017 19:30:31.408 general: debug 1: zone_settimer: zone 127.100.IN-ADDR.ARPA/IN: enter
25-Jan-2017 19:30:31.408 general: debug 1: zone_timer: zone authors.bind/CH: enter
25-Jan-2017 19:30:31.408 general: debug 1: zone_maintenance: zone authors.bind/CH: enter
25-Jan-2017 19:30:31.408 general: debug 1: zone_settimer: zone authors.bind/CH: enter
25-Jan-2017 19:31:18.411 update-security: info: client 127.0.0.1#43296/key rndc.key: signer "rndc.key" approved
25-Jan-2017 19:31:18.412 update: info: client 127.0.0.1#43296/key rndc.key: updating zone 'ncct.global/IN': adding an RR at 'raul-cubito.ncct.global' A
25-Jan-2017 19:31:18.430 general: debug 1: zone_needdump: zone ncct.global/IN: enter
25-Jan-2017 19:31:18.430 general: debug 1: zone_settimer: zone ncct.global/IN: enter
25-Jan-2017 19:31:18.430 general: debug 1: zone_settimer: zone ncct.global/IN: enter
25-Jan-2017 19:31:18.431 general: debug 1: zone_timer: zone ncct.global/IN: enter
25-Jan-2017 19:31:18.431 general: debug 1: zone_maintenance: zone ncct.global/IN: enter
25-Jan-2017 19:31:18.431 general: debug 1: zone_settimer: zone ncct.global/IN: enter
25-Jan-2017 19:31:18.518 update-security: info: client 127.0.0.1#63594/key rndc.key: update '10.IN-ADDR.ARPA/IN' denied
25-Jan-2017 19:31:18.646 update-security: info: client 127.0.0.1#18812/key rndc.key: signer "rndc.key" approved
25-Jan-2017 19:31:18.646 update: info: client 127.0.0.1#18812/key rndc.key: updating zone 'ncct.global/IN': deleting rrset at 'raul-cubito.ncct.global' A
25-Jan-2017 19:31:18.676 general: debug 1: zone_needdump: zone ncct.global/IN: enter
25-Jan-2017 19:31:18.677 general: debug 1: zone_settimer: zone ncct.global/IN: enter
25-Jan-2017 19:31:18.677 general: debug 1: zone_settimer: zone ncct.global/IN: enter
25-Jan-2017 19:31:18.677 database: debug 1: decrement_reference: delete from rbt: 0x7fbab1f1f0d0 raul-cubito.ncct.global
25-Jan-2017 19:31:23.431 general: debug 1: zone_timer: zone ncct.global/IN: enter
25-Jan-2017 19:31:23.431 general: debug 1: zone_maintenance: zone ncct.global/IN: enter
25-Jan-2017 19:31:23.431 general: debug 1: zone_settimer: zone ncct.global/IN: enter
foreman-proxy log is here:
D, [2017-01-25T19:31:18.323970 ] DEBUG -- : close: 10.1.0.231:48712
D, [2017-01-25T19:31:18.366717 ] DEBUG -- : accept: 10.1.0.231:48714
D, [2017-01-25T19:31:18.369179 ] DEBUG -- : Rack::Handler::WEBrick is invoked.
D, [2017-01-25T19:31:18.372605 ] DEBUG -- : verifying remote client 10.1.0.231 against trusted_hosts ["factory-7.ncct.global"]
D, [2017-01-25T19:31:18.375281 ] DEBUG -- : running /usr/bin/nsupdate -k /etc/rndc.key
D, [2017-01-25T19:31:18.387114 ] DEBUG -- : nsupdate: executed - server 127.0.0.1
D, [2017-01-25T19:31:18.387261 ] DEBUG -- : nsupdate: executed - update add raul-cubito.ncct.global. 86400 A 10.1.0.235
I, [2017-01-25T19:31:18.438840 ] INFO -- : 10.1.0.231 - - [25/Jan/2017:19:31:18 +0000] "POST /dns/ HTTP/1.1" 200 - 0.0666
D, [2017-01-25T19:31:18.440716 ] DEBUG -- : close: 10.1.0.231:48714
D, [2017-01-25T19:31:18.485007 ] DEBUG -- : accept: 10.1.0.231:48716
D, [2017-01-25T19:31:18.487437 ] DEBUG -- : Rack::Handler::WEBrick is invoked.
D, [2017-01-25T19:31:18.488705 ] DEBUG -- : verifying remote client 10.1.0.231 against trusted_hosts ["factory-7.ncct.global"]
D, [2017-01-25T19:31:18.491298 ] DEBUG -- : running /usr/bin/nsupdate -k /etc/rndc.key
D, [2017-01-25T19:31:18.494701 ] DEBUG -- : nsupdate: executed - server 127.0.0.1
D, [2017-01-25T19:31:18.494817 ] DEBUG -- : nsupdate: executed - update add 235.0.1.10.in-addr.arpa. 86400 PTR raul-cubito.ncct.global
D, [2017-01-25T19:31:18.525675 ] DEBUG -- : nsupdate: errors
Answer:
;; ->>HEADER<<- opcode: UPDATE, status: REFUSED, id: 31844
;; flags: qr ra; ZONE: 1, PREREQ: 0, UPDATE: 0, ADDITIONAL: 1
;; ZONE SECTION:
;10.in-addr.arpa. IN SOA
;; TSIG PSEUDOSECTION:
rndc.key. 0 ANY TSIG hmac-md5.sig-alg.reg.int. 1485372678 300 16 IrfcM6Xf0cjlizVKrvQbhQ== 31844 NOERROR 0
E, [2017-01-25T19:31:18.526086 ] ERROR -- : Update errors: Answer:
;; ->>HEADER<<- opcode: UPDATE, status: REFUSED, id: 31844
;; flags: qr ra; ZONE: 1, PREREQ: 0, UPDATE: 0, ADDITIONAL: 1
;; ZONE SECTION:
;10.in-addr.arpa. IN SOA
;; TSIG PSEUDOSECTION:
rndc.key. 0 ANY TSIG hmac-md5.sig-alg.reg.int. 1485372678 300 16 IrfcM6Xf0cjlizVKrvQbhQ== 31844 NOERROR 0
D, [2017-01-25T19:31:18.526210 ] DEBUG -- : Update errors: Answer:
;; ->>HEADER<<- opcode: UPDATE, status: REFUSED, id: 31844
;; flags: qr ra; ZONE: 1, PREREQ: 0, UPDATE: 0, ADDITIONAL: 1
;; ZONE SECTION:
;10.in-addr.arpa. IN SOA
;; TSIG PSEUDOSECTION:
rndc.key. 0 ANY TSIG hmac-md5.sig-alg.reg.int. 1485372678 300 16 IrfcM6Xf0cjlizVKrvQbhQ== 31844 NOERROR 0
(Proxy::Dns::Error)
/usr/share/foreman-proxy/modules/dns_nsupdate/dns_nsupdate_main.rb:104:in `nsupdate_disconnect'
/usr/share/foreman-proxy/modules/dns_nsupdate/dns_nsupdate_main.rb:51:in `do_create'
/usr/share/foreman-proxy/modules/dns_nsupdate/dns_nsupdate_main.rb:44:in `create_ptr_record'
/usr/share/foreman-proxy/modules/dns/dns_api.rb:33:in `block in <class:Api>'
/usr/share/gems/gems/sinatra-1.3.5/lib/sinatra/base.rb:1293:in `call'
/usr/share/gems/gems/sinatra-1.3.5/lib/sinatra/base.rb:1293:in `block in compile!'
/usr/share/gems/gems/sinatra-1.3.5/lib/sinatra/base.rb:860:in `[]'
/usr/share/gems/gems/sinatra-1.3.5/lib/sinatra/base.rb:860:in `block (3 levels) in route!'
/usr/share/gems/gems/sinatra-1.3.5/lib/sinatra/base.rb:876:in `route_eval'
/usr/share/gems/gems/sinatra-1.3.5/lib/sinatra/base.rb:860:in `block (2 levels) in route!'
/usr/share/gems/gems/sinatra-1.3.5/lib/sinatra/base.rb:897:in `block in process_route'
/usr/share/gems/gems/sinatra-1.3.5/lib/sinatra/base.rb:895:in `catch'
/usr/share/gems/gems/sinatra-1.3.5/lib/sinatra/base.rb:895:in `process_route'
/usr/share/gems/gems/sinatra-1.3.5/lib/sinatra/base.rb:859:in `block in route!'
/usr/share/gems/gems/sinatra-1.3.5/lib/sinatra/base.rb:858:in `each'
/usr/share/gems/gems/sinatra-1.3.5/lib/sinatra/base.rb:858:in `route!'
/usr/share/gems/gems/sinatra-1.3.5/lib/sinatra/base.rb:963:in `block in dispatch!'
/usr/share/gems/gems/sinatra-1.3.5/lib/sinatra/base.rb:946:in `block in invoke'
/usr/share/gems/gems/sinatra-1.3.5/lib/sinatra/base.rb:946:in `catch'
/usr/share/gems/gems/sinatra-1.3.5/lib/sinatra/base.rb:946:in `invoke'
/usr/share/gems/gems/sinatra-1.3.5/lib/sinatra/base.rb:960:in `dispatch!'
/usr/share/gems/gems/sinatra-1.3.5/lib/sinatra/base.rb:794:in `block in call!'
/usr/share/gems/gems/sinatra-1.3.5/lib/sinatra/base.rb:946:in `block in invoke'
/usr/share/gems/gems/sinatra-1.3.5/lib/sinatra/base.rb:946:in `catch'
/usr/share/gems/gems/sinatra-1.3.5/lib/sinatra/base.rb:946:in `invoke'
/usr/share/gems/gems/sinatra-1.3.5/lib/sinatra/base.rb:794:in `call!'
/usr/share/gems/gems/sinatra-1.3.5/lib/sinatra/base.rb:780:in `call'
/usr/share/gems/gems/rack-1.6.4/lib/rack/commonlogger.rb:33:in `call'
/usr/share/gems/gems/sinatra-1.3.5/lib/sinatra/base.rb:161:in `call'
/usr/share/foreman-proxy/lib/proxy/log.rb:88:in `call'
/usr/share/foreman-proxy/lib/proxy/request_id_middleware.rb:9:in `call'
/usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in `call'
/usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in `call'
/usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in `call'
/usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
/usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
/usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
/usr/share/gems/gems/rack-1.6.4/lib/rack/nulllogger.rb:9:in `call'
/usr/share/gems/gems/rack-1.6.4/lib/rack/head.rb:13:in `call'
/usr/share/gems/gems/sinatra-1.3.5/lib/sinatra/showexceptions.rb:21:in `call'
/usr/share/gems/gems/sinatra-1.3.5/lib/sinatra/base.rb:124:in `call'
/usr/share/gems/gems/sinatra-1.3.5/lib/sinatra/base.rb:1417:in `block in call'
/usr/share/gems/gems/sinatra-1.3.5/lib/sinatra/base.rb:1499:in `synchronize'
/usr/share/gems/gems/sinatra-1.3.5/lib/sinatra/base.rb:1417:in `call'
/usr/share/gems/gems/rack-1.6.4/lib/rack/urlmap.rb:66:in `block in call'
/usr/share/gems/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in `each'
/usr/share/gems/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in `call'
/usr/share/gems/gems/rack-1.6.4/lib/rack/builder.rb:153:in `call'
/usr/share/gems/gems/rack-1.6.4/lib/rack/handler/webrick.rb:88:in `service'
/usr/share/ruby/webrick/httpserver.rb:138:in `service'
/usr/share/ruby/webrick/httpserver.rb:94:in `run'
/usr/share/ruby/webrick/server.rb:295:in `block in start_thread'
I, [2017-01-25T19:31:18.526878 ] INFO -- : 10.1.0.231 - - [25/Jan/2017:19:31:18 +0000] "POST /dns/ HTTP/1.1" 400 329 0.0385
D, [2017-01-25T19:31:18.568055 ] DEBUG -- : close: 10.1.0.231:48716
D, [2017-01-25T19:31:18.615342 ] DEBUG -- : accept: 10.1.0.231:48717
D, [2017-01-25T19:31:18.617373 ] DEBUG -- : Rack::Handler::WEBrick is invoked.
D, [2017-01-25T19:31:18.618385 ] DEBUG -- : verifying remote client 10.1.0.231 against trusted_hosts ["factory-7.ncct.global"]
D, [2017-01-25T19:31:18.620211 ] DEBUG -- : running /usr/bin/nsupdate -k /etc/rndc.key
D, [2017-01-25T19:31:18.622757 ] DEBUG -- : nsupdate: executed - server 127.0.0.1
D, [2017-01-25T19:31:18.622891 ] DEBUG -- : nsupdate: executed - update delete raul-cubito.ncct.global A
I, [2017-01-25T19:31:18.685449 ] INFO -- : 10.1.0.231 - - [25/Jan/2017:19:31:18 +0000] "DELETE /dns/raul-cubito.ncct.global/A HTTP/1.1" 200 - 0.0673
D, [2017-01-25T19:31:18.688007 ] DEBUG -- : close: 10.1.0.231:48717
D, [2017-01-25T19:31:18.729434 ] DEBUG -- : accept: 10.1.0.231:48718
D, [2017-01-25T19:31:18.730888 ] DEBUG -- : Rack::Handler::WEBrick is invoked.
D, [2017-01-25T19:31:18.732015 ] DEBUG -- : verifying remote client 10.1.0.231 against trusted_hosts ["factory-7.ncct.global"]
D, [2017-01-25T19:31:18.732356 ] DEBUG -- : Loading subnets for 10.1.0.231
D, [2017-01-25T19:31:18.732585 ] DEBUG -- : Loading subnet data for 10.1.0.224/255.255.255.224
D, [2017-01-25T19:31:18.735328 ] DEBUG -- : omshell: executed - set hardware-address = 08:00:27:6a:fc:a8
D, [2017-01-25T19:31:18.735429 ] DEBUG -- : nil
D, [2017-01-25T19:31:18.735496 ] DEBUG -- : omshell: executed - open
D, [2017-01-25T19:31:18.735542 ] DEBUG -- : nil
D, [2017-01-25T19:31:18.735641 ] DEBUG -- : omshell: executed - remove
D, [2017-01-25T19:31:18.735708 ] DEBUG -- : nil
D, [2017-01-25T19:31:18.760750 ] DEBUG -- : caught :modify event on /var/lib/dhcpd/dhcpd.leases.
D, [2017-01-25T19:31:18.761434 ] DEBUG -- : Deleted a reservation: 10.1.0.235:08:00:27:6a:fc:a8:raul-cubito.ncct.global
D, [2017-01-25T19:31:18.767722 ] DEBUG -- : Removed DHCP reservation for raul-cubito.ncct.global => raul-cubito.ncct.global (10.1.0.235 / 08:00:27:6a:fc:a8)
I, [2017-01-25T19:31:18.768278 ] INFO -- : 10.1.0.231 - - [25/Jan/2017:19:31:18 +0000] "DELETE /dhcp/10.1.0.224/08:00:27:6a:fc:a8 HTTP/1.1" 200 - 0.0366
D, [2017-01-25T19:31:18.769692 ] DEBUG -- : close: 10.1.0.231:48718
System information shown via foreman-debug:
HOSTNAME: factory-7.ncct.global
OS: redhat
RELEASE: CentOS Linux release 7.2.1511 (Core)
FOREMAN: 1.14.0
RUBY: ruby 2.1.8p440 (2015-12-16 revision 53160) [x86_64-linux]
PUPPET: 4.8.1
DENIALS: 117014
/etc/named.conf
acl lan {
127.0.0.0/8;
10.0.0.0/8;
};
options {
listen-on port 53 { any; };
listen-on-v6 port 53 { };
directory "/var/named";
dump-file "/var/named/data/cache_dump.db";
statistics-file "/var/named/data/named_stats.txt";
memstatistics-file "/var/named/data/named_mem_stats.txt";
allow-query { lan; };
recursion yes;
dnssec-enable yes;
dnssec-validation yes;
dnssec-lookaside auto;
/* Path to ISC DLV key */
bindkeys-file "/etc/named.iscdlv.key";
managed-keys-directory "/var/named/dynamic";
};
logging {
channel default_debug {
file "data/named.run";
severity debug;
print-time yes;
print-severity yes;
print-category yes;
};
};
controls {
inet 127.0.0.1 allow {localhost;} keys {rndc.key;};
};
include "/etc/rndc.key";
zone "in-addr.arpa" {
type master;
file "10.0.0.0";
allow-update { key "rndc.key"; };
};
zone "ncct.global" {
type master;
file "ncct.global";
allow-update { key "rndc.key"; };
};
/etc/foreman-proxy/settings.yml
---
### File managed with puppet ###
## Module: 'foreman_proxy'
:settings_directory: /etc/foreman-proxy/settings.d
# SSL Setup
# if enabled, all communication would be verified via SSL
# NOTE that both certificates need to be signed by the same CA in order for this to work
# see http://theforeman.org/projects/smart-proxy/wiki/SSL for more information
:ssl_ca_file: /etc/puppetlabs/puppet/ssl/certs/ca.pem
:ssl_certificate: /etc/puppetlabs/puppet/ssl/certs/factory-7.ncct.global.pem
:ssl_private_key: /etc/puppetlabs/puppet/ssl/private_keys/factory-7.ncct.global.pem
# Use this option only if you need to disable certain cipher suites.
# Note: we use the OpenSSL suite name, take a look at:
# https://www.openssl.org/docs/manmaster/apps/ciphers.html#CIPHER-SUITE-NAMES
# for more information.
#:ssl_disabled_ciphers: [CIPHER-SUITE-1, CIPHER-SUITE-2]
# the hosts which the proxy accepts connections from
# commenting the following lines would mean every verified SSL connection allowed
:trusted_hosts:
- factory-7.ncct.global
# Endpoint for reverse communication
:foreman_url: https://factory-7.ncct.global
# SSL settings for client authentication against Foreman. If undefined, the values
# from general SSL options are used instead. Mainly useful when Foreman uses
# different certificates for its web UI and for smart-proxy requests.
#:foreman_ssl_ca: ssl/certs/ca.pem
#:foreman_ssl_cert: ssl/certs/fqdn.pem
#:foreman_ssl_key: ssl/private_keys/fqdn.pem
# by default smart_proxy runs in the foreground. To enable running as a daemon, uncomment 'daemon' setting
:daemon: true
# Only used when 'daemon' is set to true.
# Uncomment and modify if you want to change the default pid file '/var/run/foreman-proxy/foreman-proxy.pid'
#:daemon_pid: /var/run/foreman-proxy/foreman-proxy.pid
# host and ports configuration
# Host or IPs to bind on (e.g. *, localhost, 0.0.0.0, ::, 192.168.1.20)
:bind_host: '*'
# http is disabled by default. To enable, uncomment 'http_port' setting
# https is enabled if certificate, CA certificate, and private key are present in locations specifed by
# ssl_certificate, ssl_ca_file, and ssl_private_key correspondingly
# default values for https_port is 8443
:https_port: 8443
#:http_port: 8000
# Log configuration
# Uncomment and modify if you want to change the location of the log file or use STDOUT or SYSLOG values
:log_file: /var/log/foreman-proxy/proxy.log
# Uncomment and modify if you want to change the log level
# WARN, DEBUG, ERROR, FATAL, INFO, UNKNOWN
:log_level: DEBUG
# Log buffer size and extra buffer size (for errors). Defaults to 3000 messages in total,
# which is about 500 kB request.
:log_buffer: 2000
:log_buffer_errors: 1000
/etc/foreman-proxy/settings.d/dns.yml
---
# DNS management
:enabled: true
# valid providers:
# dns_dnscmd (Microsoft Windows native implementation)
# dns_nsupdate
# dns_nsupdate_gss (for GSS-TSIG support)
# dns_libvirt (dnsmasq via libvirt)
:use_provider: dns_nsupdate
# use this setting if you want to override default TTL setting (86400)
:dns_ttl: 86400
/etc/foreman-proxy/settings.d/dns_nsupdate.yml
---
#
# Configuration file for 'nsupdate' dns provider
#
:dns_key: /etc/rndc.key
# use this setting if you are managing a dns server which is not localhost though this proxy
:dns_server: 127.0.0.1
/var/named/10.0.0.0
$ORIGIN .
$TTL 30000 ; 8 hours 20 minutes
in-addr.arpa IN SOA ncct.global. root.ncct.global. (
46 ; serial
300 ; refresh (5 minutes)
300 ; retry (5 minutes)
300 ; expire (5 minutes)
300 ; minimum (5 minutes)
)
NS ncct.global.
$ORIGIN 0.1.10.in-addr.arpa.
$TTL 1800 ; 30 minutes
231 PTR factory-7.ncct.global.
/var/named/ncct.global
$ORIGIN .
$TTL 300000 ; 3 days 11 hours 20 minutes
ncct.global IN SOA factory-7.ncct.global. root.factory-7.ncct.global. (
47 ; serial
300 ; refresh (5 minutes)
300 ; retry (5 minutes)
300 ; expire (5 minutes)
300 ; minimum (5 minutes)
)
NS factory-7.ncct.global.
TXT "ncct.global"
$ORIGIN ncct.global.
factory-7 A 10.1.0.231
linuxds CNAME factory-7
puppet CNAME factory-7
winds CNAME factory-7
/etc/rndc.key
key "rndc.key" {
algorithm hmac-md5;
secret "iiZK1kuf7L7hob1aR7PekA==";
};
