How to synchronize Redmine database pacemaker/pcs/corosync?

Question

Currently I have 3 servers: one master and 2 clients. I installed redmine 3.3.1.stable with postgresql 9.6, and installed pacemaker on 3 servers. To synchronize a database, I follow the documentation. Every thing is working fine until when I stop the active server. The server2 redmine is showing authentication error.

Redmine error when I try to login form client after connect servers.

Completed 500 Internal Server Error in 11ms (ActiveRecord: 3.5ms)

ActiveRecord::StatementInvalid (PG::ReadOnlySqlTransaction: ERROR:  cannot execute UPDATE in a read-only transaction
: UPDATE "users" SET "last_login_on" = '2020-08-17 13:05:11.001886' WHERE "users"."type" IN ('User', 'AnonymousUser') AND "users"."id" = $1):
  app/models/user.rb:238:in `try_to_login'
  app/controllers/account_controller.rb:204:in `password_authentication'
  app/controllers/account_controller.rb:199:in `authenticate_user'
  app/controllers/account_controller.rb:40:in `login'
  lib/redmine/sudo_mode.rb:63:in `sudo_mode'

so far I unterstand redmine only allow server1 to access permission. why redmine can't give access to server2 or server3

Below I give more information about my step so far.

pcs config

pcs config
Cluster Name: mycluster
Corosync Nodes:
 server1 server2 server3
Pacemaker Nodes:
 server1 server2 server3

Resources:
 Resource: MasterVip (class=ocf provider=heartbeat type=IPaddr2)
  Attributes: ip=101.226.189.208 nic=lo cidr_netmask=32 iflabel=pgrepvip
  Meta Attrs: target-role=Started
  Operations: start interval=0s timeout=20s (MasterVip-start-interval-0s)
              stop interval=0s timeout=20s (MasterVip-stop-interval-0s)
              monitor interval=90s (MasterVip-monitor-interval-90s)
 Resource: Apache (class=ocf provider=heartbeat type=apache)
  Attributes: configfile=/etc/apache2/apache2.conf statusurl=http://localhost/server-status
  Operations: start interval=0s timeout=40s (Apache-start-interval-0s)
              stop interval=0s timeout=60s (Apache-stop-interval-0s)
              monitor interval=1min (Apache-monitor-interval-1min)

Stonith Devices:
Fencing Levels:

Location Constraints:
  Resource: Apache
    Enabled on: server1 (score:INFINITY) (role: Started) (id:cli-prefer-Apache)
Ordering Constraints:
Colocation Constraints:
  Apache with MasterVip (score:INFINITY) (id:colocation-Apache-MasterVip-INFINITY)
Ticket Constraints:

Alerts:
 No alerts defined

Resources Defaults:
 migration-threshold: 5
 resource-stickiness: 10
Operations Defaults:
 No defaults set

Cluster Properties:
 cluster-infrastructure: corosync
 cluster-name: mycluster
 dc-version: 1.1.16-94ff4df
 have-watchdog: false
 no-quorum-policy: ignore
 stonith-enabled: false

Quorum:
  Options:

master postgresql.conf

# Add settings for extensions here
listen_addresses = '*'
wal_level = hot_standby
synchronous_commit = local
archive_mode = on
archive_command = 'cp %p /var/lib/postgresql/9.6/main/archive/%f'
max_wal_senders = 10
wal_keep_segments = 30
synchronous_standby_names = 'server2'
synchronous_standby_names = 'server3'
hot_standby = on

master pg_hba.conf

    # Localhost
    host    replication     postgres     127.0.0.1/32            md5
    
    # PostgreSQL Master IP address
    host    replication     postgres      101.226.189.205/32            md5
    
    # PostgreSQL SLave IP address
    host    replication     postgres         101.226.189.206/32            md5
    ho

st    replication     postgres          101.226.189.207/32            md5

copy config to client from Master

pg_basebackup -h server1 -U postgres -D /var/lib/postgresql/9.6/main -X stream -P

Database connection status

postgres@oreo:/etc/postgresql/9.6/main$ psql -x -c "select * from pg_stat_replication;"
-[ RECORD 1 ]----+------------------------------
pid              | 18174
usesysid         | 10
usename          | postgres
application_name | server3
client_addr      | 101.226.189.207
client_hostname  |
client_port      | 35236
backend_start    | 2020-08-17 15:56:40.687282+02
backend_xmin     |
state            | streaming
sent_location    | 0/7005430
write_location   | 0/7005430
flush_location   | 0/7005430
replay_location  | 0/7005430
sync_priority    | 1
sync_state       | sync
-[ RECORD 2 ]----+------------------------------
pid              | 18175
usesysid         | 10
usename          | postgres
application_name | server2
client_addr      | 101.226.189.206
client_hostname  |
client_port      | 45862
backend_start    | 2020-08-17 15:56:40.717087+02
backend_xmin     |
state            | streaming
sent_location    | 0/7005430
write_location   | 0/7005430
flush_location   | 0/7005430
replay_location  | 0/7005430
sync_priority    | 0
sync_state       | async

This does not look like a db login problem. The error says that it is failing to execute an `update` in what is a read-only copy of the database. The fact that you get a PostgreSQL error from the server indicates that the credentials work. This problem is caused by the fact that your application's login process must write to `users` to update the `last_login_on` column. — Mike Organek, Aug 17 '20 at 15:38
thank you for your replay. we know actually you wrote here. can you give more explanation about how to make server read and write mode. we tried to change "default_transaction_read_only = off" but we got error "cannot set transaction read-write mode during recovery" — Karippery, Aug 18 '20 at 13:39
I am sorry that I cannot help you with this as I have no experience with Redmine. I commented only because I thought you were misinterpreting the error message you were getting as a DB credentials problem when it is not. Based on this latest message, it looks like either failover did not happen or your application is bound to a read-only replica. — Mike Organek, Aug 18 '20 at 13:51

score 0 · Answer 1 · answered Sep 09 '20 at 08:24

I found answer for my question. I miss one step in pacemaker resources.

The pgsqld defines the properties of a PostgreSQL instance: where it is located, where are its binaries, its configuration files, how to montor it, and so on.

The pgsql-ha resource controls all the PostgreSQL instances pgsqld in your cluster, decides where the primary is promoted and where the standbys are started.

 pcs resource create pgsqld ocf:heartbeat:pgsqlms    \
    bindir="/usr/lib/postgresql/9.6/bin"                            \
    pgdata="/etc/postgresql/9.6/main"                               \
    datadir="/var/lib/postgresql/9.6/main"                          \
    pghost="/var/run/postgresql"                                    \
    recovery_template="/etc/postgresql/9.6/main/recovery.conf.pcmk" \
    op start timeout=60s                                            \
    op stop timeout=60s                                             \
    op promote timeout=30s                                          \
    op demote timeout=120s                                          \
    op monitor interval=15s timeout=10s role="Master"               \
    op monitor interval=16s timeout=10s role="Slave"                \
    op notify timeout=60s


pcs  resource master pgsql-ha pgsqld notify=true
pcs  resource cleanup
pcs status

How to synchronize Redmine database pacemaker/pcs/corosync?

1 Answers1