I have multiple OpenStack environments. I have one set of web/ftp/db servers for a DEV environment and another set of web/ftp/db servers for a QC environment. I have two internal DNS entries, one for each environment:
dev-foo.this.net
qc-foo.this.net
Those DNS entries each point to a VIP (floating IP?) in OpenStack:
dev-vip.this.net (10.240.38.117) <- dev-foo.this.net (10.240.38.117)
qc-vip.this.net (10.240.38.192) <- qc-foo.this.net (10.240.38.192)
These VIPs are not currently mapped to any fixed IPs.
I have two HAProxy servers set up in a fail over configuration using a heartbeat to automatically fail over to the other server if the current primary dies:
proxy1.this.net (10.240.38.68)
proxy2.this.net (10.240.38.69)
It appears that which ever one of them is the current primary (and therefore the only active one) gets assigned two VIPs 1.0.0.240 and 1.0.0.241. Those are the VIPS (I assume VIP is the correct term here) I see when I run ifconfig on the primary.
eth0: inet 1.0.0.19
eth0:0: inet 1.0.0.240
eth0:1: inet 1.0.0.241
What I want/need to do is:
1> Have HAProxy take ALL traffic requests for dev-foo.this.net and send it to the DEV environment servers.
2> Have HAProxy take ALL traffic requests for qc-foo.this.net and send it to the QC environment servers.
3> Have the SSL requests SSL terminated at the proxy so I can examine the request and change it. This is because I need to translate some URLs to different URLs.
4> Have the request re-encrypted and forwarded on to the backend servers.
So multiple questions:
1> Can I bind the same IP to multiple ports on the same front end?
For instance bind 80, 443 and 22 on 1.0.0.240 in a single front end definition.
2> What causes the request traffic for dev-foo.this.net to get picked up by a specific VIP on the proxy server?
How do I know dev-foo.this.net requests will go to 1.0.0.240 on the proxy servers?
3> How do I get SSL requests re-encrypted and forwarded on to the backend servers?
4> Can I assume that the VIPs (1.0.0.240, 1.0.0.241) on the proxy servers will be the same when a fail over from one to the other happens?
I don't know how these get associated to the proxy machines.
Let me say I'm not a networking person so I may not be asking this correctly. Bear with me. I thought I might be able do something like this in the haproxy.cfg but I'm confused and need some guidance:
# -----------------------------------------
# web server stuff for DEV environment
# -----------------------------------------
frontend dev-app
mode http
bind 1.0.0.240:80
bind 1.0.0.240:443 ssl crt /etc/haproxy/ssl.pem/star.this.net.pem
option http-tunnel
option forwardfor
reqadd X-Forwarded-Proto:\ https if { ssl_fc }
use_backend dev-app-backend
backend dev-app-backend
mode http
balance roundrobin
cookie SERVERID insert indirect nocache
option forwardfor
option httplog
option httpchk HEAD /
option http-tunnel
http-check expect status 200
server www01 dweb200.this.net:443 check ssl verify none cookie www01
server www02 dweb201.this.net:443 check ssl verify none cookie www02
# -----------------------------------------
# web server stuff for QC environment
# -----------------------------------------
frontend qc-app
mode http
bind 1.0.0.241:80
bind 1.0.0.241:443 ssl crt /etc/haproxy/ssl.pem/star.this.net.pem
option http-tunnel
option forwardfor
reqadd X-Forwarded-Proto:\ https if { ssl_fc }
use_backend qc-app-backend
backend qc-app-backend
mode http
balance roundrobin
cookie SERVERID insert indirect nocache
option forwardfor
option httplog
option httpchk HEAD /
option http-tunnel
http-check expect status 200
server www01 qweb200.this.net:443 check ssl verify none cookie www01
server www02 qweb201.this.net:443 check ssl verify none cookie www02
# -----------------------------------------
# sftp server stuff for DEV environment
# -----------------------------------------
frontend dev-sftp
mode tcp
bind 1.0.0.240:22
option tcplog
default_backend dev-sftp-backend
backend dev-sftp-backend
mode tcp
balance roundrobin
option tcplog
server ftp01 dftp200.this.net:22
server ftp02 dftp201.this.net:22
# -----------------------------------------
# sftp server stuff for QC environment
# -----------------------------------------
frontend qc-sftp
mode tcp
bind 1.0.0.241:22
option tcplog
default_backend qc-sftp-backend
backend qc-sftp-backend
mode tcp
balance roundrobin
option tcplog
server ftp01 qftp200.this.net:22
server ftp02 qftp201.this.net:22
* EDIT *
For question #1. I did see this post which seems to say you can bind to multiple ports like I want to do but the rest of my questions stand unresolved.
For question #3. Looking further it seems the re-encryption happens because of the "ssl" parameter on the backend server lines. Sticky sessions are caused by the cookie parameter.
For question #2. I THINK I need to somehow configure the DNS IP to be bound to the VIP on the proxy server. In the OpenStack web UI the two floating IPs:
dev-vip.this.net (10.240.38.117) qc-vip.this.net (10.240.38.192)
are not mapped to any fixed IP addresses and I think they should be mapped, separately, to the 1.0.0.240 and 1.0.0.241 addresses on the proxy servers. That would cause the correct traffic to go to the correct backend environment QC/DEV. At least I think so. I’m not sure how to do it though, it may be something like is shown here I’m still trying to understand this part.
* FURTHER EDIT *
For Question #2 I found this which basically what I'm trying to do and it seems I need to do steps 6 and 7.
* FURTHER FURTHER EDIT * NOTE: This is for running the OpenStack CLI on Windows.
Okay so, to do the “binding” of the floating IPs (dev-vip.this.net 10.240.38.117, qc-vip.this.net 10.240.38.192) to the VIPs on the proxies (1.0.0.240 and 1.0.0.241) you have to install the OpenStack CLI . Apparently you can’t do this from the web client.
This replaces all of the other commands you will see referenced out there on the web, like neutron and nova. You have to install python, a version prior to 3. I installed 2.7.1
Once python is installed put the python main folder and the python\scripts folder in your path.
Now you need to install pip, download the install script and put it in your python\scripts folder.
Open a command line and run the script with this command “python d:\python27\scripts\get-pip.py”.
Now make sure it also installed the setuptools and wheel packages with this command “python -m pip install --upgrade pip setuptools wheel”.
Finally you can install the openstack CLI with this command “pip install python-openstackclient”.
To make the openstack CLI work you need to have certain environment vars set. Log into the openstack web dashboard for the zone your machines/ips are in. Open the Project tab, open the Compute tab and click Access & Security. On the API Access tab click download OpenStack RC File and save the file. This is a unix shell script you need to convert it to a windows cmd file and change some things it should end up looking something like this:
set OS_IDENTITY_API_VERSION=3
set OS_AUTH_URL=http://someip:5000/v3
set OS_TENANT_ID=<<your tenant id>>
set OS_TENANT_NAME=<<your tenant name>>
set OS_PROJECT_NAME=<< your project name >>
set OS_REGION_NAME=us-east-1a
set OS_USERNAME=<< YOUR USER NAME >>
set OS_PASSWORD=<< YOUR PASSWORD >>
Note that I had to remove the quotes from the values to make things work. If you are using a domain as part of the authentication you'll have to add one of the DOMAIN settings as well.
Open a new command prompt and run this cmd file to set up the env vars. Now you should be able to run the “openstack” command at the command prompt. If it works you will get a prompt back that looks like this “(openstack)” with the cursor after it. You can then run the following command to make sure everything is working from the openstack CLI “network list”. If you get a list of stuff back instead of an authentication error you are good to go.
* FURTHER FURTHER FURTHER EDIT *
Make sure to install the pfx cert file for your web site into the IIS servers on the back end so they can handle the reencrypted traffic from the proxies. I'm still trying to get the traffic to route through the proxy servers, I'm getting close. I haven't been able to get the floating IPs bound to the proxy VIPs yet and I'm pretty sure that is what is keeping me from getting the traffic routed through the proxies.
... so it looks like the rest of the problems I was having were due to firewalls and other network stuff. One of our guys fixed that part for me so I don't really know what the deal is. Where I used FQDN for the backend servers that was wrong because that didn't resolve to the 1.0.0.X addresses on those machines.
* LAST EDIT * Ohhhh, me likey.
I found that I had the httpchk line in the backend definitions in the proxy config file. That is what was causing HAProxy to think the web servers were down and therefore it wouldn't send traffic to them. I took that out and added "ca-file /etc/ssl/certs/ca-bundle.crt" to the backend server lines, so it would know where the CAs could be found. And BLAMM-O the web servers now show as active in the proxy stats page AND I can hit both https and http in both environments.
Also note that I had to change the bindings in IIS to NOT have the host name set.
