2

I want to utilize windows failover cluster generic service role for my application. and I'm trying to figure out how to perform the upgrades.

I've read there is an option to perform a "cluster-aware" upgrade, I.E: hand the cluster some MSI \ installer and let him be in charge of upgrading all the nodes.

Has anyone that used that feature can:

  1. Can describe how he did it?
  2. Are there any special requirements to enable it?
  3. Is it recommended?
barakcaf
  • 1,294
  • 2
  • 16
  • 27
  • You are hosted windows service in a cluster? – UserName Sep 24 '18 at 18:58
  • @UserName - yes (its a built-in option by Microsoft's failover cluster) – barakcaf Sep 25 '18 at 06:31
  • What you ask isn't related to programming. It's an administration operation, covered in Technet since 2008, if not earlier. The specifics vary from one OS to another so you should check the docs relevant to your target server. Newer OSs will have newer, better or easier to handle techniques to provide high availability – Panagiotis Kanavos Sep 25 '18 at 06:37
  • As for whether it's a good idea or not, it depends on the requirements. Web/REST services for example don't need a Windows cluster to ensure availability and upgrades. Perhaps there are so many dependencies that it's better to move an entire VM instead of a single service. Containers add another angle, one that doesn't have the hardware requirements of Windows Clustering – Panagiotis Kanavos Sep 25 '18 at 06:53
  • 1
    If you want I can describe how we upgrade/update our clustered[WSFC] Windows Services. – UserName Sep 25 '18 at 07:02
  • The (no longer updated) docs for Windows 2008 R2 are [here](https://learn.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2008-R2-and-2008/cc753938(v=ws.11)) The section for 2016 [is here](https://learn.microsoft.com/el-gr/windows-server/failover-clustering/create-failover-cluster#create-clustered-roles). There are various blog posts, eg [this one about creating generic service resources](https://blogs.msdn.microsoft.com/clustering/2009/06/08/creating-and-configuring-a-generic-service-resource/) from 2009. – Panagiotis Kanavos Sep 25 '18 at 07:04
  • @UserName - yes please – barakcaf Sep 25 '18 at 10:37
  • @Panagiotis Kanavos - tnx for the response - I'm familiar with the resources you referenced. I don't think its unrelated to programming since if it is possible to upgrade a FO cluster generic service automatically there will certainly be some constratins on the way the service upgrade is implemented (DB schema compatability for example) – barakcaf Sep 25 '18 at 10:40
  • @barakcaf if you want good answers you'll have to ask at serverfault.com. Yes, there are other similar questions, in fact you could consider this a duplicate (Or too broad). The answers though eventually point to a relevant question at serverfault.com. PS: You'll have to mention a specific Windows Server version. The availability features change a lot from one version to the next. – Panagiotis Kanavos Sep 25 '18 at 10:42
  • @Panagiotis Kanavos I have posted this question at serverfault as well - and haven't been answered. I also searched there and couldn't find a similar question - if you can refer me to that question you were talking about or even better to an answer I'd very much appreciate it. (I need to support as many server versions post-2008 r2 as possible) – barakcaf Sep 26 '18 at 05:23
  • 1
    Prepared the answer. I hope it will be useful. – UserName Sep 28 '18 at 16:20

1 Answers1

2

We have clustered Windows Services that are use the .NET stack. At the moment, each cluster role are hosted on two nodes only. Deployment and upgrade process is performed via Ansible. The following snippets cover upgrade part only.


For service deployment are used Nuget packages. Used .nuspec is represented below. So packages are represent the .zip archives, which contains all content in the root.

<?xml version="1.0"?>
<package>

  <metadata>
    <id>$Id$</id>

    <version>$Version$</version>
    <authors>$Authors$</authors>

    <description> $Description$ </description>
    <releaseNotes>$ReleaseNotes$</releaseNotes>
  </metadata>

  <files>
    <file src="$PackageInput$" target=" "/>
  </files>

</package>

The described below role could be used for composite cluster roles, when one cluster role contains multiple resources.

- name: 'Copy the cluster_role.ps1 to all hosts'
  win_copy:
    src : 'cluster_role.ps1'
    dest: 'cluster_role.ps1'

This task is needed to copies to all hosts the PowerShell script, which is required for detecting the owner node and for moving role between nodes.

param([String]$ClusterRoleName, [String]$ExcludeNode)

# Task: Define the owner node
if ($ClusterRoleName -ne [string]::Empty -and $ExcludeNode -eq [string]::Empty)
{
    Get-ClusterResource -Name $ClusterRoleName | Format-List -Property OwnerNode
    exit
}

# Task: Move the cluster role
if ($ClusterRoleName -ne [string]::Empty -and $ExcludeNode -ne [string]::Empty)
{
    Move-ClusterGroup $ClusterRoleName (Get-ClusterNode | Where-Object { $_.State -eq 'Up' -and $_.Name -ne $ExcludeNode })
    exit
}

- name: 'Define the owner node'
  win_shell: 'cluster_role.ps1 -ClusterRoleName {{ cluster_role }}'
  register: owner_node
  run_once: True
  when: 'cluster_role is defined'


- name: 'Define the owner node metadata'
  set_fact:
    owner_node_host: '{{ owner_node.stdout.split(":")[1] | trim }}.{{ windows_domain }}'
    owner_node_name: '{{ owner_node.stdout.split(":")[1] | trim }}'
  run_once: True
  when: 'cluster_role is defined'

These tasks are needed to detect the owner node. The first task returns the owner node, for example: s001srv000. The second task creates two variables following types:

owner_node_host  : s001srv.domain.net
owner_node_name: s001srv000 

- name: 'Apply the application roles on the inactive nodes'
  include_role:
    name: '{{ item }}'
  when  : 'cluster_role is defined and (cluster_sets is defined or cluster_full is defined) and owner_node_host != inventory_hostname'
  with_items: '{{ dependencies }}'

These task is include another roles such as downloading new version package, generating service configuration depending on environment, etc. Performs on inactive nodes.


- pause:
    prompt : 'A manual failover must be manually performed'
    minutes: 30
  run_once : True
  when: 'cluster_full is defined and environment_type == "prod"'


- name: 'Move the cluster role'
  win_shell: 'cluster_role.ps1 -ClusterRoleName {{ cluster_role }} -ExcludeNode {{ owner_node_name }}'
  run_once : True
  when: 'cluster_move is defined or cluster_full is defined'

These tasks are needed to control upgrade flow, if current environment is STG, then upgrade will be performed automatically, otherwise with a manual failover at pause moment.


- name: 'Apply the application roles on the nodes which were active a moment ago'
  include_role:
    name: '{{ item }}'                                                                                                  
  when  : 'cluster_role is defined and cluster_full is defined and owner_node_host == inventory_hostname'

These task is the same as 'Apply the application roles on the inactive nodes', but for nodes, which were active a moment ago.

UserName
  • 895
  • 7
  • 17
  • Tnx - that's very helpful! what about DB upgrade? It's a SAAS app your describing right (you deploy to environments you control)? – barakcaf Sep 30 '18 at 07:58
  • "These task is include another roles such as downloading new version package" - what type of role is this - a "generic applicaiton" role? – barakcaf Sep 30 '18 at 08:01
  • 1
    @barakcaf, deployment occur on own environments. At the moment items such deploy databases, configure users, configure linked servers, clustering are automated. Tables schemas upgrade is not automated. At the moment being test Liquibase for database migrations, but without any automation. – UserName Sep 30 '18 at 09:12
  • 1
    @barakcaf, yes it's generic application roles. All components are in private Nuget repository. In our Ansible project exists baseline file. It's describe key-value pairs like `component:version`, which are use at upgrade - downloading new version. Also all components have configuration files, that modify at upgrade according an environment configuration and a configuration template. – UserName Sep 30 '18 at 09:25
  • 1
    @barakcaf, I don't describe details about database upgrade, because it will be additional article, but if you have a questions.. – UserName Sep 30 '18 at 09:52