YM:High Availability

1 Introduction

High Availability with Hyper Converged Systems.

1.1 Benefits

The High Availability Solution (HA) for the IMC, gives the advantage of minimum server downtime, for machines which are necessary to operate the line, in case of a single hosting Server failure. The typical downtime is less than two minutes. The WCCOA-Server will be restarted on the remaining hosting Server without any user interaction.

All the essential software for operating the line are installed in virtual machines. They are easy to migrate to other hosts and most hardware independent.

For reduced complexity and cost, the High Availabiltiy Solution comes with a minimum of necessary hardware input. That reason for, the storage system for HA protected VMs is also running as virtual machines on the hosting Servers.

1.2 Virtualization

VM
A virtual machine (VM) is a software emulated computer with a own operating system. These VMs are running on a hosting machine which provides a virtual environment.

Virtual Environment
Compute Power, RAM and storage space is made available for software emulated computers by the hosting machine. A virtual environment allows a better hardware utilization of the hosting machine, because most of the computers don't run at full load at all time. The software or Operating System, which provides the virtual environment is also called Hypervisor.

Microsoft Hyper-V
Hyper-V is Microsoft’s virtualization platform. This hypervisor comes as an feature with Windows Server 2012 R2, but is also available for Windows 10.

VMware vSphere
vSphere is suite of software products by VMware. This includes products like VMware ESXi, VMware vCenter Server and a lot more.

VMware ESXi
VMware ESXi is a small lightweight linux based OS for bare metal installations. It virtualizes x86 server hardware and is supported by any notable server manufacturer.

2 Components of the HA solution

2.1 Hosting Machines

There are three computer in the IMC-switchboard:

"Admin Server", KomHstAdm with IP 10.XX.YY.1

"Kom Server 1", KomHst001 with IP 10.XX.YY.2

"Kom Server 2”, KomHst002 with IP 10.XX.YY.3

All these computers are providing a virtualized environment.

File:IpcHASystemOverview.png

The “Admin Servers” virtualized environment comes from Microsoft and is called Hyper-V.
The “Kom Servers” virtualized environment comes from VmWare and is called ESXi Hypervisor.

2.1.1 Admin Server

The Admin Server is a Windows Server 2012 R2 based machine and provides the builtin virtual environment from Microsoft, also called Hyper-V. It too, has a NTP time server service installed, providing the system time to many other clients, especially to the WCCOA-Server. On a own partition, there is also the storage space for all the available backups. The "Admin Station's" operating system is modified as less as possible.

On the Admin Server are usually running following Hyper-V VMs:

KomFOM

KomGstAdm

KomRmt

2.1.2 Kom Server 1

Makes up the High Availability Solution, togehter with Kom Server 2.

On Kom server 1 are the VMs, which are necessary to operate the line. Because of the "High Availability Feature", these VMs can run either on Kom Server 1, or 2. The VMware ESXi Hypervisor, which provides the virtual environment, is installed on an internal 8 GB Flash-Storage.
When the server starts, it will boot the ESXi Hypervisor found on the flash storage. After a complete bootup, the virtual environment is ready and starting the registered VMs automatically.

Some of the VMs are Host exclusive and can't be migrated to Kom Server 2, they are intended to run on their respective host.
Some of the VMs can also registered and restarted on Kom Server 2, they are HA protected and necessary to operate the line.

Host exclusive VMs are:

KomVMA001

KomVSA001

HA protected VMs are:

Kom

KomRdh

KomSvn

2.1.3 Kom Server 2

Makes up the High Availability Solution, togehter with Kom Server 1.

Kom Server 2 is setup like Kom server 1.
When Kom Server 1 fails, the HA protected VMs will be automatically registered and restarted at Kom Server 2.

Host exclusive VMs are:

KomVMA002

KomVSA002

HA protected VMs are:

same as Kom Server 1, but only when Kom Server 1 fails

2.2 Shared Software Storage

For HA protected VMs, the referenced files of these VMs must be made accessible to both of the hosting machines at any time. The hardware device which provides such a storage is called "Storage Area Network” (SAN).

2.2.1 Basic idea with a hardware SAN

This solution is not used with the IMC!

File:EsxiWithSan.png

Virtual machines which are stored on such a shared storage can either be started on Server 1 or Server 2.
A special VM called vCenter, takes care that the VMs are handled correctly.

2.2.2 Extended concept with Hyperconvergent Systems)

This solution comes with the IMC!

Above is shown the basic idea, but at Brückner lines the virtualization is taken one step further. Also the SAN is virtualized. This means that no hardware SAN exists.
Only the function Shared Storage exists, which is provided by special VMs called Virtual Storage Appliance (VSA), KomVSA00x.

With the use of only two Virtual Storage Appliances comes also the necessity of using a third, special VM, the Failover Manager (KomFOM). In case of a single Kom Server failure, the KomFOM is, together with the remaining KomVSA00x, holding the majority of all member systems.

The majority of all member systems is called Quorum.
The quorum is achieved automatically, as soon as two member systems get in contact. This happens automatically.

The three member systems for the Virtual Storage Solution are:

KomVSA001, running on Kom Server 1, KomHst001,

KomVSA002, running on Kom Server 2, KomHst002

KomFOM, running on Admin Host, KomHstAdm

Each member system has a managing component, for checking the status among each other.
Only the KomVSA00x VMs are also providing the storage space for the HA protected VMs.

2.2.2.1 Virtual Storage Appliance, KomVSA00x

For a better understanding, only the VSAs are shown, because they are making up the storage space!

On each Kom Server, one VSA-VM KomVSA00x is installed. The VM is host exclusiv and will be started automatically by the hypervisor.
As long as the VMs KomVSA001 and/or KomVSA002 are not running there is no virtual SAN available.

File:Vsan off.jpg

As soon as the VSA VMs are up and a quorum is established, the virtual SAN is available.
The VSA VMs are also synchronizing their dataset, which includes the files for the HA protected VMs, between them. They are running in a RAID 1 configuration.

File:Vsan on S1.jpg

The virtual SAN contains three Volumes, which are separate storage spaces:

System_Volume: Contains all files for the operating systems of the HA protected VMs.

Visu_Volume: Contains only the virtual harddisk file for the database of the WCCOA, Kom VM.

ILM_Volume: Contains only the virtual harddisk file for the database of the KomRdh VM.

These volumes can be accessed by both Kom Servers.
However, HA protected VMs can be registered and running only on one Kom Server at any given time.

File:Vsan on.jpg

If one of the virtual storage VMs has temporarely gone offline, it will be automatically resynchronized to the recent dataset, as soon as it comes alive again.
Any isolated KomVSA00x will just block all data I/O traffic, to keep the storage dataset consistant for a later resynchronization.

2.2.2.2 Failover Manager, KomFOM)

The Failover Manager (FOM) is running as a Hyper-V VM on a independent hosting machine, the Admin Host Server.
The FOM has only a managing component, without providing any storage space.

In case of a hardware failure of one of the two Kom Servers, the KomFOM helps the remaining KomVSA00x to keep the virtual storage open for data I/O. Together they keep the quorum.

2.3 vCenter Server, KomVct

The vCenter Server (KomVct) is a virtual HA protected machine. The main purpose for this machine is to collect and organize all the assets, which are running in the vSphere environment. This includes the host systems, the HA protected VMs and host exclusive VMs.
It too, enables several features which are necessary for the High Availabilty. Therefore it does the configuration for the two host systems to run as a HA environment.

The main features are High Availablity together with Live Migration. The vCenter makes sure, that HA protected VMs are registered at one of the available host systems.
With the vCenter Server, all the other VMs can be configured, monitored and moved (migrated) from one host machine to the other one.

3 Virtual Machines

Virtual Machines can either run under ESXi from VMware or Hyper-V from Microsoft.

Only virtual machines which where not discussed before, will be mentioned here.
This excludes:

KomFOM

KomVSA001

KomVSA002

KomVct

3.1 Hyper-V VMs

These VMs are running under Hyper-V and are managed by the Hyper-V manager, available at the Admin Host.

3.1.1 Admin Guest, KomGstAdm

Admin Guest is a "Windows Server 2012 R2" based virtual machine. It is essentially used to manage and backup the VMware VMs running on Kom Server 1 and 2.
All the VMware VM backups are directly transfered via a network share to a separate partition on the Admin Host Machine. Also the configuration files for the two Kom Servers can be saved by this machine and stored at the Admin Host.
It too holds all the programs, necessary for the setup and the maintenance of the VMware vSphere Environment and the software storage solution VSA by HP's LeftHandOS.

3.1.2 Remote Workstation, KomRmt

The Remote Station has all the programs installed, to configure and manage the electrical equipment of the line. Most important the Simatic Manager with the Starter software.
This machine is intended for service use by Brueckner personnel only.

3.2 VMware VMs

The VMs are running under VMware EXSI and are managed by the vCenter Server KomVct, available via vSphere Client.
Most important is the VM WinCCOA VM. It has the same name as your line, e.g. Sri. The other VMs have the line name at the beginning plus an extension which gives a hint to its usage.

3.2.1 Kom

This VM runs the SCADA system, WinCC OA.

3.2.2 KomRdh

This VM runs the Roll Data History, a long term database with production parameters for every single mill roll.

3.2.3 KomSvn

This Subversion VM is used for the electrical engineering during the checkup. It is only used by Brueckner staff.

3.2.4 KomVMA00x

The Virtual Management Assistant VM is a host exclusive machine. It receives shutdown signals from the UPS and the shutdown icons, available on the operator stations.
When a shutdown signal is received, either from UPS or the icons script, the VM will start a power down sequence and orderly shut off the VMware VMs and the according Kom Server.

4 Standard procedures

This section describes some procedures for the operator.

4.1 Manual Start procedure

Initial situation: Admin Station, Server 1 and Server 2 are powered off.
Intention: Power on the IMC-system and get it proper running.

• Switch on the “Admin Station”.
• Switch on the “Server 1” by pressing the power button. The light will change to green.
• Wait 5 minutes than switch on “Server 2”.

The servers will make their “Power On Self Tests”, start the hypervisors and after that all the VM’s.
This can take up to 35 minutes.

4.2 Manual Stop procedure

Initial situation: Admin Station, Server 1 and Server 2 are powered on.
Intention: Power off the servers in a proper way. Leave the "Admin Station" untouched.

To initiate a complete shutdown-sequence manually there is a “Shutdown S1 and S2” icon on each operator station.
This shutdown icon will be available when you login as a user which has access to the windows desktop.

File:S1 AND S2 ShutDown.png

If you doubleclick this icon a window will open to ask you if you really want to shutdown.
Enter “yes” and the shutdown sequence will start.

File:Serv1 ShutDownClicked.png

After that all the VM’s on the server will be stopped, server 2 will shutdown and power off, after that server 1 also.
When the servers are down you will see that the color of the power-down button on the server will change from
green to amber.

You will have to more icons, one to only shutdown server 1 and one for only shutdown server 2.
These buttons can be used in case one of the server have to be switched off e.g. for maintenance reasons. They will normally not be used.

Attention
Different to the UPS-shutdown signal the signal triggered by operator will only shutdown the two servers
but not the Admin Station.

If you also want to shutdown the Admin Station, log in and shut it down via Windows Start button.

4.3 Power down by UPS

The UPS will provide a constant voltage for the whole IMC-System.
However, after a powercut the UPS will run only a certain time on battery.
After this time the UPS sends a shutdown signal to the server and the admin station.

The server will shutdown all the VM’s and then the hypervisor and power off.
A few minutes later also the admin station will shutdown.

5 Advanced procedures for troubleshooting and administration

5.1 Server administration

To administrate the server you will normally connect remotely to "Admin Guest", a special VM
created exactly for that purpose. It runs as a hyper-V VM on the "admin station".

You can connect to the "Admin Guest" via RDP,
see: Server administration with the "Admin Guest Station"

However, there are two more possible ways to access and handle the server.

• VGA-Display port of server

  This port provides only a very limited user interface to set the most basic parameters and

  to do some basic tasks.

• Service port

  This port is used to get detailed hardware information and control even without running operating system.

5.1.1 Server administration with the "Admin Guest" station

The “Admin Guest Station” provides all the programs and resources which are needed for server
administration and backups.
With the “Admin Guest Station” you can do:

• Access ESXi Hypervisor

  with vSphere client.

• Create Backups of ESXi VM’s

  with Veeam Backup & Replication

• Manage the virtual storage appliances. (“virtual SAN”)

  with CMC, the Central Management Console:

5.1.1.1 Essential programs installed on "Admin Guest"

To get access to these programs you must login to "Admin Guest".
You do so by starting an "Remote Desktop Connection (RDP)" from another PC using
IP-address 10.XX.YY.11 as user "root".

5.1.1.1.1 vSphere client

With the vSphere client you can access KomHst001 (server 1), KomHst002 (server 2) and VM KomVct.

"vCenter", named "KomVct" on the lines, is a special VM made by VMware to manage and
administrate other VM's. You can access vCenter in the same way as a server with "vSphere Client".
vCenter has IP: 10.xx.yy.12

To start the vSphere client doubleclick icon “VmWare vSphere Client”, enter vCenter IP and login.

File:AdminGuestLogin.png

After login as user “root” and selecting “View – Inventory – Inventory” you will see
a list of virtual machines (VM’s) running on this server.

Via the vSphere Client you can perform following steps:

Stop a virtual machine:

Select a VM – right mouse: Power – Shut down guest

Start a virtual machine:

Select a VM – right mouse: Power – Power on

5.1.1.1.1.1 Access ESXi Servers directly

To do some administration tasks for the VMware VM's you will normally connect with vSphere Client to
the vCenter.
However, you can also connect with vCenter to the servers directly.
With vCenter you will have a view onto the system as a "High-Availabilty" cluster.
When you connect to the server directly you will get a view of the virtualized environment on this single server.

5.1.1.1.2 Veeam Backup & Replication

Doubleclick to Icon "Veeam Backup & Replication:

File:VeeamBackup.png

You will find that for each VM a backup job exists.
By starting the backup job the VM will be saved to the disk on the Admin Station.

5.1.1.1.3 Central Management Console (CMC)

With CMC you manage the "Virtual Storage" provides by the two running VSA's.
Doubleclick to Icon "CMC", select KomMgmGrp and login as user "admin".
Open "KomCluster" and then "Volumes". The picture below is taken from "Nao", replace "Nao" with your line name.
File:CMC.png

There you can check for are any errors.

Attention: Don't change any settings but call Brueckner support in case of problems.

5.1.2 VGA Display

Only a minimal graphical user interface is available via hardware VGA-monitor port.

File:ESXiMinUi.png

With this user interface some basic settings as IP-address or alike can be set.
From there the server can also be stopped or restarted.

The full functionality of the ESXi hypervisor is only accessible via the “vSphere Client” which can be started
e.g. on the “Admin Guest”. You can connect to the “Admin Guest” via RDP.

Attention:
Actually the hardware VGA-monitor in the IMC-switchboard is not connected to the server
but to the “Admin Station”.
But you can access the monitor port also remotely via web-browser from any Windows Station or from the
“Admin Station” via service port (iLO), see “Sever Console via Service Port” below.

5.1.3 Service Port

Each computer has an additional network port for service. It’s called "iLO" by HP.
With this service port you can see the status of the server and other hardware information.
You can also access the "Server console" to get the monitor screen.

Adresses:
10.xx.yy.6: Service Port Admin Station
10.xx.yy.7: Service Port Server 1
10.xx.yy.8: Service Port Server 2

Access this port via web-browser e.g. “Internet explorer”, login as user "admin"

File:ILO Login.png

Server monitor via Service Port

The monitor is connected via VGA-cable to the “Admin Station”, there is no KVM-switch to connect
the monitor to the port of the server. The reason for this is that you can access the monitor port remotely
via Web Browser from any Windows station or from the “Admin Station”.

To do so:

• Connect to "Admin Guest" VM via RDP.
• Open Internet Explorer and enter address of iLO service port http://10.xx.yy.7
  
• Login as user “admin”
• Select menu “Remote Console/Remote console” -> Under “Java Integrated Remote Console” select
  

  “Web Start” -> Some file will be downloaded, start this file -> next and you will be connected to the
  

  Server console.
  

5.2 Administration of "Admin Station"

Administration of the "Admin Station" is normally limited to handle the VM running there.
Normally that are the VM's "Admin Guest","Remote" and "FOM".
The handling of the Hyper-V VM's is done by the "Hyper-V Manager".

5.2.1 Hyper-V Manager on the "Admin Station"

Start the Hyper-V manager by clicking to this icon in the taskbar:

File:HyperV ManagerTaskbar.png

You will see there the two virtual machines mentioned above:
(If you have “ThinClients” you will see here a VM for each ThinClient)

File:HyperV Manager.png

Via the Hyper-V Manager you can perform following steps:

To stop a virtual machine:

Select a VM – right mouse: Power – Shut down
(Don’t use “Turn off”, it’s like pulling the cable from a physical machine)

To start a virtual machine:

Select a VM – right mouse: Power – Power on

5.2.1.1 Remote Station VM

The “Remote Station” has all the programs installed to configure and manage the line.
You can connect via RDP (172.ss.tt.15) or TightVNC.

File:RmtStation.png

5.2.1.2 Fail Over Manager (FOM)

This VM just watches the state of the storage VSA VM's and is necessary in case one of the server fails.

5.2.1.3 Admin Guest

You can connect to the “Admin Guest” via RDP (10.xx.yy.11).

5.3 Summary

IMC-Overview

Remark:
Replace xx.yy with your project number. E.g. Sri 1835 –> xx=18, yy=35.
Replace ss.tt. with your service sub-segment addresses.

File:XXXXGeneral IPC-Overview.png

Programs to connect:
RDP: Remote Desktop Protocol. Client can be found on each Windows System.
vSphere Client: Client to access the ESXi hypervisor. Found on CodLef and Remote Station.
Putty: Ssh client, can be found on each operator station and Remote Station.
TightVNC: VNC client, can be found one each operator station and Remote Station.
Web browser: Can be found on each Window System.


Table of used IP's
(Abbreviations in column Type: "HW": Hardware, "V-VM": Virtual Machine VMware, "H-VM": Virtual Machine Hyper-V)

IP	Type	Name	Abbreviation	Operating System	Connect with	User and password rule
10.xx.yy.1	HW	Admin Host	KomHstAdm	Windows Server 2012 R2	RDP	root (Password rule: Rootroot0)
10.xx.yy.2	HW	Server 1	KomHst001	ESXi	vSphere Client	root (Password rule: rootroot)
10.xx.yy.3	HW	Server 2	KomHst002	ESXi	vSphere Client	root (Password rule: rootroot)
10.xx.yy.6	HW	Service Port Admin Host	KomHstAdmSvc	n.a.	Web Browser	admin (Password rule: enigma created)
10.xx.yy.7	HW	Service Port Server 1	KomHst001Svc	n.a.	Web Browser	admin (Password rule: enigma created)
10.xx.yy.8	HW	Service Port Server 2	KomHst002Svc	n.a.	Web Browser	admin (Password rule: enigma created)
10.xx.yy.11	H-VM	Admin Guest VM	KomGstAdm	Windows Server 2012 R2	RDP	root (Password rule: Rootroot0)
10.xx.yy.12	H-VM	Vcenter VM	KomVct	SuSE	vSphere Client	root (Password rule: rootroot)
10.xx.yy.13	H-VM	Fail Over Manager (FOM)	KomFOM	n.a.	Via vSphere Client console	admin (Password rule: enigma created)
172.ss.tt.15	H-VM	Remote Station	KomRmt	Windows 7 or 10	RDP, TightVNC	root and others (Password rule: enigma created)
10.xx.yy.31	V-VM	WinCC OA Visualization	Kom	Centos 7.2	Putty	root and others (Password rule: enigma created)
10.xx.yy.36	V-VM	Virtual Machine Assistant (VMA)	KomVMA001	SuSe	Putty	vi-admin (Password rule: vi-admin)
10.xx.yy.37	V-VM	Virtual Machine Assistant (VMA)	KomVMA002	SuSe	Putty	vi-admin (Password rule: vi-admin)
10.xx.yy.40	H-VM	Roll Data Handling (RDH)	KomRdh	W7 or W10	RDP	root (Password rule: enigma created)