Difference between revisions of "Redundant Servers - ICX-AlphaCom"

Revision as of 14:13, 28 June 2022

Redundant Servers - ICX-AlphaCom

The Redundant Server concept

The concept is 1:1 redundancy of servers. The pair of servers has the same configuration, and only one of them is active at any time.

The High availability solution is maximizing uptime during:

System level failures
Environment-level failures (power outage, fire, earthquake, etc.)
Planned outage for system maintenance and upgrades

Abbreviations

HA: High Availability
HA pair: A pair of servers, where one is operational, and the other is in standby with the same configuration.
Primary Server (Configuration Master): The server in a HA pair that has the configuration as programmed from AlphaPro. This role is defined via the license. The role is fixed to a physical server.
Secondary Server (Configuration slave): The server in a HA pair that fetches the configuration from the Primary Server. This role is fixed to a physical server.
Operational IP address: A common IP address for a HA pair that stations and SIP devices connect to. The Operational IP Address is bound to the currently Operational server.
Operational server: The server which is currently using the operational IP address and to which the stations and devices are registered to. This role is not fixed to a physical server.
Standby server: The server which currently is not operational. It is not using the Operational IP Address. This role is not fixed to a physical server.
Operational interface: The ethernet interface which the Operational IP address is bound to, eth0 or eth1.

Software requirements

All versions of ICX-AlphaCom software support the Redundancy Server feature. However, it is strongly recommended to always use the latest software to get the latest features, bug fixes and security updates.

Important notes:

Both servers in a HA pair must run on the same software version
Both servers in a HA pair must be of the same hardware type. ICX-500 and ICX-Core cannot be mixed.
The servers must be connected to the same network (subnet)

Licenses

Three different licenses are available for the Redundancy feature:

1002606006 - ILR-U64 - Redundancy License 64 Users
1002606008 - ILR-U256 - Redundancy License 256 Users
1002606009 - ILR-U512 - Redundancy License 512 Users

These licenses can be stacked.

The system must have a Redundancy License installed covering all stations and devices used on the system. That is the sum of the following licenses:

IP Station licenses
SIP Station licenses
SoftClient licenses
Amplifier Channels licenses
IP-ARIO licenses

The same license file is installed both in the Primary Server and in the Secondary Server. Information in the license file determines which server will be the Primary and which will be the Secondary.

Information in the license file decides which server is the Primary and Secondary server

For test and demonstrations one can use the free license period. In general there must either be a free license in both servers, or valid licenses in both servers. When the servers are running in Free License period, which one to be the Configuration Master (Primary Server) is set by a check box in the Web Interface of the servers.

Configuration Master is set by a check box when running in the Free License period

Description

The principle

Two servers are set up as a redundant HA pair, sharing the same configuration, including node number. One of the servers is the operational server, the other one is in standby. The standby server monitors the operational server via the IP network and sends a request for a status update every three seconds. The standby server will try to take over if it looses the connection to the operational server, or if it receive a message that the operational server has made a reset.

Each time the operational server resets, the standby server will take over, i.e. the servers will swap roles at every reset. This also applies to a "small reset" (application reset).

The network interface

The network interfaces of the server is configured in the standard way with separate static IP addresses. The static addresses are used for maintenance, AlphaPro and AlphaWeb. The operational address is an additional IP address, shared by the servers in the pair. Only one of them is active on that address at any time. All operational devices, like stations, SIP devices, other AlphaCom node, must be set up to connect to the operational address.

The operational address can be bound to eth0 or eth1.
There must be configured a static IP address on the interface which will host operational address.
The operational address must be within the subnet address range defined for the interface.
A HA pair using only one ethernet port each will need a total of 3 IP addresses.
A HA pair using both ethernet ports will need a total of 5 IP addresses.

The Intercom Application (AMCd) is running also while a server is in standby mode, but no stations will register to it because the operational IP address is used by the other (operational) server. If the operational server fails, the standby server will take over the operational IP address. Stations will then reconnect to the same IP address, but now to the new server.

Distribution of configuration

The Primary Server in the HA pair is designated as Configuration Master. AlphaPro updates are sent to the Primary Server. The Secondary Server will automatically be updated with copy of the configuration from the Primary Server. The Primary Server checks every 15 seconds if the configuration has been changed. If changed, the Secondary server is notified, and it will fetch and apply the configuration file via HTTP (port 80) or HTTPS (port 443). The operational server also updates the standby server with copy of the SIP registration database (because SIP registration times may be as long as an hour).

Configuration

Redundant Servers - Configuration example

The configuration of the Redundant Server feature is done from the Web interface of the servers. This is best explained by an example. The below confiuration example is with reference to the drawing to the right.

Configure the IP address in both servers

In System Configuration > Interfaces configure the Ethernet port 1 of the Primary Server with IP address 10.9.5.91
In System Configuration > Interfaces configure the Ethernet port 1 of the Secondary Server with IP address 10.9.5.92

Install licenses

In System Configuration > Licensing, install the licenses. The same license file is used on both servers.
Check the license file and notice the MAC addresses, they tell which server will be the Primary Server and which will be the Secondary Server.

Configure the Redundancy feature in the Primary Server

On the Primary Server go to System Configuration > High Availability, click 'Create HA config'. Set:

Operating interface: eth1 (10.9.5.91) (This is the local ethernet port used by this server)
Peer exchange name: Any descriptive text (Typically you enter the name of the other server)
Operational IP address: 10.9.5.90 (This is the common IP address used by both servers, and to which all stations and devices register to. Must be identical on both servers)
Peer maintenance IP address: 10.9.5.92 (This is the IP address of the other server)
Username: alpha (This is the "Read" user name as defined in System Configuration > User Management. Default is alpha/com. If the "Read" user is changed, this field must be updated accordingly)
Password: com (This is the "Read" password name as defined in System Configuration > User Management. Default is alpha/com. If the "Read" user is changed, this field must be updated accordingly)
HTTP or TLS/HTTPS: Here you choose if the communication between the servers should use HTTP (port 80) or HTTPS (port 443).
- Make sure HTTP (port 80) or HTTPS (port 443) is enabled in the Filters settings for Eth1
Validate and Save

Configure the Redundancy feature in the Secondary Server

On the Secondary Server go to System Configuration > High Availability, click 'Create HA config'. Set:

Operating interface: eth1 (10.9.5.92) (This is the local ethernet port used by this server)
Peer exchange name: Any descriptive text (Typically you enter the name of the other server)
Operational IP address: 10.9.5.90 (This is the common IP address used by both servers, and to which all stations and devices register to. Must be identical on both servers)
Peer maintenance IP address: 10.9.5.91 (This is the IP address of the other server)
Username: alpha (This is the "Read" user name as defined in System Configuration > User Management. Default is alpha/com. If the "Read" user is changed, this field must be updated accordingly)
Password: com (This is the "Read" password name as defined in System Configuration > User Management. Default is alpha/com. If the "Read" user is changed, this field must be updated accordingly)
HTTP or TLS/HTTPS: Here you choose if the communication between the servers should use HTTP (port 80) or HTTPS (port 443).
- Make sure HTTP (port 80) or HTTPS (port 443) is enabled in the Filters settings for Eth1
Validate and Save

AlphaPro settings

In AlphaPro go to Exchange & System > System:

In the first IP address field enter the local IP address of the Primary Server. Enable AlphaPro to Connect to this address (here 10.9.5.91). Uncheck "AlphaNet".
In the second IP address field enter the Operational IP address of the HA pair. Check "AlphaNet".

(You can ignore the description in AlphaPro that the first IP address field is for Eth0 and the second field is for Eth1).

IP settings in AlphaPro

AlphaPro cannot connect to the Secondary Server. A connection attempt will be rejected by the server.

Status information and Test

Status information

The blue Active LED on the front and rear side of the ICX-500 is lit on the unit which is currently operational. This LED is off on the standby server.

The Active LED shows that this server is currently the operational server

The current status is also displayed in the Web interface of the servers. In System Monitoring > Node Information:

Current status is shown in the web interface

Explanation:

Redundant Servers (IPHA): Shows if this server is either Config Master (= Primary Server) or Configuration Slave (= Secondary Server)
Redundant Server State: Shows the current state of this server - Operational or Standby

The status is also displayed in System Configuration > High Availability:

Current status displayed in System Configuration > High Availability:

Haipd SW version: shows the current High Availability IP Daemon version
server peer name: The name of the other server
HA State: operational or standby
Connected to <other server> since: shows the time and date since the two servers were paired
Downloaded /api/haipd/amcd_conf.tar.gz: shows when the configuration file was last updated.
Downloaded /api/haipd/state_data.tar.gz: shows when the state data file was last updated.

Test and verification

Force takeover: The standby server can be forced to take over the operation in several ways:

Small Reset or Reboot from the Web interface of the Operational Server
Disconnect the ethernet cable on the Operational Server
Power off the Operational Server

Observe takeover: Observe that the standby server changes state to Operational and that:

The Active LED becomes lit
Status changes in the web interface of the servers

Verify that:

Stations and Devices are registered and operational
AlphaNet (if installed) is operational
SIP trunks (if installed) are operational
Any integrations to 3rd party systems are operational

Logging:

Status information are logged in the System log
Additional logging of status can be generated by adding some extra configuration. See IPHA - Monitoring server status

Synchronization of Data - Additional info

Configuration from AlphaPro

The configuration data sent from AlphaPro is written to the working (RAM) memory of the Primary Server. When the update is finished, the configuration data is copied to the flash memory for permanant storage. The secondary server will be notified about the change, and will fetch and apply the new configuration data.

After system reset or startup the working RAM memory is cleared. Configuration data is now loaded to the working memory from the flash.

Other configuration

Some configuration can also be modified by the users:

Programming DAK keys on the station - code 784
Changing the volume settings on the station

And some configuration can be modified by data commands, typically used within the Event Handler:

$VOL - Modify station volume
$GRM - Include/excluse a station from group
$SSC - Modify the CoS (Class of Service) for a station
$SNV - Modify values in the working RAM memory

These configuration changes are written to the working memory. However they are not immediately copied to the flash memory. This is to preserve the longevity of the flash memory itself. When a change is made, a 4 hour timer is started. At timeout the changes are written to the flash memory, and the configuration is syncronized with the Secondary server. When doing a software reset, any pending changes will be copied to flash before the reset is performed. (No copying if sudden power reset).

The syncronization is from Primary Server to Secondary Server only, not in the other direction.
Configuration changes done when the Secondary Server is operational will not be synchronized with the Primary server, hence any changes will be lost when the Primary becomes operational again.
If users or events are changing configuration data, it should be considered to configure the system in such a way that the Primary Server is always operational when the HA pair is running.

State data

State data is stored in a dedicated battery backed up RAM memory. State data is temporarily user data such as:

State data is not synchronized between the servers.

If it is important to preserve the state data, it should be considered to configure the system in such a way that the Primary Server is always operational when the HA pair is running.

Set Primary Server to always be the operational server (optional)

To "force" the Primary Server to always be the operational server when the HA pair is up and running, add this Event Handler to the system:

In AlphaPro, Exchange & System > Events, Insert a new event using Event Type 27, Subevent 231, When Change To = ON:

Primary Server set to always be the operational server

Action Command when using software version 1.2.3.6 or later:
Action commands:

IF %syse(230)
STOP
ENDIF
IF %op(%syse(232),=,0)
$SMALL_RESET
ENDIF

Action Command when using software version 1.2.3.5 or earlier:
Action commands:

IF %syse(230)
STOP
ENDIF
IF %op(%syse(232),=,0)
$RESET
ENDIF