Tag: Failover

Categories
Knowledge Support Support exacqVision Enterprise

Failover and Failback Issue in exacqVision Server and Enterprise Manager

Description 

Level 3 Support has identified an issue with Failover functions in ExacqVision Server software manifesting as failback failure. The component responsible is ExacqVision Server software, but errors can be detected from the Enterprise Manager Failover Groups page as well, leading some customers to report this as an issue with the Enterprise Manager software. It is recommended that ExacqVision Server software version 23.03.3.0 be installed on protected and spare NVRs as a workaround until the future release can address this issue.

Product 

  • ExacqVision Server Software Version 23.06.1.0, 23.09.6.0, 23.09.9.0, 24.03.4.0
  • ExacqVision Enterprise Manager Software Versions:
    23.06.2.0 23.09.7.1, 23.09.9.0, 24.03.4.0

Steps to Reproduce 

  1. Upgrade exacqVision Server, Client and Enterprise Manager to versions 23.06.x.x. thru 24.03.x.x from previous versions.
  2. Test Failover/Failback functionality.

Expected Results 

Failover and Failback functions should complete without error.

Actual Results 

Failover and Failback functions do not complete, or complete with errors – sometimes forcing the abandonment of data, which then does not get returned to the protected NVR.

Solution

Prior to taking any of the steps below it is advised to take a backup configuration file of exacqVision Server, along with a backup license key of all NVRs affected.

It is preferred to also have prior configuration backups available for NVRs taken at earlier times.

It is also preferred to have a backup copy of Enterprise Manager’s configuration and/or a copy of the PostgreSQL or Microsoft SQL database.

  1. Stop all Enterprise Manager services to delete this failover group from shell. See: https://support.exacq.com/#/knowledge-base/article/4998
  2. Start all Enterprise Manager services and Delete the NVRs that were included in the failover group from Enterprise Manager.
    (this is a good time to take another backup)
  3. Unmanage the NVRs that were included in the failover group from exacqVision Client > Enterprise > Security tab.
  4. Downgrade (or if using older version, upgrade) exacqVision Servers to exacqVision Server version 23.03.3.0
    (At this step it is advised to restore from a previously taken backup that is known to have no issues at the time it was taken, if available, but this may not always be required)
  5. Return the Spare to an “Active Recorder” in the Client if still showing as spare – remove exacqVision Server software from the spare (not keeping configuration files for future installs) and then install a fresh version of exacqVision Server 23.03.3.0.
  6. If needed you can restore Enterprise Manager from BOTH the previous EM backup AND the Database Backup or by connecting to existing SQL database (if this is verified to have no issues). Alternatively, add back NVRs if issues are known to be present with the backup file or configuration at the time this was taken. This also may not always be required if EM configuration is in current good standing at this step.
  7. Verify no connection issues are present with the Servers and then Re-create the previous Failover Group and test functionality.

AES-733, AES-705

Categories
Knowledge Support exacqVision Enterprise Support Products Uncategorized

Windows Firewall Rules to prevent Failover / Failback function issues

By default VFBA uses port 28774 and TCP to communicate with EM and the Server from the Spare NVR. Rules need to be configured for outbound and inbound firewall rules on the SPARE server for failback to work as expected.

To check the settings Open up Windows  Defender Firewall with Advanced Security and click on Inbound Rules on the SPARE server.

Look for a rule allowing communication on port 28774, if you do not see this in the list, then click on New Rule in the upper right hand corner in the Actions section.

Select Port and click on Next.

Select Port 28774.

Click on Next and Select Allow the connection.

Make sure all three boxes are checked.

Click on Next.
In the Name box type a name for this rule. I.E.- VFBA or evFailover, and click Finish.

Repeat this action for Outbound Rules on port 28774. It is neccessary to have both INBOUND and OUTBOUND rules configured on port 28774 on the SPARE server.

After configuring the inbound and outbound rules you can check the status by doing the following:

Open up task manager and click on the Performance Tab>Open Resource Monitor

Look at the Network Tab and under Listening Ports, verify it says Allowed and Not restricted.

Once any changes were made it is recommended that we restart the SPARE server and EM services and then check EM failover to see if the process can succeed. In some cases it may be necessary to whitelist vfba.exe and ExacqVision program components (core.exe, edvrclient.exe, including .ps and .psi file types & ExacqVision Enterprise System Manager components) when Antivirus software is in use within the configuration of your Antivirus program.

Categories
exacqVision Enterprise

Corrupted Failover group causing EM to drop servers

Issue

When running EM version 22.09 or lower if you see your servers constantly connecting and then disconnecting, please check the EM logs specifically the importer log you should find something similar to this log..

[02/Dec/2022 09:52:52] WARNING [MainProcess:6016,Thread-1:8532,importer.py:292] FailoverProcess died unexpectedly
[02/Dec/2022 09:52:52] INFO [MainProcess:6016,Thread-1:8532,importer.py:575] Shutting down brain.
[02/Dec/2022 09:52:52] INFO [MainProcess:6016,Thread-1:8532,importer.py:444] Cleaning up importer resources
[02/Dec/2022 09:52:52] INFO [MainProcess:6016,Thread-1:8532,importer.py:478] Allowing sub-processes to shutdown’

The failover process is dying constantly which is killing the importer, if you check your failover groups the issue would show as “data restore pending” if you aren’t able to abandon the data please delete the failover group via EM shell.

Product

  • exacqVision Enterprise Manager version 22.09 or lower

Solution

Abandon data from corrupted failover group or delete failover group via EM shell

Categories
Uncategorized

EM failover is not moving the security integration list to the spare server.

Description 

When the protected server goes down, and the EM server moves the configs to the Spare server, it will not move the Security integration list to the Spare server.

Product 

ExacqVision Server 32bit

Steps to Reproduce 

Installing ExacqVision 32bit version and trying to failover to it

Solution

Installing ExacqVision server 64bit version

Categories
Knowledge Support Support exacqVision Enterprise Categories

exacqVision Enterprise Manager Spare shows status of Failover Locked

Description 

An ownership issue between EM and a protected server will often result in the status of the spare and protected server showing “Failover Locked”.

Product 

  • exacqVision Enterprise Manager

Steps to Reproduce 

  • From the exacqVision Enterprise Manager dashboard select Failover Groups
  • Click on the Name of the Failover Group

Expected Results 

  • Spare shows a status of Spare Standby 

Actual Results 

  • Spare shows a status of Failover Locked

Solution

  1. Unmanage systems as described in the Release system management section of the exacqVision Client User Manual.
  2. Restart the exacqVision Server service on the failover and protected servers
  3. Verify the spare is now showing a status of Standby 
Categories
Video Library Configuration exacqVision Enterprise Categories Products

Enterprise Manager Failover

Enterprise Manager Failover NEW

Enable automatic system failover to prevent loss in recording when an ExacqVision server loses network connectivity or is brought down for maintenance.

Learn how to create a failover group for automated failover, perform a manually triggered failover, and how to failback to the original state.

Categories
Knowledge Support Support exacqVision Enterprise Categories Products

Stuck Pinned Spare/Data Restore Pending due to firewall blocking vfba process

Exacq Support has identified an issue preventing fail-back due to VFBA.exe being blocked; which uses port 28774.

Windows Versions

It is recommended to create both inbound and outbound Windows firewall rules on the SPARE server to allow communications through port 28774 for the Windows Firewall.

** Due to issues with the Window OS, disabling Windows Firewall does not allow this thru, and the creation of the FIREWALL rule is required. **

On a Windows Machine, do a search for FIREWALL and click on the option for Windows Firewall with Advanced Security.

Click on the inbound/outbound rules and select NEW RULE from the right.

Highlight Port and click NEXT

Select Port 28774 and click next.

Be sure to repeat this for BOTH Inbound and Outbound connections.

In some cases it may be necessary to “whitelist” ExacqVision components (core.exe, edvrclient.exe, including .ps and .psi file types, and ExacqVision Enterprise System Manager components) Enterprise Manager, Server and Client software along with vfba.exe within any used Antivirus configuration.

Categories
Video Library Demo exacqVision Enterprise Categories Products

2021-07 EM Failover + API Training

Categories
Video Library Demo exacqVision Enterprise Categories Products

2021-07 EM Failover Training

Categories
exacqVision Enterprise Categories Products

ESM: Spare With Expired or Non-Enterprise License Not Monitored

A server without an Enterprise license enabled for health monitoring can be selected as a spare server. The protected server will fail over to this spare server, but Enterprise System Manager (ESM) will not monitor the spare server. This can also occur if the spare server’s Enterprise license has expired.

<br>

As a workaround, do not designate a spare that does not have a license that allows it to be monitored by ESM.