Tag: Failover Group

Categories
Knowledge Support Support exacqVision Enterprise

Failover and Failback Issue in exacqVision Server and Enterprise Manager

Description 

Level 3 Support has identified an issue with Failover functions in ExacqVision Server software manifesting as failback failure. The component responsible is ExacqVision Server software, but errors can be detected from the Enterprise Manager Failover Groups page as well, leading some customers to report this as an issue with the Enterprise Manager software. It is recommended that ExacqVision Server software version 23.03.3.0 be installed on protected and spare NVRs as a workaround until the future release can address this issue.

Product 

  • ExacqVision Server Software Version 23.06.1.0, 23.09.6.0, 23.09.9.0, 24.03.4.0
  • ExacqVision Enterprise Manager Software Versions:
    23.06.2.0 23.09.7.1, 23.09.9.0, 24.03.4.0

Steps to Reproduce 

  1. Upgrade exacqVision Server, Client and Enterprise Manager to versions 23.06.x.x. thru 24.03.x.x from previous versions.
  2. Test Failover/Failback functionality.

Expected Results 

Failover and Failback functions should complete without error.

Actual Results 

Failover and Failback functions do not complete, or complete with errors – sometimes forcing the abandonment of data, which then does not get returned to the protected NVR.

Solution

Prior to taking any of the steps below it is advised to take a backup configuration file of exacqVision Server, along with a backup license key of all NVRs affected.

It is preferred to also have prior configuration backups available for NVRs taken at earlier times.

It is also preferred to have a backup copy of Enterprise Manager’s configuration and/or a copy of the PostgreSQL or Microsoft SQL database.

  1. Stop all Enterprise Manager services to delete this failover group from shell. See: https://support.exacq.com/#/knowledge-base/article/4998
  2. Start all Enterprise Manager services and Delete the NVRs that were included in the failover group from Enterprise Manager.
    (this is a good time to take another backup)
  3. Unmanage the NVRs that were included in the failover group from exacqVision Client > Enterprise > Security tab.
  4. Downgrade (or if using older version, upgrade) exacqVision Servers to exacqVision Server version 23.03.3.0
    (At this step it is advised to restore from a previously taken backup that is known to have no issues at the time it was taken, if available, but this may not always be required)
  5. Return the Spare to an “Active Recorder” in the Client if still showing as spare – remove exacqVision Server software from the spare (not keeping configuration files for future installs) and then install a fresh version of exacqVision Server 23.03.3.0.
  6. If needed you can restore Enterprise Manager from BOTH the previous EM backup AND the Database Backup or by connecting to existing SQL database (if this is verified to have no issues). Alternatively, add back NVRs if issues are known to be present with the backup file or configuration at the time this was taken. This also may not always be required if EM configuration is in current good standing at this step.
  7. Verify no connection issues are present with the Servers and then Re-create the previous Failover Group and test functionality.

AES-733, AES-705

Categories
exacqVision Enterprise

Corrupted Failover group causing EM to drop servers

Issue

When running EM version 22.09 or lower if you see your servers constantly connecting and then disconnecting, please check the EM logs specifically the importer log you should find something similar to this log..

[02/Dec/2022 09:52:52] WARNING [MainProcess:6016,Thread-1:8532,importer.py:292] FailoverProcess died unexpectedly
[02/Dec/2022 09:52:52] INFO [MainProcess:6016,Thread-1:8532,importer.py:575] Shutting down brain.
[02/Dec/2022 09:52:52] INFO [MainProcess:6016,Thread-1:8532,importer.py:444] Cleaning up importer resources
[02/Dec/2022 09:52:52] INFO [MainProcess:6016,Thread-1:8532,importer.py:478] Allowing sub-processes to shutdown’

The failover process is dying constantly which is killing the importer, if you check your failover groups the issue would show as “data restore pending” if you aren’t able to abandon the data please delete the failover group via EM shell.

Product

  • exacqVision Enterprise Manager version 22.09 or lower

Solution

Abandon data from corrupted failover group or delete failover group via EM shell

Categories
Knowledge Support exacqVision Enterprise Support Categories Products

ESM spare server can’t be removed from failover group unless designation as spare is removed

Description

ESM spare failover server can not be removed from failover group on server edit form, unless the “Spare” designation is removed first.

<br>

Version Introduced

v3.0.2.72327

<br>

Platform

ALL

<br>

Steps to reproduce

Create ESM failover group

Add server to group as spare server. CAUTION: This will change server configuation.

On the server detail page. Click edit server icon to go to server edit form.

Attempt to remove spare from failover group by Clicking failover group dropdown

<br>

Expected result

Failover Group dropdown should have “No Failover Group” option.

<br>

Actual result

Failover Group dropdown does not have “No Failover Group” option.

<br>

Work Around

Click “Designate As Spare” dropdown and select “No”.

Click “Failover Group” dropdown, and “No Failover Group” option will be available.

<br>

Version Fixed

none

<br>

Keywords

ESM, failover

Categories
Knowledge Support Support exacqVision Enterprise Categories Products

Enterprise System Manager Shell

To open Enterprise Manager Shell in Windows, run Command Prompt as an administrator and change directories to the installation directory by running the following command:

cd C:\Program Files\exacqvision\enterprisemanager \enterprisesystemmanager\

Then type:

enterprisesystemmanager.exe shell

To open Enterprise Manager Shell in Linux, open a Terminal window and type:

sudo /usr/local/exacq/esm/enterprisesystemmanager shell

<br>

The following commands can be used in the shell:

Exit the Shell.

Exit with ‘Ctrl+Z’

Test SMTP settings

from evemail.models import EmailServer
es = EmailServer.objects.all()[0]
con = es.get_connection()
con.open()

You will see details about any connection errors, or an email connection object that is opened using the information in ESM.

Show the number of emails in the queue

from evemail.models import EventEmail, EventBatchEmail
EventEmail.objects.count()
EventBatchEmail.objects.count()

Delete emails in the queue

from evemail.models import EventEmail, EventBatchEmail
EventEmail.objects.all().delete()
EventBatchEmail.objects.all().delete()

Reset the Enterprise Manager Root Admin user password

*** This should never be given out to a customer ***

Stop all Enterprise Manager services, then enter shell. Once in shell, enter the following commands:

from organization.models import EnterpriseUser
u = EnterpriseUser.objects.get(username="admin")
u.set_password("admin256")
u.save()
import sys
sys.exit()

Start all Enterprise Manager Services

NOTE: It may be necessary to replace the field username=”admin” above with the actual username configured at time of installation, if not using default value. In example: username=”enterpriseadmin”

Deleting a Failover Group 

from failover.models import *
FailoverLog.objects.all().delete() 
FailoverGroup.objects.all() 

This will give you a list of failover groups.
​Then you can delete the group at the proper index using 

FailoverGroup.objects.all()[0].delete() 

​Replacing the number with the index of the group to delete, of course 

import sys
sys.exit()

AES-70

Undeleting data removed using the web GUI

from server.models.core import Server
from django.utils import timezone
readd_date = timezone.now() - timezone.timedelta(days=2)
servers = Server.objects.filter(removalDate__gt=readd_date)
for server in servers:
    server.undelete()

Servers deleted through the GUI are flagged for deletion. As long as the datarolloff retention period has not passed, the servers deleted within the last 2 days can be undeleted with the above commands. You can adjust the days needed if the customer needs something from prior to that by adjusting the “days” number in line 3.