Author: Brandon Powell

Categories
exacqVision Client Linux x64 exacqVision Client Windows Knowledge Support Support exacqVision Client

Remote Connections node not visible in ExacqVision Client

Some ExacqVision Client users report being unable to see the “Remote Connections” tab in some Environments.

NOTE: This is typically an Operating System or Domain permissions issue.

Products 

  • ExacqVision Server Version 24.09.4.0 and up.
  • ExacqVision Client Version 24.09.3.0 and up.

Steps to Reproduce 

In a restricted environment, or with a restricted user, launch the ExacqVision Client from the Desktop Icon, or folder.

Expected Results 

When signed in as an administrator to the ExacqVision Client software, the “Remote Connections” tab is visible in the navigation tree.

Actual Results 

The “Remote Connections” node does not populate in the navigation tree as expected, even when signed into the ExacqVision Client as an administrator.

Solution

Windows:
Right-Click the icon and select the option to “Run as Administrator” or, if the option exists, “Run as Elevated”.

Linux:
Invoke the ExacqVision client with sudo commands or as the root user.

<br>

Categories
exacqVision Server Linux x64 Support exacqVision Hardware

Some Exacq Ubuntu images may disable Unattended-Upgrades package features by default

Ubuntu’s unattended-upgrades commands and package features may not work as expected when using an Exacq Ubuntu image on ExacqVision hardware. By default on Ubuntu Exacq images, unattended-upgrades package features are purposely disabled from running due to the potential of losing video – and how the use of this feature cannot guarantee uptime of the ExacqVision Server software host.

This unattended-upgrades package is usually intended to install security updates automatically, by default, so that machines that are not proactively maintained do not become a liability.

For more information regarding ExacqVision’s stance on updating or upgrading operating systems on ExacqVision NVRs please see:
https://support.exacq.com/#/knowledge-base/article/5144

Product 

  • ExacqVision Server Hardware
  • ExacqVision Server Software
    Ubuntu Operating Systems only.

Step to Reproduce 

Schedule unattended-upgrades package features as listed by Ubuntu documentation.
See: https://ubuntu.com/blog/3-ways-to-apply-security-patches-in-linux

Notes:
The unattended-upgrades package is over a decade old, and is enabled by default on all currently supported versions of Ubuntu offered directly from Ubuntu. The ExacqVision image is configured differently, with this disabled as to prevent loss of video.

Expected Results 

Unattended-upgrades package features should work as expected when scheduled.

Actual Results 

Unattended-upgrades package features fail without dialogue.

Solution

1) Run the terminal commands below:

sudo apt-get -y install unattended-upgrades

sudo apt-get install apt-listchanges

sudo dpkg-reconfigure –priority=low unattended-upgrades

2) Create and/or edit the file at /etc/apt/apt.conf.d/10periodic and /etc/apt/apt.conf.d/20auto-upgrades with the following:

APT::Periodic::Enable “1”;

APT::Periodic::Update-Package-Lists “1”;

APT::Periodic::Download-Upgradeable-Packages “1”;

APT::Periodic::AutocleanInterval “1”;

APT::Periodic::Unattended-Upgrade “1”;

3) Run the following terminal commands:

systemctl unmask apt-daily.service

systemctl unmask apt-daily-upgrade.service

systemctl enable apt-daily.service

systemctl enable apt-daily-upgrade.service

4) sudo systemctl edit apt-daily-upgrade.service

Edit and Comment out “ExceStartPre=” line,

repeat this action for:

sudo systemctl edit apt-daily-upgrade.service

5) Run the following terminal command:

sudo systemctl daemon-reload

6) Run all of the following terminal commands in order listed:

systemctl enable apt-daily.timer

systemctl enable apt-daily-upgrade.timer

systemctl start apt-daily.timer

systemctl start apt-daily-upgrade.timer

systemctl start apt-daily.service

systemctl start apt-daily-upgrade.service

7) systemctl list-timers to see if the timers are active and when they run next

8) tail -n 100 /var/log/unattended-upgrades/unattended-upgrades.log to see if it ran at the time start service was ran

<br>

Categories
exacqVision Enterprise Windows Knowledge Support Support exacqVision Enterprise Products

Enterprise Manager memory resource usage high due to importer

Exacq Support has identified an issue with Enterprise Manager importer processes using an excessive amount of memory resource, often requiring reboot due to system becoming unresponsive when installed on some Windows machines. This is estimated to affect versions 23.06.2.0 thru latest versions of ExacqVision Enterprise Manager for Windows. Developers are aware and working on resolving these issues.

NOTE: This high memory resource usage often corresponds with errors seen in importer log exceptions. You can trace the PID this process starts up on in importer-error logs to the process in task manager in Windows.

Product 

  • ExacqVision Enterprise Manager
  • 23.06.2.0 – 24.06.2.0

Solution

Engineering has developed a temporary workaround consisting of a PowerShell script developed to scan for and kill certain importer processes when their memory usage is at unacceptable levels. This script runs every 5 minutes, unless code is altered. This script should only be used as a last resort for Support personnel to deploy when all troubleshooting and other remediation steps have been exhausted. In some cases it may be recommended to alter the MemoryCeiling function of this script to better suite certain customer scenarios.

It is also recommended that this script be triggered as a scheduled task from task scheduler. This often requires that Execution Policies be bypassed to launch successfully on reboot. Detailed instructions are below:

1) Download the importerMonitor.ps1 script from:

https://support.exacq.com/#/file-manager/33becef0-0ab8-4b0f-9127-94884ac262a6/em-dev-build

2) Unzip the file and place the importerMonitor.ps1 file in a directory. Often times, it is best to place this in: “%\Program Files\exacqvision\”

3) Open Task Scheduler in windows and Create Task, then name task importerMonitor and configure an admin user and to run whether user is logged on or not. Be sure to check the box for Run with highest privileges, and configure for OS in use.

4) Under triggers, select New… Then configure for “At Startup”.

5) Under the Actions tab, select New… and configure as follows:
– Start a Program
– Program/script: Powershell.exe
– Add arguments: -ExecutionPolicy Bypass – File “C:\Program Files\exacqVision\importerMonitor.ps1”

6) Configure the conditions tab as pictured:

7) Configure the Settings tab as pictured:

8) Reboot to test the Task at startup, and verify it worked as expected by observing the History tab in Task Scheduler for this task:



<br>

Categories
Knowledge Support exacqVision Enterprise Support

SEQ – A guide to initial configuration for debugging Enterprise Manager Logs  

This guide will outline the steps needed to configure the tool SEQ on a PC in order to use scripts provided by Engineers at Exacq Technologies to aid in debugging Enterprise Manager Support Diagnostic Logs.

Pre-Requisites:

  • This guide will assume you have downloaded Enterprise Manager support diagnostic logs for the purpose of debugging.
  • This guide will assume you have downloaded the importEMlogs.ps1 powershell script.
  • This guide will assume you have downloaded the latest available version of the SEQ application program.

Initial installation and configuration steps

  1. You must first install latest version of SEQ on your PC
  2. You must then configure SEQ as follows:
    2.A. change to the directory of saved log file in Powershell
    in example:
    cd C:\Users\youruser\Downloads\Diagnostics_year_month_day_time\EnterpriseManager
  3. generate an API key by accessing the SEQ UI, located at http://localhost:5341 > Settings > API KEYS
  4. open the importEMlogs.ps1 file provided by Engineering in notepad and observe the following commands:

    To set a default server URL and API key, run the commands below:
    seqcli config -k connection.serverUrl -v https://your-seq-server

    seqcli config -k connection.apiKey -v your-api-key

in example – you could run the following commands:
(use HTTP unless certs are configured)
seqcli config -k connection.serverUrl -v http://localhost:5341
seqcli config -k connection.apiKey -v YOURAPIKEY123456789

5. It may be necessary to set the system environmental variable for SEQCLI to work properly. You can do so by searching Windows for ” Environmental Variables “. Open the windows and add the entry for SEQCLI, as listed in the photo below:

Your SEQ instance should now be properly configured for use allowing you to send the Diagnostic Data to the SEQ server now installed.

Additional steps are now required after completion of the initial configuration in order to start using SEQ to debug Enterprise Manager Support Diagnostic Logs, as follows:

1. You must first place importEMlogs.ps1 powershell script provided by Engineering inside the unzipped “EnterpriseManager” directory of the support diagnostic log file you wish to debug.

2. change directory to the proper logfile directory “EnterpriseManager” listed above, and then run the following command:

.\importEMlogs.ps1 -ProcessLogs -OverwriteExisting -Environment "CaseNo/AesNo/tagline" -StartFromLastInstall -SendToSEQ

You may now view the logs in the SEQ UI by navigating to Events and setting the proper filters. More documentation regarding the usage of SEQ can be found at: https://docs.datalust.co/docs/the-seq-query-language
<br>

Documentation for SEQ filtering can be found HERE.

Categories
Plugin Exclude from Global Search Log Level Error exacqVision Server Categories Logs Illustra3PI

Camera did not send valid quality for codec H264

Camera did not send valid quality for codec H264

Description

exacqVision Client connection status for camera shows “unable to create video stream” status.

Notes

Often this is a firmware compatibility issue. There are some cases that this has been resolved by factory defaulting the camera and/or removing and recreating illustra3pi.xml file in server folder.



Categories
exacqVision Enterprise Windows exacqVision Enterprise Linux exacqVision Enterprise Other exacqVision Enterprise 23.09 exacqVision Enterprise 23.09 Knowledge Support Support exacqVision Enterprise

Failing email notifications after update to Enterprise Manager version 23.09.9.0

April 4th, 2024

Enterprise Manager may fail to send email notifications after update to 23.09.9.0.

ExacqVision support has identified an issue with failing email notifications after updating Enterprise Manager from a previous version to version 23.09.9.0 and up. This occurs when checking “Anonymous” as the send user, or additional configuration required for Exchange servers has not been completed.

NOTE: Email notifications have been confirmed to succeed using same SMTP configuration on version 23.06.2.0 but fails when update to 23.09.9.0 is completed.

Product 

exacqVision Enterprise Manager Version 23.09.9.0

Steps to Reproduce 

Update Enterprise Manager to version 23.09.9.0 from a previous version

Expected Results 

Email notifications should continue to succeed with existing configuration.

Actual Results 

Email notifications may fail with all or one of the following log errors:
Failed email verification: SMTP AUTH extension not supported by server.

Failed email verification: (550, b'5.7.60 SMTP; Client does not have permissions to send as this sender')

Failed email verification: STARTTLS extension not supported by server.

Failed email verification: (535, b'5.7.3 Authentication unsuccessful')

Solution

  1. disable “anonymous users” as send from address in Enterprise Manager and replace with an authorized SMTP user.

2. Turn on Basic Authentication method in Exchange/ SMTP server if option is available.

3. Make sure the user account sending notifications from Enterprise Manager has SMTP Authentication turned on in their exchange profile, and also that the “Send From” address used in Enterprise Manager was added to the exchange user as an email alias.

Categories
Knowledge Support Support exacqVision Enterprise

Failover and Failback Issue in exacqVision Server and Enterprise Manager

Description 

Level 3 Support has identified an issue with Failover functions in ExacqVision Server software manifesting as failback failure. The component responsible is ExacqVision Server software, but errors can be detected from the Enterprise Manager Failover Groups page as well, leading some customers to report this as an issue with the Enterprise Manager software. It is recommended that ExacqVision Server software version 23.03.3.0 be installed on protected and spare NVRs as a workaround until the future release can address this issue.

Product 

  • ExacqVision Server Software Version 23.06.1.0, 23.09.6.0, 23.09.9.0, 24.03.4.0
  • ExacqVision Enterprise Manager Software Versions:
    23.06.2.0 23.09.7.1, 23.09.9.0, 24.03.4.0

Steps to Reproduce 

  1. Upgrade exacqVision Server, Client and Enterprise Manager to versions 23.06.x.x. thru 24.03.x.x from previous versions.
  2. Test Failover/Failback functionality.

Expected Results 

Failover and Failback functions should complete without error.

Actual Results 

Failover and Failback functions do not complete, or complete with errors – sometimes forcing the abandonment of data, which then does not get returned to the protected NVR.

Solution

Prior to taking any of the steps below it is advised to take a backup configuration file of exacqVision Server, along with a backup license key of all NVRs affected.

It is preferred to also have prior configuration backups available for NVRs taken at earlier times.

It is also preferred to have a backup copy of Enterprise Manager’s configuration and/or a copy of the PostgreSQL or Microsoft SQL database.

  1. Stop all Enterprise Manager services to delete this failover group from shell. See: https://support.exacq.com/#/knowledge-base/article/4998
  2. Start all Enterprise Manager services and Delete the NVRs that were included in the failover group from Enterprise Manager.
    (this is a good time to take another backup)
  3. Unmanage the NVRs that were included in the failover group from exacqVision Client > Enterprise > Security tab.
  4. Downgrade (or if using older version, upgrade) exacqVision Servers to exacqVision Server version 23.03.3.0
    (At this step it is advised to restore from a previously taken backup that is known to have no issues at the time it was taken, if available, but this may not always be required)
  5. Return the Spare to an “Active Recorder” in the Client if still showing as spare – remove exacqVision Server software from the spare (not keeping configuration files for future installs) and then install a fresh version of exacqVision Server 23.03.3.0.
  6. If needed you can restore Enterprise Manager from BOTH the previous EM backup AND the Database Backup or by connecting to existing SQL database (if this is verified to have no issues). Alternatively, add back NVRs if issues are known to be present with the backup file or configuration at the time this was taken. This also may not always be required if EM configuration is in current good standing at this step.
  7. Verify no connection issues are present with the Servers and then Re-create the previous Failover Group and test functionality.

AES-733, AES-705

Categories
Knowledge Support Support Categories exacqVision Hardware Products

Re-image fails on Linux G-Series Micro units above Ubuntu version 18.04

exacqVision G-Series Micro units may fail to write Exacq’s custom image of Ubuntu OS versions back to drive if the version is above Ubuntu 18.04 LTS.

Product

G-Series Micro

Steps to Reproduce

  1. Re-image a G-Series Micro Unit to an Ubuntu OS image above 18.04 LTS

Expected Results

The sysprep process should not fail

Actual Results

The sysprep process fails

Solution

  1. Re-image to Ubuntu 18.04 LTS from sysprep USB key instead of using 20.04 LTS or 22.04 LTS image.
  2. Run the following commands to step up in OS versions:
    sudo apt update
    sudo apt upgrade
    sudo apt dist-upgrade
    sudo apt autoremove
    sudo do-release-upgrade -f DistUpgradeViewGtk3

<br>

Reference: AES-737

Categories
Knowledge Support exacqVision Enterprise exacqVision Client exacqVision Server

Updating Enterprise Manager to 23.09.5.0 can cause legacy server versions to have user account log in failure.

Support has identified an issue with updating Enterprise Manager to version 23.09.5.0 prior to running updates to ExacqVision server 23.09.6.0. This issue will manifest after updating Enterprise Manager by users experiencing disconnection from the client with errors ” log in failed / invalid username or password or account locked or disabled” or ” log in failed – invalid response”.

Product

  • ExacqVision Enterprise Manager
  • ExacqVision Server

Steps to Reproduce

  • Update Enterprise Manager to version 23.09.5.0 while leaving ExacqVision server versions on 23.06 or lower.

Expected Results

– Users should remain logged in successfully.

Actual Results

– Users will be disconnected from the Client with error – ” log in failed / invalid username or password or account locked or disabled” or ” log in failed – invalid response”

Solution

Engineering is currently working on a patch to resolve this issue.

For customers that updated from 23.06 to 23.09 prior to the patch, installing the version with the patch should fix all login issues customers are facing.

For customers that performed a fresh install of 23.09 prior to the patch, user server passwords will become scrambled and customers will need to change/reset user passwords.

Until the patch is released, if customer has an Enterprise Manager backup that contains no issues from 23.06 that can be applied, the following steps can be taken:

  1. Take a backup from 23.09 prior to making any changes.
  2. Access a recent prior backup off of Enterprise Manager (this must be taken on version 23.06 previously).
  3. Completely remove Enterprise Manager.
  4. Install Enterprise Manager previous version 23.06.0.0.
  5. Follow on screen prompts during install.
  6. Login to Enterprise Manager as the root admin user.
  7. Upload previously taken backup using the upload button in the backups screen in Enterprise Manager user interface.
  8. Wait for this to complete.
  9. Restore from uploaded backup.
  10. Wait for restoration completion.
  11. Confirm log-in capability has been restored.
  12. Re-setup scheduled backups as these will be defaulted.

    It should be noted that downgrading is not a normal solution, and should be prevented with Enterprise Manager. These instructions are only for situations where the customer has already performed this update and is now locked out of Client software and critically needs to be restored in a timely manner.
Categories
exacqVision Enterprise Windows exacqVision Enterprise Linux Knowledge Support Support exacqVision Enterprise exacqVision Client exacqVision Server exacqVision Webservice

Outbound Connections tab in Client shows intermittent disconnecting then connecting

Some servers connecting to Integrator Services Portal or Enterprise Manager may show disconnected on outbound connections tab, then connected, and then disconnect again intermittently.

Product 

ExacqVision Server software
ExacqVision Client software

Steps to Reproduce 

  • Connect the ExacqVision server to ISP or EM via the outbound connections tab in the Enterprise section or Configure System section of the ExacqVision Client software.

Expected Results 

Status should remain “Connected”

Actual Results 

Status will show “Connected”, then momentarily show “disconnected”, then back to “Connected”, continuously cycling through these statuses.

Solution

  1. Observe if streampi.xml lists the MAC address for SerialNumber value & shows “unknown” as ModelNumber value (if it doesn’t, could be a different issue).
  2. Remove server from ISP or EM instance and restart ALL EM services.
  3. Remove entry from outbound connections tab in client for this server
  4. Edit streampi.xml to reflect proper original values for both SerialNumber and ModelNumber and save the changes.
  5. Restart ExacqVision Server service
  6. Add server back to ISP/EM instance as outbound connection with proper serial number to match what is in streampi.xml (HAS to be SerialNumber).  
  7. Click on server details > edit > verify proper serial number is listed. If this is not the proper value ( this may show a different value than what you would expect to see for this server), change it to proper serial number and save the changes.
  8. Add EM/ISP instance back to outbound connections tab of client.
  9. Restart ExacqVision Webservice service.
  10. Await “Connected” status to verify resolution.