This guide will outline the steps needed to configure the tool SEQ on a PC in order to use scripts provided by Engineers at Exacq Technologies to aid in debugging Enterprise Manager Support Diagnostic Logs.
Pre-Requisites:
This guide will assume you have downloaded Enterprise Manager support diagnostic logs for the purpose of debugging.
This guide will assume you have downloaded the importEMlogs.ps1 powershell script.
This guide will assume you have downloaded the latest available version of the SEQ application program.
Initial installation and configuration steps
You must first install latest version of SEQ on your PC
You must then configure SEQ as follows: 2.A. change to the directory of saved log file in Powershell in example: cd C:\Users\youruser\Downloads\Diagnostics_year_month_day_time\EnterpriseManager
generate an API key by accessing the SEQ UI, located at http://localhost:5341 > Settings > API KEYS
open the importEMlogs.ps1 file provided by Engineering in notepad and observe the following commands:
To set a default server URL and API key, run the commands below: seqcli config -k connection.serverUrl -v https://your-seq-server
in example – you could run the following commands: (use HTTP unless certs are configured) seqcli config -k connection.serverUrl -v http://localhost:5341 seqcli config -k connection.apiKey -v YOURAPIKEY123456789
5. It may be necessary to set the system environmental variable for SEQCLI to work properly. You can do so by searching Windows for ” Environmental Variables “. Open the windows and add the entry for SEQCLI, as listed in the photo below:
Your SEQ instance should now be properly configured for use allowing you to send the Diagnostic Data to the SEQ server now installed.
Additional steps are now required after completion of the initial configuration in order to start using SEQ to debug Enterprise Manager Support Diagnostic Logs, as follows:
1. You must first place importEMlogs.ps1 powershell script provided by Engineering inside the unzipped “EnterpriseManager” directory of the support diagnostic log file you wish to debug.
2. change directory to the proper logfile directory “EnterpriseManager” listed above, and then run the following command:
You may now view the logs in the SEQ UI by navigating to Events and setting the proper filters. More documentation regarding the usage of SEQ can be found at: https://docs.datalust.co/docs/the-seq-query-language <br>
Documentation for SEQ filtering can be found HERE.
Only the version of PostgreSQL installed by the Enterprise Manager installer is supported.
<br>
The exacqVision Enterprise Manager application provides a PostgreSQL installation to store Enterprise Manager data by default.
The PostgreSQL Global Development Group may offer application updates that are not yet included within the ExacqVision Enterprise Manager installer. At times, users will seek to upgrade to the very latest version of PostgreSQL.
Updating the PostgreSQL database application independently from Enterprise Manager updates may break your Enterprise Manager operation and/or result in lost data. Changes made between versions of PostgreSQL and Enterprise Manager have not been tested. Only the version of PostgreSQL installed by the Enterprise Manager installer is supported.
Downgrading ExacqVision Enterprise Manager from 23.06.0.0 to any version lower will prevent users from accessing the ExacqVision Client software due to the migration to AES-128 from ARC4 Encryption methods used on earlier ExacqVision Enterprise Manager versions.
Downgrading or “Rolling Back” Enterprise Manager software from versions 23.06.0.0 and up to a lower version is not recommended due to this encryption migration.
Note: It is advised to take an Enterprise Manager backup of your system prior to attempting any upgrades/downgrades. Best Practices would include taking a database back up of PostgreSQL or Microsoft SQL.
Product
ExacqVision Enterprise Manager versions 23.06.0.0 and higher subsequently released versions.
Steps to Reproduce
Downgrade ExacqVision Enterprise Manager software to any prior version from 23.06.0.0
Expected Results
This downgrade should complete reflecting the new version, and all functionality should remain intact.
Actual Results
ExacqVision Client users will receive the error: “Invalid Username/Password account locked or disabled” upon trying to log in after the downgrade has been performed.
Solution
Do not downgrade from ExacqVision Enterprise Manager versions 23.06.0.0 to a lower version. If you find this needs to happen for an unforeseen reason it is recommended to uninstall the current version of Enterprise Manager 23.06.0.0 or higher, followed by installing the desired legacy version which will require rebuilding the configuration.
Some customers might have High CPU usage on their EM servers. In which the CPU runs at 100%.
Product
EM- Version 23.0.3 and older.
Steps to Identify the Cause
Log into the EM server and validate that the importers are running hard. Normally there will be several importer services opened and being used. You will have one that is taxed more than other as it is the brain importer.
Go to the User Role Section and see if there are any user roles with the Trash Can icon next to it. If so this could be an indicator that there is NO User attached to the User Role. In this case there are three options that can be performed. 1. Add at least 1 user to the user role. 2. Unpromote the user role. 3. Delete the user role if it is not needed.
Once we have identified this is the case for the UserRole we NEED to make sure the Postgres DB is in fact growing.
First we need to stop the EM services.
Open up windows and search for PGadmin and launch the application.
Use the tree to open up the Enterprise folder and log in using the password of postgres .
In the upper portion there is a Tools menu you can select to open a Query.
Run the following query. select count(*) from camera_camerapermission
NOTE* Each Time you run the new Query make sure you do not have the last statement in the query box. Run a new Query. In the query box copy and paste the following in and execute the query. This will take a few minutes if there are a lot of table counts.
If the table shows millions of entry’s we need to run the following script in pgadmin and execute the query.
delete from camera_camerapermission where id in ( select id from ( select id, row_number() OVER w as rnum from camera_camerapermission window w as ( partition by device_id, permission_id order by id ) ) t where t.rnum > 1);
select count(*) from camera_camerapermission;
When the query is done executing the count will show up with a certain number depending on the tables. Take note to the count number. Then start the EM services, and wait a few minutes and execute the query again. select count (*) from camera_camerapermissions The table will grow for about 24 hours.
This should stop the table from growing at an extremely fast rate which taxes the system CPU.
Expected Results
The table should not be growing.
Actual Results
In Fact when there is a UserRole with no user- the table will bloat.
Solution
Engineering has identified the potential cause. This might not be the actual table responsible in all situations. However they have began a new ticket in Jira to investigate this and make sure we put a fail safe into prohibit the creation of an Empty User Role.
Original Issue seen in: Jira- AES-16 Trac 19900 Current Ticket AES-465
IMPORTANT: It is advised to update to 22.3.0 so the customer have the capability of doing a backup, then proceed with the update to the version which uses elastic search.
Prefix: It is not advised to revert back to a version using Apache Solr unless necessary. It is safer to do a fresh install if possible.
WARNING: You must revert back to the SAME EXACT VERSION this is due to the fact that enterprise-holodeck has the same front-end files for that version!
WINDOW:
1.) Stop services before proceeding. net stop enterprise-apache net stop enterprise-elasticsearch net stop enterprise-datarolloff net stop enterprise-sendemail net stop enterprise-importer net stop enterprise-webservice
2.) Remove Elasticsearch. In PS cd “C:\Program Files\exacqVision\EnterpriseManager\elasticsearch\bin\”
PS C:\Program Files\exacqVision\EnterpriseManager\elasticsearch\bin> Type: .\elasticsearch-service-enterprise-remove.bat WARNING: CHECK THAT THE SERVICE ‘enterprise-elasticsearch’ HAS BEEN REMOVED BEFORE PROCEEDING.** PS Type: Get-Service -Name enterprise-elasticsearch You should see output ‘Get-Service : Cannot find any service with service name ‘enterprise-elasticsearch’.’.
cd “C:\Program Files\exacqVision\EnterpriseManager” PS C:\Program Files\exacqVision\EnterpriseManager> rm elasticsearch
3.) Uninstall Apache. In PowerShell as admin go to: C:\Program Files\exacqVision\EnterpriseManager\apache\bin> Type: .\httpd.exe -k uninstall -n enterprise-apache Check that the service ‘enterprise-apache’ has been removed before proceeding.
4.) Backup Apache.
In PowerShell go to: C:\Program Files\exacqVision\EnterpriseManager> Type: mv apache ..\es_apache.old
5.) Revert MediaRoot and StaticRoot with the following.
In PowerShell CD into .\EnterpriseSystemManger\ Type: .\installer.exe decrypt
To edit the file with notepad run the following commands: From EnerpriseSystemManger Type: notepad.exe EnerpriseSystemManger.tmp
You will see something similar in notepad at first.
ONLY CHANGE THE FOLLOWING ITEMS with a text editor. WARNING: Make sure the paths MATCH EXACTLY including forward and backward slashes MediaRoot=C:\Program Files\exacqVision\EnterpriseManager/apache_solr/apache2/htdocs/media StaticRoot=C:\Program Files\exacqVision\EnterpriseManager/apache_solr/apache2/htdocs/static.
Example:
In Power Shell Type .\installer.exe encrypt
Now Type: del EnterpriseSystemManager.tmp
6.) Rename apache_solr-<timestamp>.old. In PowerShell as Admin go to: C:\Program Files\exacqVision\EnterpriseManager> Type: Rename-Item .\apache_solr-20220405160617.old\ .\apache_solr
7.) Remove EM_Documentation.txt. CD into C:\Program Files\exacqVision\EnterpriseManager> Type: del.\EM_Documentation.txt
8.) Run an update to the old version. IMPORTANT: At this point we will run the SAME EXACT VERSION of the installer we migrated from! WARNING: Make sure folders, shells, notepad.exe etc. are closed that are referencing anything inside the EM install directory! WARNING: You must NOT have an ‘apache_solr.old’ folder in the directory.CRITICAL: Run the installer outside of EM when prompted YOU MUST CHECK optional update ‘Apache Solr 7.5.0’ !
9.) Restore all htdocs installers. PowerShell into: cd “C:\Program Files\exacqVision” PS C:\Program Files\exacqVision> cp .\es_apache.old\htdocs\media\installers\*.exe .\EnterpriseManager\apache_solr\apache2\htdocs\media\installers\ Note: This will load the files from step 4. to make sure everything matches.
10.)Clean up firewall rules. Delete inbound firewall rule with name ‘exacqVision Enterprise Manager Apache’.
11.)Confirm downgrade is functional. a.) Navigate to 127.0.0.1:8983 it should open up Apache Solr management portal. b.) Confirm the following rules exist. solrApache solrJetty c.) Navigate to {EM_HOST}:{EM_PORT} and login to EM.
12.)Load domain users into Apache Solr. Navigate to {EM_HOST}:{EM_PORT} and login to EM. Go to the sidebar and click ‘Domain’. If a domain is being used click ‘refresh now’ this will load domain users back into Apache Solr.
Prefix: It is not advised to revert back to a version using Apache Solr unless necessary. WARNING: You must revert back to the SAME EXACT VERSION this is due to the fact that enterprise-holodeck has the same front-end for that version!
1.)Stop services before proceeding. sudo systemctl stop enterprise-elasticsearch sudo systemctl stop enterprise-datarolloff sudo systemctl stop enterprise-sendemail sudo systemctl stop enterprise-importer sudo systemctl stop enterprise-webservice Note: Stopping enterprise-webservice will also stop the Apache service as well.
3.)Backup Apache. Note: Apache in Ubuntu does not have any files configured outside of the Apache directory. Create a manual backup of Apache for later and move it outside of the esm directory. sudo mv /usr/local/exacq/esm/apache /usr/local/exacq/es_apache.old
4.)Reload services for systemctl. sudo systemctl daemon-reload sudo systemctl status enterprise-elasticsearch You should see output ‘Unit enterprise-elasticsearch.service could not be found’.
5.)Revert MediaRoot and StaticRoot. cd /usr/local/exacq/esm exacqu@ubuntu:/usr/local/exacq/esm$ ./installer decrypt
6.)Rename apache_solr-<timestamp>.old. cd /usr/local/exacq/esm exacqu@ubuntu:/usr/local/exacq/esm$ sudo mv apache_solr-<timestamp>.old apache_solr
7.)Run an update to the old version.
IMPORTANT: At this point we will run the SAME EXACT VERSION of the installer we migrated from! WARNING: You must NOT have an ‘apache_solr.old’ folder in the directory.
CRITICAL: The installer MUST NOT BE run in silent mode since silent mode WILL NOT PROMPT for the optional updates! CRITICAL: Run the installer outside of EM when prompted YOU MUST CHECK optional update ‘Apache Solr 7.5.0’!
8.)Restore all htdocs installers. sudo cp /usr/local/exacq/es_apache.old/htdocs/media/installers/*.deb /usr/local/exacq/esm/apache_solr/apache2/media/installers/
9.)Confirm downgrade is functional. a.)Navigate to 127.0.0.1:8983 it should open up Apache Solr management portal. b.)Firewall rules must be applied for Apache Solr management portal. sudo iptables -S
Confirm the following rules exist: -A INPUT -s 127.0.0.1/32 -p tcp -m tcp –dport 8983 -j ACCEPT -A INPUT -p tcp -m tcp –dport 8983 -j DROP
c.) Navigate to {EM_HOST}:{EM_PORT} and login to EM.
10.)Load domain users into Apache Solr. Navigate to {EM_HOST}:{EM_PORT} and login to EM. Go to the sidebar and click ‘Domain’. If a domain is being used click ‘refresh now’ this will load domain users back into Apache Solr.
When changing a User from Local to Local (Managed) i.e. Unmanaged to Managed user in exacqVision Enterprise Manager the crying stick man appears immediately after applying the change.
Product
ExacqVision Enterprise Manager
Steps to Reproduce
From the EM Dashboard, select Users
On the Users page select UNMANAGED SERVER USERS to display a list of unamanaged users
Locate the desired user and click the corresponding Manager user icon
Expected Results
The change applies and you are still on the EM Users Page
Actual Results
Crying Stickman page appears
Solution
The change is actually made, the issue is with the redirect back to EM, to verify the change was made.
Click the Return to Dashboard icon on the crying stick man page.
From the EM Dashboard, selects Users
The user should no longer be listed on the Unamanged Server Users page but on the Users page instead
Also check the user via the ExacqVision Client
It should now show Local (Managed)
The fix will be in the next release of ExacqVision Enterprise Manager version 22.06
NOTE: The following issue was resolved with the release of exacqVision Enterprise Manager v21.09.1
08/27/2021 v21.09.1 * Bug Fix - Fix failover when password strengthening is enabled on proteced server
When either trying to do a Manual Failover to test or a Protected Server goes offline and you may get a message in EM that states Data Restore Pending and it is stuck in this state…
This is more than likely due Password Strengthening being Enabled on the ExacqVision Server…
“Once this is Enabled, passwords are salted and strengthened to prevent brute force attacks. Augmented password authentication protects against man-in-the-middle and client masquerading attacks.”
However, this can create problems with the way EM transmits the configuration back and forth to each other. If you have plans to use EM for Failover/Failback protection…DO NOTENABLE this feature.
Once it is Enabled, you cannot simply just disable it; since it modifies your Useraccounts.xml file.
WORKAROUND: Currently the only way to resolve this is to completely recreate a new Useraccounts.xml file or import an older system configuration from before you Enabled Password Strengthening.
In order to create a new Useraccounts.xml file I recommend going to Users in the ExacqVision Client Software and writing down your users and what type of groups they are in and the properties of those groups.. (Do they have customer user groups and privileges configured)
Download the latest ExacqVision Server Version and save it on the Spare and Protected Server
Stop the ExacqVision Server Service on the Spare and on the protected Servers you are working with; otherwise it will keep overwriting the useraccounts.xml file.
From Windows Services, find ExacqVision Server and right click on it and choose StopAfter the ExacqVision Server Service has stopped.
Browse to C:\Program Files\ExacqVision\Server and locate the file Useraccounts.xml and rename it to useraccounts.old
Install the downloaded file of the ExacqVision Server you have; it will prompt you to create a new password for the admin account (Try to use the same admin password as before)
After the ExacqVision Server is installed, In the client software: Login with the new Username and Password under Add Systems…Make sure it says connected
Repeat the same process for the Spare Server
Start the ExacqVision Server Service on the Spare and Protected server’s after you successfully have logged in to both and they show connected.
Back in EM, under the Failover Groups after a minute or two you should see the Satus of the Spare/Protected change (See Pics Below)
The compatibility Warning simply means there is a difference between the configuration or hardware of the Spare vs Protected Server
Once you are back at the above screen you can proceed with trying to either do a Manual Failover or simply stop the ExacqVision Server Service on the Protected Server to simulate a down server.
Once it fails over and you have verified it failed over, fail it back by either taking it out of Manual Failover or Starting the ExacqVision Server Service
Exacq Support has identified an issue preventing fail-back due to VFBA.exe being blocked; which uses port 28774.
Windows Versions
It is recommended to create both inbound and outbound Windows firewall rules on the SPARE server to allow communications through port 28774 for the Windows Firewall.
** Due to issues with the Window OS, disabling Windows Firewall does not allow this thru, and the creation of the FIREWALL rule is required. **
On a Windows Machine, do a search for FIREWALL and click on the option for Windows Firewall with Advanced Security.
Click on the inbound/outbound rules and select NEW RULE from the right.
Highlight Port and click NEXT
Select Port 28774 and click next.
Be sure to repeat this for BOTH Inbound and Outbound connections.
In some cases it may be necessary to “whitelist” ExacqVision components (core.exe, edvrclient.exe, including .ps and .psi file types, and ExacqVision Enterprise System Manager components) Enterprise Manager, Server and Client software along with vfba.exe within any used Antivirus configuration.
PostgreSQL 9.6 is currently bundle with exacqVision Enterprise Manager. This version will reach end of life in November 2021.
The PostgreSQL Global Development Group will release the final update for 9.6 on November 11, 2021. See their versioning page for details at https://www.postgresql.org/support/versioning/
