Categories
Knowledge Support Support exacqVision Server Categories Products

Troubleshooting Record Content Age Event

Record Content Age is available when the Health event type is selected on the Event Linking page in exacqVision Client. Record Content Age is triggered when the configuration setting for the Desired Content Age (which is configurable on the Storage configuration page) is greater than Oldest Content as reported on the Storage page. Oldest Content reported does not account for any “At Least” storage rules or bookmarked video/data.

<br>

Troubleshooting Goal

Ensure that your actual system configuration is comparable with the configuration provided to the Exacq Configuration Calculator (https://exacq.com/config/). If your actual configuration exceeds the planned configuration, it is expected to have a reduced capacity for recorded data. The pages required in this goal can be accessed in the exacqVision Client tree as shown here:

<br>

<br>

Days Stored: Ensure that the Days Stored provided to the calculator is greater than the Oldest Content reported by the system on the Storage page of exacqVision client.

Number of connected cameras: 
How many cameras are connected to your system? Make sure it is fewer than what is specified on the calculator. Connecting more cameras will result in reduced age in overall content.

Frame Rate: Check the value of Frame Rate for all your cameras and video inputs on the Camera Recording page. Higher frame rates result in more disk usage per unit of time. A frame rate of 10-15 fps is considered sufficient for most customer’s needs.

<br>

<br>

Format: Check the Format setting on the Camera Recording page. H.264 or MPEG4 formats, if available on a camera, result in more compact disk storage than JPEG, increasing longevity of overall recorded data.

Resolution and Quality: Check the Resolution and Quality settings on the Camera Recording page for your cameras. Higher resolution and higher quality result in greater disk usage.( You can open the camera’s configuration page in exacqVision Client to see the average image size). Reducing resolution and quality settings increases the recorded media longevity. Try adjusting resolution and quality settings on individual camera to see how it affects video viewing experience before deciding on a final setting. Quality can also be adjusted on an individual camera’s settings page.

Recording Schedule: Check the Schedule configuration. Free Run recording (colored green) requires the most disk space compared with Alarm or Motion Recording, which are recorded only when predefined activity occurs. Reducing the amount of free run recording increases recording longevity.

<br>

<br>

Motion Recording: Determine whether any camera configured for motion recording is recording content when no actual motion is present in the scene. You can investigate this by executing a search during the times when no motion is expected. If video results are returned, our motion recording needs to be adjusted to record only the content that you care about.

*Moving objects present in the scene (like a tree or traffic) that you do not care about (but their motion triggers recording) can be masked. Alternatively, you can also adjust motion window to include the area in the camera scene that is important for recording. Motion Mask and Motion Window configuration are available on a camera’s settings page.

*A camera’s sensitivity to light can also trigger motion. You can adjust the Sensitivity configuration on an camera’s individual settings page.

*Make sure all day/night cameras are switching to night mode as needed.

*Make sure continuously touring PTZ cameras are not configured to record motion (unless required).


Alarm Recording: 
If any of your video sources are configured to record based on an alarm condition, you can check whether event linking configuration is triggered too frequently, resulting in unwanted recorded content. On the Search for Events page, search for events during the time when you know a particular event should not happen. Reducing pre- and post- trigger settings will result in reduced disk space usage.

<br>

How to determine which device takes up the most space

If your system consists of many different IP camera models, the following command-line command might be useful in identifying which media source to investigate based on the device that takes up the most storage.

Note for Windows servers:
 The commands below have been tested using PowerShell 2.0 on Windows 7 SP1 32-bit. Earlier versions of Windows should be able to download PowerShell 2.0 from here: http://support.microsoft.com/kb/968930

Execute the following command on the command prompt to list existing manufacturer-device-storage rule file naming patterns on your ev server:

Linux:

find /mnt/edvr/1/2013 -type f -name “*.ps” -printf “%f\n” | sed ‘s/^…..//’ | sort | uniq


Windows:

get-childitem D:\2013 -include “*.ps” -name -recurse | %{$_ -replace “^.*\d{4}-“, “” } | sort | unique


NOTE: Those commands can require more than ten minutes to complete, depending on the size of your system and drive speed.

Your output might look something like this:


00070480.ps

00150280.ps

00160280.ps

00160380.ps

00180280.ps

00190280.ps

00190380.ps

00190480.ps

00190580.ps

Refer to the KB article on exacqVision file naming (https://exacq.com/kb/?kbid=30471), which should help you understand how exacqVision’s file names translate to specific cameras on your system.

For each of the reported devices in the list, you can see the amount of storage that a particular device uses. For example, for device with 00150280 pattern:


Linux:

find /mnt/edvr/1/2013 -name “*00150280.ps” -exec ls -l ‘{}’ \; | awk ‘BEGIN {sum=0} {sum+=$5} END{print sum}’


Windows:

get-childitem D:\2013 -include “*00150280.ps” -recurse | measure-object -property length -sum | %{ $_.Sum; }

The resulting output is the total number of bytes used by a media device on your disk. If you find the media input that is responsible for the largest usage, you can then follow the adjustment recommendations to reduce space demands of this source.

<br>

Using Storage Rules to Increase Oldest Content

If you find a media source on your server that requires high resolution recording, but you do not necessarily need to preserve those recordings for a long time, you can use storage rules to control recorded age for a particular input.

*On the Storage page, select the desired input from the Expiration Configuration panel and click Add.

*Select the newly added input on the right-hand side panel and select the At Most in the Type drop-down box.

*Select the desired number of days.

*Click Apply to save the configuration.

Note for exacqVision System Manager (ESM) Users

ESM might report content age alarms when the system is not actually in an alarmed state. Upon startup in exacqVision Server versions before 6.0, the server software assumes a non-alarmed state for the content age alarm and sends notification to ESM after it becomes alarmed. If you stop the exacqVision service while in a content age alarm state and then start the service again expecting the alarm to be cleared, you might continue to see an active content age alarm in ESM (but not exacqVision Client). To work around this, force the server into a content age alarm state and verify that the alarm is reported in exacqVision Client. Then adjust the Desired Content Age setting to turn off the alarm. This should also clear the alarm state in ESM.

<br>

Troubleshooting-Record-Content-Age-Event.pdf
Categories
Knowledge Support Support exacqVision Server Categories Products

Windows-based ExacqVision Systems Must Record Directly to Fixed Drives

The Windows version of ExacqVision Server must record directly to drives that Windows identifies as fixed drives, such as iSCSI drives.

Network-attached storage (NAS), or mapped drives, cannot be used for direct recording because the ExacqVision system cannot detect whether the drives are being used for other purposes. Such conflicts can cause recording issues on the ExacqVision system.

However, you can archive to NAS (network-attached storage) drives. More information can be found in the ExacqVision User Manual, which can be downloaded here.

<br>

Categories
Knowledge Support Support exacqVision Server Categories Products

Set iSCSI Storage to Start Before ExacqVision Server in Windows

If you are recording video to an iSCSI target in Windows, it is important to ensure the iSCSI device is available in the operating system before the ExacqVision Server starts. Please follow the instructions from the Microsoft iSCSI Software Initiator Version 2.x Users Guide:

Note: The following information was written and published by Microsoft Corporation, which is responsible for the accuracy of the information.

<br>

Running automatic start services on iSCSI disks

The disk startup sequence in Windows when using the Microsoft iSCSI software initiator kernel mode driver is different from the startup sequence when using an iSCSI or other HBA. Disks exposed by the Microsoft iSCSI initiator kernel mode driver are available for applications and services much later in the boot process and in some cases might not be available until after the service control manager begins to start automatic start services. The Microsoft iSCSI initiator service includes functionality to synchronize automatic start services and the appearance of iSCSI disk. The iSCSI service can be configured with a list of disk volumes that are required to be present before the start of automatic start services.

In order to install automatic start services on volumes created from iSCSI disks you need to follow the following steps:

  1. Login to all of the targets your machine will be using. Ensure that these are the only targets that are logged in. Be sure that you have also made them persistent logins by using the iscsicli command PersistentLoginTarget or clicking on the appropriate option in the control panel applet.
  2. Configure all volumes on top of the disks using Disk Administrator
  3. Use the iscsicli commands BindPersistentVolumes, AddPersistentVolume, RemovePersistentVolume and ClearPersistentVolumes (or appropriate button in the control panel applet) to allow the iSCSI service to configure the list of persistent volumes

On Windows 2000 only, make the automatic service dependent upon the msiscsi service.

Note: If you have made any services dependent upon the Microsoft iSCSI initiator service, you will need to manually stop the service before upgrading or removing the Microsoft iSCSI Software initiator package.

<br>

Categories
Knowledge Support Support exacqVision Client Categories Products

exacqVision Video File Naming Format

204000160380.ps

min/sec manufacturer device id storage rule

<br>

exacqVision files are stored on each drive enabled for recording in a directory structure as follows:

  • Year (such as 2018)
  • Month (such as 01 for January)
  • Date (such as 30 for the 30th)
  • Hour (such as 23 for 23:00 or 11 p.m.)

<br>

Within each directory, video is stored in files with duration of five minutes. Each device (IP camera, compression board, encoder) has its own files. The file naming convention is as follows: MMSS-XXXXYYZZ.ps and MMSS-XXXXYYZZ.psi. The *.ps file contains the actual compressed video data, and the *.psi file is an index for frame locations within the associated *.ps file. Here is the meaning of each part of the filename:

MMSS is the minutes and seconds which the file begins.

XXXX is the four-hex-digit plug-in ID associated with the recording device. The first X is normally 0, with additional multistreaming contexts incremented up to 7. The remaining three Xs are based on the camera manufacturer, such as 0004 for an eDVR board or 0018 for Arecont.
See also: ExacqVision IP Plugin IDs

EXAMPLE:A single-stream Arecont camera would have a plug-in ID of 0018, but a multistream Arecont camera would have plug-in IDs of 0018, 1018, 2018, and so on as high as 7018. Even if a context is deleted, the file will remain on the system as needed.

YY is the device number within the plugin family.

ZZ is the storage rule associated with the video file. The value is 80 if there is no storage rule associated with this video recording.

<br>

Categories
Knowledge Support Support exacqVision Server Categories Products

How do I monitor Disk I/O?

Disk I/O (input/output) plays a crucial role in how you experience the exacqVision software. If disks are unable to handle the amount of data being written to them, you could see reliability issues.

To help pinpoint the source of these issues on your server, use the following steps as a guide based on your system operating system.

<br><br>

Linux (Ubuntu)

How to view I/O wait

Ubuntu, and most versions of Linux, come with the ‘top’ command already installed.

Press CTRL+ALT+T to open a Terminal window and enter ‘top’ by typing top and pressing Enter.

Towards the top-left, find the row that shows ‘%Cpu(s)’ and follow this line over to the right. The ‘wa’ statistic here reports the percentage of time the CPU is waiting for disk I/O to finish.

You may exit ‘top’ by pressing ‘q’.

I/O wait is an indicator to look further. This is where tools such as ‘iotop’ can help determine which processes have high I/O percentages.

How to find the causes of high I/O wait times

The ‘iotop’ command does not come pre-installed with Ubuntu, so you will probably need to install it.

When connected to the internet, enter the following commands to install ‘iotop’.

sudo apt-get update
sudo apt-get install iotop

You will probably need to run ‘iotop’ with elevated privileges using the sudo command, so enter sudo iotop.

The top left and right sections display the overall read and write throughput.

Below this, ‘iotop’ displays a live updating list of processes and their individual read and write I/O values.

The column you are most concerned with is that of ‘IO’. This column reports the percentage of time the process on that row was performing I/O against its total processing time.

Let’s say it takes approximately 1 second to write 200 MB to disk. You’ve issued a Save command to write 160 MB to file, which took 800ms, then your I/O wait time would be 80%.

Many systems today have multiple cores. Use Terminal again to enter the following command cat /proc/cpuinfo and find the number of Cores. You may need to scroll up a bit to find this.

Calculate 1/# of Cores.

If your wait time percent is regularly near or above this amount, your CPU is spending a lot of time waiting for disks to catch up, and thus slowing your system.

<br><br>

Windows

Monitoring disk I/O with Performance Monitor

Windows packages a tool called Performance Monitor (perfmon.exe) on it’s operating system. Open this application by clicking on the Windows (Start) icon on the toolbar and typing “Performance Monitor,” then click to open it.

from the left-hand navigation panel, expand ‘Monitoring Tools’ and select ‘Performance Monitor.’

The monitor panel will usually be very empty by default, so you will need to add the specific counters you wish to monitor.

Click the green ‘Add’ button from the toolbar above the main panel.

Beneath ‘Available Counters,’ scroll to find ‘PhysicalDisk’ and expand it. The statistics you will be interested in are:

  • ‘% Disk Time’
  • ‘% Idle Time’
  • ‘Current Disk Queue Length’
  • ‘Disk Reads/sec’
  • ‘Disk Writes/sec’

When you highlight each of these, the below panel allows you to select individual drives, ‘All Instances’, or ‘_Total’. Total will sum all activity between multiple instances, whereas selecting ‘All Instances’ adds them all individually to your displayed data. On your NVR you will probably only be concerned with the recording disk(s).

After selecting each counter and the instances below, be sure to click the ‘Add>>’ button below to move it to the ‘Added counters’ panel on the right.

NOTE: If your system is utilizing RAID or you have partitions spanning multiple disks in a JBOD configuration you can also monitor these using statistics found under ‘LogicalDisk,’ which provides statistics for partitions rather than physical disks.

Before moving on, you may also wish to expand ‘Processor’ from the ‘Available counters’ panel and add:

  • ‘% Idle Time’

When you have finished adding all your counters, click the ‘OK’ button.

By default the Performance Monitor displays a graph, but this can be difficult to read with many counters added at once, nor does it display the current values well. From the toolbar, use the ‘Display Type’ icon to cycle through the type options with each click, or use the drop-down arrow beside it to select ‘Report.’

You are no able to monitor the live disk I/O values and processor time.

The figure you will be most concerned with is the ‘% Disk Time.’ You could also choose to display the ‘% Disk Read Time’ and ‘% Disk Write Time,’ but by using ‘% Disk Time, this is the sum of both reading and writing, so the total I/O for that disk.

Many systems today have multiple cores. Press CTRL+SHIFT+Esc to open Windows Task Manager, then click on the ‘Performance’ tab to find the number of cores on your system. This example from a Windows 10 machine shows 6 cores.

Calculate 1/# of Cores.

If your idle time percent is regularly near or above this amount, then your CPU is spending a lot of time waiting for disks to catch up, and thus slowing your system. I/O wait is merely and indicator to look further.

Adding the ‘Available MBytes’ instance from the ‘Memory’ counter in Performance Monitor will show you if a large amount of memory is being consumed while data is waiting to be written to disk. The ‘Current Disk Queue Length’ instance from the ‘PhysicalDisk’ counter will help display how many operations are in the disk queue waiting to be completed.

Finally, you may search under ‘Process’ for ‘IO Data Bytes/sec’ for the ‘exacqd’ instance to display the amount of ‘IO Bytes’ the Exacq application is performing. High processor idle time, high % Disk Time, combined with high memory usage can point to a disk I/O bottleneck on the system.

<br><br>

Factors that can affect I/O

  • Average I/O performance per disk
    Higher Read/Write speeds mean more data transactions per second.
  • RAID Arrays
    Having several disks in an array increases I/O transactions.
  • RAID Factor
    While RAID factors are varied to provide a number of levels to mitigate the possibility of data loss through fault-tolerance, the RAID factor can determine if more read/write transactions are necessary for each operation. For example, all other things being equal, RAID 5 has better I/O performance than RAID 6 due to the additional parity required by RAID 6.


<br>

Categories
Knowledge Support Documentation Support exacqVision Server Products

RAID Setup Using 3ware Controller – LEGACY

To configure a RAID 5 array using a 3ware controller, complete the following steps:

<br>

  1. Restart the computer.
  2. When prompted, press Alt-3 to enter the 3ware RAID BIOS.
  3. If prompted, type admin256 as the password.
  4. Press any key to acknowledge the warning.
  5. The available drives are listed under Exportable Units. If you are re-creating a failed RAID array, some drives will still be a part of the old array. You must delete the old array before creating the new array. To do this, complete the following steps:

    <br>

    a.) Select the drives in the failed RAID array by pressing Enter when they are highlighted and using the up and down arrows on the keyboard.
    b.) Press Tab to enter the menu.
    c.) Select the Delete Unit square and press Enter.
    d.) Press Enter to confirm. This allows all the drives to be included into one RAID array.

    <br>
  1. Navigate to the top of the list of drives listed as Exportable Units. Highlight Direct Attached and press Enter. Each drive should have asterisk (*) next to it.
  2. Press Tab to enter the menu.
  3. Navigate to the Settings square and press Enter.
  4. Select Controller Policies and press Enter.
  5. Select each policy you want to change and press Enter. Select the correct setting from the list, or type the desired value and press Enter.

    a.) Export JBOD Disks should be No.
    b.) Staggered Spinup should be Enabled.
    c.) Delay Before Spinup should be 2.
    d.) Number of Drives Per Spinup should be 1.
    e.) Delay Between Spinups should be 1 second.
    f.) Staggered Method should be SATA00B.
    g.) Auto-Carving should be Enabled.
    h.) Carving Factor should be 2048.
    i.) Auto-Rebuild should be Enabled.
    j.) Auto-Verify Preferred Start Day should be Saturday.
    k.) Auto-Verify Preferred Start Time should be 12:00am.

    <br>
  1. Navigate to OK and press Enter.
  2. Navigate to Create Unit and press Enter.
  3. Select each setting you want to change and press Enter. Select the correct setting from the list, or type the desired value and press Enter.

    a.) Array Name should be Exacq.
    b.) RAID Configuration match the raid level desired (5 or 6).
    c.) Stripe Size should be 256KB.
    d.) Write Cache Setting should be Enabled.
    e.) Read Cache Setting should be Intelligent.
    f.) StorSave Profile should be Performance.
    g.) Auto-Verify should be Enabled.
    h.) Rapid RAID Recovery should be Fast Rebuild/Shutdown.

    <br>
  1. Select Advanced and press Enter.
  2. Select each setting you want to change and press Enter. Select the correct setting from the list, or type the desired value and press Enter.

    a.) Boot Volume Size should be 33.
    b.) Drive Querying Mode should be Enabled.
    c.) Continue On Error When Rebuild should be Disabled.
    d.) Initialization Method should be Background.

    <br>
  1. Navigate to OK and press Enter.
  2. Navigate to OK and press Enter.
  3. Press Y to confirm cache without a battery.
  4. Note values and press any key.
  5. Press F8.
  6. Press Y to Save Configuration and Exit.

    <br>

Now you can install the operating system.
<br>

Categories
Knowledge Support Support exacqVision Server Categories Products

Can I use an iSCSI storage server with my exacqVision Server?

Yes. Exacq supports the use of an iSCSI initiator to connect to a storage server. This is available in both Windows and Linux.

<br>

Categories
Knowledge Support Support exacqVision Server Categories Products

3rd Party Linux servers and recording to a NAS?

This process is not officially supported or endorsed by Exacq. Use at your own risk.

When the exacqVision Server software starts, it will look into the following location for storage purposes:

/mnt/edvr/*

NOTE* – This is typically a number on systems built by Exacq.

Any volume that is mounted into the /mnt/edvr location will be available for storage.

NAS Setup

Here is an example setup using FreeNAS:

  1. In the BIOS Setup, set the hardware RAID (Striped with Parity).
  2. Set up FreeNAS on the computer server hard drive.
  3. Assign the system an IP Address.
  4. Set up NFS Share using FreeNAS.

Linux NVR

  1. If needed, log off as restricted user and log back in as an admin user.
  2. Launch the Terminal application.
  3. Create storage directoryfor your NAS location by typing:
    sudo mkdir /mnt/edvr/2
    NOTE – Ensure that there is not already a “2” directory – if so, pick a higher number
  4. Edit the /etc/fstab file to include the mounting of the NAS at startup by adding line
    %servername%:%/mnt/vce-nas/% /mnt/edvr/2 nfs defaults 0 0
    NOTE: %servername% is your full server name or IP address, and %/mnt/vce-nas/% is your NAS server mount point.
  5. Restart the exacqVision NVR Server to activate NAS Mount.
    sudo service edvrserver restart
  6. In the exacqVision Client, use the Storage tab to enable recording to /mnt/edvr/2.
  7. Verify that recordings are being stored on new mount point.
Categories
Knowledge Support Support exacqVision Server Categories Products

Removing an Unwanted “Orphaned” Drive From the Storage Page

Title

Removing an Unwanted “Orphaned” Drive From the Storage Page

Description 

ExacqVision retains any drive which has ever been added to the NVR.  This can result in unwanted or “Orphaned” drives showing up on the Storage Page.  Removing these drives requires manual intervention. 

Product 

ExacqVision Server

Solution

Windows 

  1. Log into the operating system of the server as an administrator, you cannot perform these steps from a client workstation.
  2. Click the Windows key + R to open the run box 
  3. Type services.msc and press enter 
  4. Locate and right-click on exacqVision Server, select stop 
  5. Open Notepad as an administrator 
  6. Browse to and open C:\Program Files\exacqVision\Server\psfpi.xml
  7. Locate and delete the line containing the orphaned drive.
  8. Save and close psfpi.xml 
  9. Back in Services, right-click on exacqVision Server, select start
  10. Verify the drive is no longer present in the ExacqVision Client

Linux 

  1. Log into the operating system as an administrator
  2. Open a Terminal Window and stop the ExacqVision Server Service 
        sudo service edvrserver stop

Note: If prompted enter the admin operating systems password then press enter.  Linux will not give any feedback while entering the password. 

  1. Edit psfpi.xml using the following command 
        sudo nano /usr/local/exacq/server/psfpi.xml
  1. Using the arrow keys, move to the beginning of the line containing the orphaned drive 
  2. Press control+k to delete the line 
  3. Press control+o to save the file
  4. Press control+x to exit 
  5. Start the exacqVision Server Service 
        sudo service edvrserver start
  1. Verify the drive is no longer present in the exacqVision Client

Example

In the example below the drive labeled /mnt/edvr/2 is unwanted and will need to be manually removed. 
ExacqVision Client before and after orphaned drive removed

<br>