This article explains how to determine whether an ExacqVision server’s hardware has been overloaded, how ExacqVision software manages the overload, and how to address the root cause.
Video recording and retrieval can place a significant load on a server’s data storage subsystem. This load can be highly variable on systems configured for motion-based recording or with cameras that have variable bit rates. Video searches also place a large, short-term load on the storage system during retrieval.
When the server’s storage is unable to read and write data at the rate determined by the cameras and user searches, data writes are queued in physical memory. Although this is a useful feature for short-duration overloads, longer or more consistent overloads exhaust the memory available to the ExacqVision Server software.
ExacqVision software (version 4.6.5 and above on Windows, 4.6.10 and above on Linux) contains an overload management feature. This feature enables the server to continue operating with degraded write performance. During periods of server storage overloads, ExacqVision Server software pauses recording of all cameras to allow the storage system to catch up. An overload is defined as either of the following situations:
Less than one eighth of the physical memory is available.
ExacqVision’s committed virtual memory exceeds half the available virtual memory space.
<br>
Testing
The following test should be performed on systems that might be experiencing overloads:
a. On the Client Configuration screen, select the server from the site tree.
b. On the System Log tab, set the system log level to Warning, and enter a long search period. Click Search.
c. In the search box, enter the word “throttling”
d. A “throttling video” message indicates the first occurrence of video throttling. A subsequent “not throttling video” occurs 60 seconds after the last throttling activity.
Lower the frame rate or quality on cameras until the memory usage does not increase significantly during searches.<br><br>
Install an additional server, transferring some of the cameras from the overloaded server.<br><br>
Install a higher-performance disk system in the existing server.<br><br>
(LEGACY) 32-bit Windows only: If the server has less than 4GB of RAM, you can install additional memory to help with short-term overloads (such as multiple simultaneous searches). However, this will not correct systematic overloads (too much steady camera video). Memory in excess of 4GB typically produces little or no delay in the onset of video throttling, although it could be helpful in 64-bit Windows.
Disk I/O (input/output) plays a crucial role in how you experience the exacqVision software. If disks are unable to handle the amount of data being written to them, you could see reliability issues.
To help pinpoint the source of these issues on your server, use the following steps as a guide based on your system operating system.
<br><br>
Linux (Ubuntu)
How to view I/O wait
Ubuntu, and most versions of Linux, come with the ‘top’ command already installed.
Press CTRL+ALT+T to open a Terminal window and enter ‘top’ by typing top and pressing Enter.
Towards the top-left, find the row that shows ‘%Cpu(s)’ and follow this line over to the right. The ‘wa’ statistic here reports the percentage of time the CPU is waiting for disk I/O to finish.
You may exit ‘top’ by pressing ‘q’.
I/O wait is an indicator to look further. This is where tools such as ‘iotop’ can help determine which processes have high I/O percentages.
How to find the causes of high I/O wait times
The ‘iotop’ command does not come pre-installed with Ubuntu, so you will probably need to install it.
When connected to the internet, enter the following commands to install ‘iotop’.
sudo apt-get update sudo apt-get install iotop
You will probably need to run ‘iotop’ with elevated privileges using the sudo command, so enter sudo iotop.
The top left and right sections display the overall read and write throughput.
Below this, ‘iotop’ displays a live updating list of processes and their individual read and write I/O values.
The column you are most concerned with is that of ‘IO’. This column reports the percentage of time the process on that row was performing I/O against its total processing time.
Let’s say it takes approximately 1 second to write 200 MB to disk. You’ve issued a Save command to write 160 MB to file, which took 800ms, then your I/O wait time would be 80%.
Many systems today have multiple cores. Use Terminal again to enter the following command cat /proc/cpuinfo and find the number of Cores. You may need to scroll up a bit to find this.
Calculate 1/# of Cores.
If your wait time percent is regularly near or above this amount, your CPU is spending a lot of time waiting for disks to catch up, and thus slowing your system.
<br><br>
Windows
Monitoring disk I/O with Performance Monitor
Windows packages a tool called Performance Monitor (perfmon.exe) on it’s operating system. Open this application by clicking on the Windows (Start) icon on the toolbar and typing “Performance Monitor,” then click to open it.
from the left-hand navigation panel, expand ‘Monitoring Tools’ and select ‘Performance Monitor.’
The monitor panel will usually be very empty by default, so you will need to add the specific counters you wish to monitor.
Click the green ‘Add’ button from the toolbar above the main panel.
Beneath ‘Available Counters,’ scroll to find ‘PhysicalDisk’ and expand it. The statistics you will be interested in are:
‘% Disk Time’
‘% Idle Time’
‘Current Disk Queue Length’
‘Disk Reads/sec’
‘Disk Writes/sec’
When you highlight each of these, the below panel allows you to select individual drives, ‘All Instances’, or ‘_Total’. Total will sum all activity between multiple instances, whereas selecting ‘All Instances’ adds them all individually to your displayed data. On your NVR you will probably only be concerned with the recording disk(s).
After selecting each counter and the instances below, be sure to click the ‘Add>>’ button below to move it to the ‘Added counters’ panel on the right.
NOTE: If your system is utilizing RAID or you have partitions spanning multiple disks in a JBOD configuration you can also monitor these using statistics found under ‘LogicalDisk,’ which provides statistics for partitions rather than physical disks.
Before moving on, you may also wish to expand ‘Processor’ from the ‘Available counters’ panel and add:
‘% Idle Time’
When you have finished adding all your counters, click the ‘OK’ button.
By default the Performance Monitor displays a graph, but this can be difficult to read with many counters added at once, nor does it display the current values well. From the toolbar, use the ‘Display Type’ icon to cycle through the type options with each click, or use the drop-down arrow beside it to select ‘Report.’
You are no able to monitor the live disk I/O values and processor time.
The figure you will be most concerned with is the ‘% Disk Time.’ You could also choose to display the ‘% Disk Read Time’ and ‘% Disk Write Time,’ but by using ‘% Disk Time, this is the sum of both reading and writing, so the total I/O for that disk.
Many systems today have multiple cores. Press CTRL+SHIFT+Esc to open Windows Task Manager, then click on the ‘Performance’ tab to find the number of cores on your system. This example from a Windows 10 machine shows 6 cores.
Calculate 1/# of Cores.
If your idle time percent is regularly near or above this amount, then your CPU is spending a lot of time waiting for disks to catch up, and thus slowing your system. I/O wait is merely and indicator to look further.
Adding the ‘Available MBytes’ instance from the ‘Memory’ counter in Performance Monitor will show you if a large amount of memory is being consumed while data is waiting to be written to disk. The ‘Current Disk Queue Length’ instance from the ‘PhysicalDisk’ counter will help display how many operations are in the disk queue waiting to be completed.
Finally, you may search under ‘Process’ for ‘IO Data Bytes/sec’ for the ‘exacqd’ instance to display the amount of ‘IO Bytes’ the Exacq application is performing. High processor idle time, high % Disk Time, combined with high memory usage can point to a disk I/O bottleneck on the system.
<br><br>
Factors that can affect I/O
Average I/O performance per disk Higher Read/Write speeds mean more data transactions per second.
RAID Arrays Having several disks in an array increases I/O transactions.
RAID Factor While RAID factors are varied to provide a number of levels to mitigate the possibility of data loss through fault-tolerance, the RAID factor can determine if more read/write transactions are necessary for each operation. For example, all other things being equal, RAID 5 has better I/O performance than RAID 6 due to the additional parity required by RAID 6.
