Disk I/O (input/output) plays a crucial role in how you experience the exacqVision software. If disks are unable to handle the amount of data being written to them, you could see reliability issues.
To help pinpoint the source of these issues on your server, use the following steps as a guide based on your system operating system.
<br><br>
Linux (Ubuntu)
How to view I/O wait
Ubuntu, and most versions of Linux, come with the ‘top’ command already installed.
Press CTRL+ALT+T to open a Terminal window and enter ‘top’ by typing top and pressing Enter.
Towards the top-left, find the row that shows ‘%Cpu(s)’ and follow this line over to the right. The ‘wa’ statistic here reports the percentage of time the CPU is waiting for disk I/O to finish.
You may exit ‘top’ by pressing ‘q’.
I/O wait is an indicator to look further. This is where tools such as ‘iotop’ can help determine which processes have high I/O percentages.
How to find the causes of high I/O wait times
The ‘iotop’ command does not come pre-installed with Ubuntu, so you will probably need to install it.
When connected to the internet, enter the following commands to install ‘iotop’.
sudo apt-get updatesudo apt-get install iotop
You will probably need to run ‘iotop’ with elevated privileges using the sudo command, so enter sudo iotop.
The top left and right sections display the overall read and write throughput.
Below this, ‘iotop’ displays a live updating list of processes and their individual read and write I/O values.
The column you are most concerned with is that of ‘IO’. This column reports the percentage of time the process on that row was performing I/O against its total processing time.
Let’s say it takes approximately 1 second to write 200 MB to disk. You’ve issued a Save command to write 160 MB to file, which took 800ms, then your I/O wait time would be 80%.
Many systems today have multiple cores. Use Terminal again to enter the following command cat /proc/cpuinfo and find the number of Cores. You may need to scroll up a bit to find this.
Calculate 1/# of Cores.
If your wait time percent is regularly near or above this amount, your CPU is spending a lot of time waiting for disks to catch up, and thus slowing your system.
<br><br>
Windows
Monitoring disk I/O with Performance Monitor
Windows packages a tool called Performance Monitor (perfmon.exe) on it’s operating system. Open this application by clicking on the Windows (Start) icon on the toolbar and typing “Performance Monitor,” then click to open it.
from the left-hand navigation panel, expand ‘Monitoring Tools’ and select ‘Performance Monitor.’
The monitor panel will usually be very empty by default, so you will need to add the specific counters you wish to monitor.
Click the green ‘Add’ button from the toolbar above the main panel.
Beneath ‘Available Counters,’ scroll to find ‘PhysicalDisk’ and expand it. The statistics you will be interested in are:
- ‘% Disk Time’
- ‘% Idle Time’
- ‘Current Disk Queue Length’
- ‘Disk Reads/sec’
- ‘Disk Writes/sec’
When you highlight each of these, the below panel allows you to select individual drives, ‘All Instances’, or ‘_Total’. Total will sum all activity between multiple instances, whereas selecting ‘All Instances’ adds them all individually to your displayed data. On your NVR you will probably only be concerned with the recording disk(s).
After selecting each counter and the instances below, be sure to click the ‘Add>>’ button below to move it to the ‘Added counters’ panel on the right.
NOTE: If your system is utilizing RAID or you have partitions spanning multiple disks in a JBOD configuration you can also monitor these using statistics found under ‘LogicalDisk,’ which provides statistics for partitions rather than physical disks.
Before moving on, you may also wish to expand ‘Processor’ from the ‘Available counters’ panel and add:
- ‘% Idle Time’
When you have finished adding all your counters, click the ‘OK’ button.
By default the Performance Monitor displays a graph, but this can be difficult to read with many counters added at once, nor does it display the current values well. From the toolbar, use the ‘Display Type’ icon to cycle through the type options with each click, or use the drop-down arrow beside it to select ‘Report.’
You are no able to monitor the live disk I/O values and processor time.
The figure you will be most concerned with is the ‘% Disk Time.’ You could also choose to display the ‘% Disk Read Time’ and ‘% Disk Write Time,’ but by using ‘% Disk Time, this is the sum of both reading and writing, so the total I/O for that disk.
Many systems today have multiple cores. Press CTRL+SHIFT+Esc to open Windows Task Manager, then click on the ‘Performance’ tab to find the number of cores on your system. This example from a Windows 10 machine shows 6 cores.
Calculate 1/# of Cores.
If your idle time percent is regularly near or above this amount, then your CPU is spending a lot of time waiting for disks to catch up, and thus slowing your system. I/O wait is merely and indicator to look further.
Adding the ‘Available MBytes’ instance from the ‘Memory’ counter in Performance Monitor will show you if a large amount of memory is being consumed while data is waiting to be written to disk. The ‘Current Disk Queue Length’ instance from the ‘PhysicalDisk’ counter will help display how many operations are in the disk queue waiting to be completed.
Finally, you may search under ‘Process’ for ‘IO Data Bytes/sec’ for the ‘exacqd’ instance to display the amount of ‘IO Bytes’ the Exacq application is performing. High processor idle time, high % Disk Time, combined with high memory usage can point to a disk I/O bottleneck on the system.
<br><br>
Factors that can affect I/O
- Average I/O performance per disk
Higher Read/Write speeds mean more data transactions per second. - RAID Arrays
Having several disks in an array increases I/O transactions. - RAID Factor
While RAID factors are varied to provide a number of levels to mitigate the possibility of data loss through fault-tolerance, the RAID factor can determine if more read/write transactions are necessary for each operation. For example, all other things being equal, RAID 5 has better I/O performance than RAID 6 due to the additional parity required by RAID 6.
<br>