System watchdogs when configuring Extended Storage

Description

Affected platforms: Linux, Ubuntu 32-bit with PAE-configuration (systems with more than 4GB of RAM).

Affected configuration: 1 DVR + n S-Series, where s-series are all used for Extended Storage and are being connected to the single DVR.

<br>

Symptoms

A DVR system watchdogs while configuring many extended storage (iscsi) volumes.

<br>

Troubleshooting

• This is happening due to the system running out of Low Memory. Low Memory stats can be checked using command: free -lm. Values below 100MB for free low memory indicate a system that may be at risk. Low memory values tend to fluctuate a lot, so a reading close 100MB may mean that some time it can dip below 50MB, which is a definite danger zone.
• Running out of low memory is not the same as running out of all memory. The system may report plenty of memory available using either top command or dstat command. Total available memory does not correlate with available low memory.
• When free low memory gets low, then oom-killer (out-of-memory killer) OS process is triggered and it kills the process on the system that it believes will free up the most memory. The following command will report the process that will be killed first: dstat --top-oom (usually it is core).
• When oom-killer is activated, the following message can be seen in /var/log/syslog: Out of memory: Kill process (core) score or sacrifice child.
• When oom-killer kills core process, a watchdog will trigger and the system will reboot.

<br>

Workarounds

In-house testing revealed that at most 4 80TB S-Series can be used for extended storage on a single recorder. This provides 20 iscsi volume with 13TB of usable space each. Configuring more iscsi storage (in TBs not number of volumes) may lead to an unstable system that could run out of low memory and therefore watchdog. If a customer has more than 4 S-Series, any S-Series can be converted to a z-series. Customer’s configuration would have to be altered to accommodate recording to more than 1 DVR.

<br>

Solution

This is a limitation of a 32-bit Linux OS. We’re expecting this issue to be resolved with the release of a 64-bit Ubuntu image for BBE systems.

<br>

Recommended Network Configuration:

Camera and iscsi networks should be isolated. For example, given a recorder with at least 2 NICs, one of the NICs should be dedicated to receiving camera traffic. Another NIC should be dedicated to sending iscsi traffic to the S-Series.
An S-Series should have one of its NICs dedicated to receiving iscsi traffic from the recorder and nothing else.

Given this setup, you can estimate whether the underlying network quality will work for the extended storage environment configuration. Use the following command to check the network traffic volume: iftop -i eth0 -n
Compare received network traffic volume on the camera NIC to the transmitted volume on the iscsi NIC using the command above.

If the reported iscsi NIC network volume is less than the received camera volume, it is an indication that the iscsi configuration will not be able to keep up.

In the case of iscsi connection not keeping up with the incoming camera traffic, it is recommended to use local recording instead of extended storage. An s-series can be converted to become a Z-series and some cameras should be moved from the original DVR to the converted one.