Storage Management (Recorder 2.0)

The NAVR retention policies and descriptions for trimming logs and recordings involve three different mechanisms. All three mechanisms are configurable in the initial configuration utility, and two of them are available in the client’s setup functionality when the service is running.

Activity Logs

Activity logs consist of information detailing time as well as possibly (as applicable) the user, media device, IP addresses, etc. associated with:

• Successful and failed recording object creations. (This isn’t the recording itself; this is the process where an object is created in software to perform the recording.)
• When a recording object initially receives media from the source it is to be recording (or fails to).
• When a recording object is shut down after creating, whether it ever received media or not.
• When the listeners for API or client connections are started/stopped.
• If a connection is attempted from a blocked IP address.
• If a client connection is attempted with no remaining client licenses.
• When a client logs off.
• When a client requests streaming media (live or recorded) from the NAVR.
• When a client elects to stop streaming media (live or recorded) from the NAVR.
• If the NAVR succeeds or fails at connection to a media source (camera/audio source).
• Analytic Events
• When a media source disconnects from the NAVR.
• If a client makes an invalid media request (asking for a camera/media that doesn’t exist, or that they don’t have access for).

1. Trim all logs older than X days: removes any of these items after the given time period. These aren’t recordings or even the metadata for recordings, this is more like diagnostic and audit data. If no value is set (i.e. the number field is empty), 30 days is the default.
2. Trim all successful media request logs older than X days: removes all logs related to a client viewing or streaming media IF the request was valid (i.e. the user had permission to do so). If the user did not have permission to access that media stream, or asked for a stream that didn’t exist, etc. the configuration of item 1 would prevail. These are basically audit logs for authorized attempts of who viewed what and when. If no value is set (i.e. the number field is empty), 14 days is the default. If this time span is configured as longer than the value for item 1, entries will be trimmed for that reason and this configuration won’t matter.
3. Trim all successful device connection events older than X days removes all logs related to a recording object first receiving, and finishing receiving media from a media source (camera, audio, etc.). This only trims items where there was no problem starting/finishing the connection. If there was an issue with starting or ending the connection between NAVR and the media source, the configuration of item 1 would prevail. These are basically diagnostic logs of successful connections so that we can know when things worked, instead of just that no error was written. If no value is set (i.e. the number field is empty), 14 days is the default. If this time is ever longer than the value for item 1, entries will be trimmed for that reason and this configuration won’t matter.

These trims are performed daily at the configured time, and then the log indexes rebuilt in order to maintain fast log searches. Because each individual record is small and deleting chunks of logs is more efficient than removing individual logs, but rebuilding the indexes has some overhead, this is a daily scheduled event.

Recordings

These are the actual recordings and metadata for those recordings including recording start and end date/time, device identity, and the type of recording (audio, RTSP, MJPG), if the recording was triggered by an event/user or was a normally scheduled recording. Recording cleaning is carried out constantly, as recordings can be large and therefore it is important to remove them once they are expired as well as the fact that it is no more efficient to delete a large number of recordings at once than individually.

The FIFO (First In, First Out) process is as follows:
1) The oldest recording of any type (Scheduled or Event) is found. If the start of this recording is older than Delete media AND database records for all recordings older than X days, then the recording metadata (i.e. what we search on and display as information in the client) as well as the media (actual recording) are deleted. No record of this recording remains at all, this is simply falling off the end of the FIFO queue. This step is repeated until there are no recordings left on the system older than the given time period. This time period defaults to 365 days.

If this value is set to a value lower than a previous value in the client while the service is running, e.g. someone changes it from 365 days to 0 days, the oldest recordings will begin deleting immediately and in that case all recordings will be deleted

2) If the item “Delete media for event recordings older than X days” is checked and has a valid value, then the oldest event recording (manually started or triggered from an outside mechanism) that has not been marked as deleted is found. If that recording start time is older than the given time period, the media (actual audio/video data) will be deleted, but the metadata will remain, with a note that the media was deleted due to exceeding the staleness time, and the datetime that deletion occurred. This recovers the VAST (99.99+%) amount of data used by the recording, but allows operators to still search for and see that an event recording was made, when, and on what devices. There is no default time period for this item, if it is not configured, then event recordings are treated no differently than any recording, and the procedure in step 1 of this section is followed. (If this value is set to a value lower than a previous value in the client while the service is running, e.g. someone changes it from 365 days to 0 days, the oldest recordings will begin deleting immediately and, in that case, all recorded media would be deleted)

3) Same as item 2 in this section, but for scheduled recordings.

Repository Trim Size

The item that cannot be changed while the system is running is the Repository trim size. This is just how big the actual recording repository directory is allowed to get. As soon as it hits that size, recorded media will begin FIFO deleting to stay at/under that limit. This only comes into play AFTER the trim by date/time (described above) is performed. If the remaining recording sizes still exceed the configured repository size, and then only for recordings more than 2 calendar days old the following procedure will occur. A single entry will be written to the NAVR log (regardless of how many recordings are about to be removed) indicating that a trim due to exceeded repository size is being performed. Similar to steps 2 and 3 of the Recordings section above, the metadata for the recordings will remain in the database, and will indicate when the recording media was deleted, and that it was deleted due to the configured repository size limit. That metadata will be removed as per step 1 of the Recordings section. This step is checked constantly, just like the other media trim mechanisms under Recordings. If the repository maximum size is decreased, then as soon as the NAVR is running, it will begin clearing media in order to get under the given maximum repository size.