on Jun 14, 2016Article Content
| Article Number | 000032301 |
| Applies To | RSA Product Set: NetWitness Logs & Network/Security Analytics RSA Product/Service Type: NetWitness Appliances (including Hybrid & All-in-One appliances), VLC hosts RSA Version/Condition: 10.4.x, 10.5.x, 10.6.x Platform: CentOS O/S Version: 6 |
| Issue | The /var/log/messages log file grows to occupy all of the available space in the /var/log partition preventing services such as the nwlogcollector (and other services) from starting. |
| Cause | This issue is most often seen in RSA NetWitness Hybrid and All-In-One (AIO) appliances and Virtual Log Collectors (VLCs) due to the volume of entries that the nwlogcollector service writes to /var/log/messages. VLCs often have a smaller /var/log volume (e.g. 3.9G) than physical appliances (e.g. 9.8G). In order to detect the problem, log into the affected host using SSH and run the following commands. The outputs in the examples below were taken from a VLC
As can be seen in the hi-lighted section in red above, /var/log volume has reached 100% utilization. To locate which files and directories are occupying the most space.
We have identified the issue now, /var/log/messages is causing /var/log to fill up quickly. An alternative way of doing this would be to use the 'ls' command and sorting file size to examine the files in /var/log directory (Hint: Could add the -R switch as well to recurse into subdirectories, however, the -S switch only sorts files within each directory):
Note: If the utilisation of 'df -hP' and 'du -ahx' don't match then this is likely due to a failure of logrotate when writing to a new file. Run the following command to check for deleted but not released log files:
To release the space being taken by the deleted file (but held by rsyslogd as it still has an open file handle), you will either need to reboot the OS or restart the syslog services
|
| Resolution | The logrotate service's configuration need to be adjusted by editing /etc/logrotate.d/syslog to allow the normal rotation of /var/log/messages. |
| Workaround | BEFORE: The current configuration of logrotate for syslog services in 10.6.x is as follows:
AFTER: This file needs to be edited to the following:
In this way we are going to rotate /var/log/messages on a weekly basis (retaining 4 compressed logs) or when the file reaches the size of 250 MB (whichever comes first). The dateext means that the date of rotate will be appended to the filename e.g. messages-20190212 Test that the configuration is correct by running logrotate manually using the following command:
If you are unsure of any of the steps above or experience any issues, contact RSA Customer Support and reference this article for further assistance. |
| Notes | If after applying the above steps logrotate is not working, then the syslog service may need to be restarted as shown below.
Note: Other non-standard packages installed on the host such as syslog-ng may also cause logrotate to fail due to additional file handles on /var/log/messages. RSA Support would recommend that these non-standard packages be removed. You may be able to find these processes using the following command:
|