12.1 Logging and Rsyslog
So what is logging. This is the output of the system that is forwarded to files so that we and applications can read it. We can use log files for alot of things for example:
Monitoring (checking if our servers are healthy and ok)
SIEM (Security monitoring, keeping track of the security of our servers)
Auditing (Investigating if we have worked accorinding to the rules we have set for our selfs.
These are only a few things which we can use the system logs for. There are alot more of them. Linux systems will almost all use Rsyslog for logging, more indepth information on rsyslog can be found in below link.
Rsyslog configuration
The main configuration file for rsyslog is /etc/rsyslog.conf. We can edit this file with vim I recommend this highly because of the colour highlights in this file. There are alot of comments in this file which are very handy but makes it a little unreadable without the colours.
There are 3 parts in the /etc/rsyslog.conf file. The #### MODULES #### part which contains the modules used for rsyslog. The #### GLOBAL DIRECTIVES #### which shows all the files and directories used by rsyslog. And as last there are the rules #### RULES #### which shows all the rules that rsyslog is executing.
Main log: /var/log/messages (holds alot of log information)
Security log: /var/log/secure (holds all of the login attempts for example)
You can test the /var/log/secure logging by opening 2 terminals to your VM. On 1 of your VM’s execute the following command tail -f /var/log/secure (Hit return(enter) a few times to create some space). On the other VM do an incorrect login on your VM with for example su - root from a none root user. This should generate a few lines of logging on your first terminal that is tracing the tail -f /var/log/secure.
Setup a new rsyslog rule
If you want to add your own rules this can be done in 2 ways. You can add them at the bottom of the /etc/rsyslog.conf file. Or you can create a new file ending in .conf in the /etc/rsyslog.d/ directory.
Lets try option 2:
Change the rsyslog configuration to log all messages with severity debug to /var/log/messages-debug on your VM by adding the the line *.debug /var/log/messages-debug to the /etc/rsyslog.d/debug.conf file.
echo "*.debug /var/log/messages-debug" > /etc/rsyslog.d/debug.conf
You can check if it worked with:
[root@rhcsa ~]# cat /etc/rsyslog.d/debug.conf
*.debug /var/log/messages-debug
This will add the line *.debug /var/log/messages-debug to the /etc/rsyslog.d/debug.conf file and allow for debug logging to the new file.
You have to restart the rsyslog service on your VM to make the new log rule take effect.
systemctl restart rsyslog
Testing our new rule
Let’s do a tail on the new log file so we can see what will happen.
Monitor the /var/log/messages-debug with the tail command on VM:
tail -f /var/log/messages-debug
While monitoring the file we need to generate a debug message, we can do that with the below logger command. Execute this in a new terminal window so that we can keep the initial monitoring running.
[root@rhcsa ~]# logger -p user.debug "Debug Message Test"
After executing the logger command you can check the initial terminal with the monitoring again to see that the debug message has been received.