reworked logging

docs/userguide/logging.rst

@@ -1,18 +1,28 @@
 Logging
 =======
 
-Most components of ZoneMinder can emit informational, warning, error and debug messages in a standard format. These messages can be logged in one or more locations. By default all messages produced by scripts are logged in <script name>.log files which are placed in the directory defined by the ZM_PATH_LOGS configuration variable. This is initially defined as ‘/tmp’ though it can be overridden (see the Options and Users section above). So for example, the zmpkg.pl script will output messages to /tmp/zmpkg.pl, an example of these messages is::
+Most components of ZoneMinder can emit informational, warning, error and debug messages in a standard format. These messages can be logged in one or more locations. By default all messages produced by scripts are logged in <script name>.log files which are placed in the directory defined by the ``ZM_PATH_LOGS`` configuration variable. This is initially defined as ``/var/log/zm`` (on debian based systems) though it can be overridden to a custom path (the path is usually defined in ``/etc/zm/conf.d/01-system-paths.conf``, but to override it, you should create your own config file, not overwrite this file). So for example, the ``zmdc.pl`` script will output messages to ``/var/log/zmdc.log``, an example of these messages is::
 
-  03/01/06 13:46:00.166046 zmpkg[11148].INF [Command: start]
+  10/24/2019 08:01:19.291513 zmdc[6414].INF [ZMServer:408] [Starting pending process, zma -m 2]
+  10/24/2019 08:01:19.296575 zmdc[6414].INF [ZMServer:408] ['zma -m 2' starting at 19/10/24 08:01:19, pid = 15740]
+  10/24/2019 08:01:19.296927 zmdc[15740].INF [ZMServer:408] ['zma -m 2' started at 19/10/24 08:01:19]
 
 where the first part refers to the date and time of the entry, the next section is the name (or an abbreviated version) of the script, followed by the process id in square brackets, a severity code (INF, WAR, ERR or DBG) and the debug text. If you change the location of the log directory, ensure it refers to an existing directory which the web user has permissions to write to. Also ensure that no logs are present in that directory the web user does not have permission to open. This can happen if you run commands or scripts as the root user for testing at some point. If this occurs then subsequent non-privileged runs will fails due to being unable to open the log files.
 
-As well as specific script logging above, information, warning and error messages are logged via the system syslog service. This is a standard component on Linux systems and allows logging of all sorts of messages in a standard way and using a standard format. On most systems, unless otherwise configured, messages produced by ZoneMinder will go to the /var/log/messages file. On some distributions they may end up in another file, but usually still in /var/log. Messages in this file are similar to those in the script log files but differ slightly. For example the above event in the system log file looks like::
+As well as specific script logging above, information, warning and error messages are logged via the system syslog service. This is a standard component on Linux systems and allows logging of all sorts of messages in a standard way and using a standard format. On most systems, unless otherwise configured, messages produced by ZoneMinder will go to the ``/var/log/messages`` or ``/var/log/syslog`` file. On some distributions they may end up in another file, but usually still in /var/log. Messages in this file are similar to those in the script log files but differ slightly. For example the above event in the system log file looks like::
  
   Jan  3 13:46:00 shuttle52 zmpkg[11148]: INF [Command: start]
 
 where you can see that the date is formatted differently (and only to 1 second precision) and there is an additional field for the hostname (as syslog can operate over a network). As well as ZoneMinder entries in this file you may also see entries from various other system components. You should ensure that your syslogd daemon is running for syslog messages to be correctly handled.
 
+
+Customizing logging properly in ZoneMinder
+-------------------------------------------
+Understanding how logging works in ZoneMinder is key to being able to isolate/pinpoint issues well. Please refer to :doc:`/userguide/options/options_logging`
+
+
+Other Notes
+-------------
 A number of users have asked how to suppress or redirect ZoneMinder messages that are written to this file. This most often occurs due to not wanting other system messages to be overwhelmed and obscured by the ZoneMinder produced ones (which can be quite frequent by default). In order to control syslog messages you need to locate and edit the syslog.conf file on your system. This will often be in the /etc directory. This file allows configuration of syslog so that certain classes and categories of messages are routed to different files or highlighted to a console, or just ignored. Full details of the format of this file is outside the scope of this document (typing ‘man syslog.conf’ will give you more information) but the most often requested changes are easy to implement.
 
 The syslog service uses the concept of priorities and facilities where the former refers to the importance of the message and the latter refers to that part of the system from which it originated. Standard priorities include ‘info’, ‘warning’, ‘err’ and ‘debug’ and ZoneMinder uses these priorities when generating the corresponding class of message. Standard facilities include ‘mail’, ‘cron’ and ‘security’ etc but as well this, there are eight ‘local’ facilities that can be used by machine specific message generators. ZoneMinder produces it’s messages via the ‘local1’ facility.

docs/userguide/options/options_logging.rst

@@ -1,8 +1,38 @@
 Options - Logging
 -----------------
 
-.. image:: images/Options_Logging.png
+ZoneMinder has a powerful logging system. Understanding how to configure logging will help you track issues better. The logging options are accessed via ``Options->Logging``. Let's follow along with an example. But before that, here is a basic construct of how logging works:
 
+* Every component of ZoneMinder can generate different types of logs. Typically, ``ERR`` refers to an error condition that you should look at (in some cases, they are transient during startup/shutdown in which case they are usually benign). ``INF`` logs are informational, ``WAR`` are warning logs that might have a potential to cause issues, whilst ``DBG`` are debug logs that are useful when you need to debug a problems
+* You can decide where these logs are written. Typically ZoneMinder writes logs to multiple sources:
+  * Syslog 
+  * Database
+  * individual files belonging to each component inside the logging folder configured
+
+Consider for example, that you are trying to figure out why your "zmc 11" (i.e. Monitor 11) is not working. Obviously, you need to enable debug logs if you are not able to figure out what is going on with standard info logs. But you wouldn't want to write debug logs to the Database. Maybe, you also don't want it polluting your syslog and only want to write debug logs to the debug file of _that_ component (``/var/log/zm/zmc_m11.log`` for example). That is where customizing your logging is useful.
+
+Logging example
+~~~~~~~~~~~~~~~~~
+
+.. image:: images/Options_Logging_A.png
+
+In the example above, I've configured my logging as follows:
+
+* I only want to log INFO level logs to Syslog
+* I want DEBUG logs to only go to the conmponent file
+* When it comes to my WEBLOG (what I see in the ZM Log window) and Database log, I only want FATAL logs (you may want to set this to WAR or INF)
+* I don't want to save FFMPEG logs (this was a new feature added). FFMPEG generates a log of logs on its own that you should only enable if you are trying to figure out video playback related issues
+* I have enabled LOG_DEBUG (unless you enable this, DEBUG logs won't be logged)
+* The ``LOG_DEBUG_TARGET`` is useful if you don't want to enable DEBUG logs for every component. In this case, I'm only interested in debugging the ZM Event Server and Monitor 11. Nothing else will have debug logs enabled.
+* I prefer to keep the ``LOG_DEBUG_FILE`` to empty. This creates nicely separate files in my log folder with component names
+
+The other logging parameters are left to their defaults, like so:
+
+.. image:: images/Options_Logging_B.png
+
+
+A more comprehensive explanation of the various log options
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 LOG_LEVEL_SYSLOG - ZoneMinder logging is now more integrated between components and allows you to specify the destination for logging output and the individual levels for each. This option lets you control the level of logging output that goes to the system log. ZoneMinder binaries have always logged to the system log but now scripts and web logging is also included. To preserve the previous behaviour you should ensure this value is set to Info or Warning. This option controls the maximum level of logging that will be written, so Info includes Warnings and Errors etc. To disable entirely, set this option to None. You should use caution when setting this option to Debug as it can severely affect system performance. If you want debug you will also need to set a level and component below
 
 LOG_LEVEL_FILE - ZoneMinder logging is now more integrated between components and allows you to specify the destination for logging output and the individual levels for each. This option lets you control the level of logging output that goes to individual log files written by specific components. This is how logging worked previously and although useful for tracking down issues in specific components it also resulted in many disparate log files. To preserve this behaviour you should ensure this value is set to Info or Warning. This option controls the maximum level of logging that will be written, so Info includes Warnings and Errors etc. To disable entirely, set this option to None. You should use caution when setting this option to Debug as it can severely affect system performance though file output has less impact than the other options. If you want debug you will also need to set a level and component below