After startup SEC reads lines from files, named pipes, or standard input, matches the lines with patterns (regular expressions, Perl subroutines, etc.) for recognizing input events, and correlates events according to the rules in its configuration file(s). In order to detect a composite event, rules are used sequentially in the same order as they are given in the configuration file. If there are two or more configuration files, rule sequences from different files are used virtually in parallel. SEC can be configured to produce its output by executing user-specified shell commands, and by using utilities like snmptrap(1) or snmpnotify(1), SEC can generate network management events as output. Other options for producing output events are described further in this man page.
Following rule types are currently implemented in SEC for detecting composite events:
Single - match input event and execute an action immediately.
SingleWithScript - match input event and depending on the exit value of an external script, execute an action.
SingleWithSuppress - match input event and execute an action immediately, but ignore following matching events for the next t seconds.
Pair - match input event, execute an action immediately, and ignore following matching events until some other input event arrives. On the arrival of the second event execute another action.
PairWithWindow - match input event and wait for t seconds for other input event to arrive. If that event is not observed within a given time window, execute an action. If the event arrives on time, execute another action.
SingleWithThreshold - count matching input events during t seconds and if a given threshold is exceeded, execute an action and ignore all matching events during the rest of the time window.
SingleWith2Thresholds - count matching input events during t1 seconds and if a given threshold is exceeded, execute an action. Then start the counting of matching events again and if their number per t2 seconds drops below the second threshold, execute another action.
Suppress - suppress matching input events (used to keep the events from being matched by later rules).
Calendar - execute an action at specific times.
If a rule defines the correlation of several events in a certain time window, the rule will start an event correlation operation when the first matching event occurs. In order to distinguish between different event correlation operations that have been started by the rules, every operation is assigned a unique key that is generated from the rule ID, the name of the configuration file the rule belongs to, and the textual description of the composite event that the correlation operation is trying to detect (see RULES AND EVENT CORRELATION OPERATIONS section for more information).
Rules support not only the execution of shell commands as actions, but also the creation and deletion of contexts that decide whether a particular rule can be applied at a given moment, the generation of synthetic events that will act as input for other rules, the clearing of correlation operations that have been started by other rules, etc. This makes it possible to combine several rules and form more complex event correlation schemes. Contexts can not only be used for activating or deactivating a particular rule, but they can also serve as event stores (in logsurfer(1) the contexts are used in a similar way). There are actions for adding events to a context, reporting all events associated with a context, etc. By using the features of SEC in an appropriate way, one can accomplish a wide range of event correlation and event consolidation tasks.
Note that one can introduce options both with a single dash (-) and double dash (--), and also use both an equal sign (=) and whitespace as a separator between the option name and the option value, e.g., -conf=<file pattern> and --conf <file pattern> are equivalent.
Before describing each rule type in detail, patterns and pattern types, context expressions, and action lists are discussed, since they are important parts of the rule definition.
ptype=substr
pattern=Backup done:\tsuccess
The pattern matches lines containing "Backup done:<TAB>success".
ptype=regexp
pattern=(?i)(\S+\.mydomain).*printer: toner\/ink low
The pattern matches "printer: toner/ink low" messages in a case insensitive manner from printers belonging to .mydomain. Note that the printer hostname is assigned to $1, while the whole message line is assigned to $0. As another example, the following pattern definition
ptype=regexp2
pattern=^AAA\nBBB$
produces a match if the last two input lines are AAA and BBB, and sets $0 to "AAA\nBBB".
function(L1, L2, ..., L<number>, S1, S2, ..., S<number>)
(if the input line has been generated with the event action, its input source name will be set to 'undef'). If the function returns several values or a single value that is TRUE in boolean context, the pattern matches. If the pattern matches, return values will be assigned to the special variables $1, $2, ..., and the special variable $0 will be set to "L1\nL2\n...\nL<number>" (i.e., to the matching input line(s)). These special variables can be used in some other parts of the rule definition. As an example, consider the following pattern definition:
ptype=perlfunc2
pattern=sub { return ($_[0] cmp $_[1]); }
The pattern compares last two input lines in a stringwise manner ($_[1] holds the last line and $_[0] the preceding one), and matches if the lines are different. Note that the result of the comparison is assigned to $1, while the two input lines are concatenated (with the newline character between them) and assigned to $0. As another example, the following pattern definition
ptype=perlfunc
pattern=sub { if ($_[0] =~ /(abc|def)/) { \
return defined($_[1]) ? $_[1] : "SEC"; } return 0; }
produces a match if the input line contains either the string "abc" or the string "def", and sets $0 to the matching line and $1 to the name of the input source.
Note that since Pair and PairWithWindow rules have two pattern definitions, special variables $<number> set by the first pattern are shadowed when the second pattern matches and sets the variables. In order to access the shadowed variables, they must be referred to as %<number> (e.g., instead of $1 one has to write %1). Also note that the second pattern of Pair and PairWithWindow rules may contain $<number> variables, if the second pattern is of type SubStr, NSubStr, Regexp, or NRegExp. The variables are substituted at runtime with the values set by the first pattern.
If the operand contains the arrow (->), the text following the arrow is considered to be a Perl function that will be compiled at SEC startup by calling the Perl eval() function, and eval() must return a code reference for the operand to be valid (also see VARIABLES AND EVAL section for more information). If any text precedes the arrow, it is considered to be the list of parameters for the function. Parameters are separated by whitespace and may contain $<number> and %<number> special variables. In order to evaluate the Perl function operand, the function with its parameter list will be called in scalar context. If the return value of the function is TRUE in boolean context, the truth value of the operand is TRUE, otherwise its truth value is FALSE.
If the operand begins with the equal sign (=), the following text is considered to be a Perl miniprogram. The miniprogram may contain $<number> and %<number> special variables. In order to evaluate the Perl miniprogram operand, it will be executed by calling the Perl eval() function in scalar context (also see VARIABLES AND EVAL section for more information). If the return value of the miniprogram is TRUE in boolean context, the truth value of the operand is TRUE, otherwise its truth value is FALSE.
Note that since Perl functions, miniprograms, and parameter lists may contain strings that clash with the operators of the context expression (e.g., '!'), it is strongly recommended to enclose them in parentheses, e.g.,
($1 $2) -> (sub { return ($_[0] != $_[1]); })
=({my($temp) = 0; return !$temp;})
If the operand is not a Perl function or program, it is considered to be a context name. Context name may contain $<number> and %<number> special variables. If the context name refers to an existing context, the truth value of the operand is TRUE, otherwise its truth value is FALSE.
If the whole context expression is enclosed in square brackets [ ], e.g., [MYCONTEXT1 && !MYCONTEXT2], SEC evaluates the expression _before_ pattern matching operation (normally the pattern is compared with input line(s) first, so that $<number> and %<number> variables in the context expression could be replaced with their values). Evaluating context expression first could save CPU time when many of the input lines are known to match the pattern but not the context expression, and the expression does not contain any variables.
As an example, consider the following context expressions:
-> ( sub { my(@stat) = stat("/var/log/messages"); \
return (!scalar(@stat) || time() - $stat[9] > 3600); } )
($1 $2) -> (sub { return ($_[0] != $_[1]); }) && C1
!(C1 || C2) && =("$1" eq "myhost.mydomain")
The first expression is TRUE when the /var/log/messages file does not exist or was last modified more than 1 hour ago. The second expression is TRUE when variables $1 and $2 are numerically not equal and the context C1 exists. The third expression is TRUE when contexts C1 and C2 do not exist and the $1 variable equals to the string "myhost.mydomain". Note that since && is a short-circuiting operation, the Perl code of the third expression is not evaluated if C1 and/or C2 exist.
%s - event description string (set either by the
desc
or
desc2
parameter of the rule definition).
%t - textual timestamp (as returned by
date(1)).
%u - numeric timestamp (as returned by
time(2)).
%<alnum_name> - user-defined variables that can be set at SEC runtime with
certain actions like
assign
and
eval
(<alnum_name> must begin with a letter and may contain letters, digits, and
underscores). In order to disambiguate <alnum_name> from the following text,
<alnum_name> must be enclosed in braces: %{<alnum_name>}.
In order to use semicolons inside a non-constant action parameter, the parameter must be enclosed in parentheses (the outermost set of parentheses will be removed by SEC during configuration file parsing).
The following actions are supported:
Examples:
spawn /bin/tail -f /var/log/trapd.log
Follow the /var/log/trapd.log file and feed to SEC input all lines that are appended to the file.
pipe '%t: $0' /bin/mail -s "alert message" root@localhost
Mail the timestamp and the value of the $0 variable to the local root.
add ftp_$1 $0; set ftp_$1 1800 (report ftp_$1 /bin/mail root@localhost)
Add the value of the $0 variable to the event store of the context ftp_<the value of $1>. Also extend the context's lifetime for 30 minutes, so that when the context expires, its event store will be mailed to the local root.
eval %funcptr ( sub { my(@buf) = split(/\n/, $_[0]); \
my(@ret) = grep(!/^#/, @buf); return @ret; } ); \
copy C1 %in; call %out %funcptr %in; fill C1 %out
Create a subroutine for weeding out comment lines from the input list, and use this subroutine for removing comment lines from the event store of the context C1.
action=eval %o (print ")";)
will be considered an invalid action list (however, note that
action=eval %o (print "()";)
would be passed by SEC, since now parentheses are balanced). In order to avoid such unwanted behavior, each parenthesis without a counterpart must be masked with a backslash (the backslash will be removed by SEC during configuration file parsing), e.g.,
Note that context, desc, and action parameters may contain $<number> variables.
Examples:
type=single
continue=takenext
ptype=regexp
pattern=ftpd\[(\d+)\]: \S+ \(ristov2.*FTP session opened
desc=ftp session opened for ristov2 pid $1
action=create ftp_$1
type=single
continue=takenext
ptype=regexp
pattern=ftpd\[(\d+)\]:
context=ftp_$1
desc=ftp session event for ristov2 pid $1
action=add ftp_$1 $0; set ftp_$1 1800 \
(report ftp_$1 /bin/mail root@localhost)
type=single
ptype=regexp
pattern=ftpd\[(\d+)\]: \S+ \(ristov2.*FTP session closed
desc=ftp session closed for ristov2 pid $1
action=report ftp_$1 /bin/mail root@localhost; \
delete ftp_$1
The first rule creates the context with the name ftp_<pid> when someone connects from host ristov2 with ftp. The second rule adds all logfile lines that are associated with the session <pid> to the event store of the context ftp_<pid> (before adding a line, the rule checks if the context exists). After adding a line, the rule extends context's lifetime for 30 minutes and sets the action list that will be executed when the context expires. The third rule mails collected logfile lines to root@localhost when the session <pid> is closed. Collected lines will also be mailed when the session <pid> has been inactive for 30 minutes (no logfile lines observed for that session).
Note that the logfile line that has matched the first rule will be passed to the second rule and will become the first line in the event store (the first rule has the continue parameter set to TakeNext). The second rule has also its continue parameter set to TakeNext, since otherwise no logfile lines would reach the third rule.
Note that context, script, desc, action, and action2 parameters may contain $<number> variables.
Also note that since the runtime of the script that is specified by the script parameter is not limited in any way, waiting for that script to complete could freeze the entire event processing for an indefinite amount of time. Therefore, once the pattern and context parameters have matched input line(s), SEC does not wait for the script to complete but rather continues its work. The exit value of the script will be fetched when the script actually terminates, and depending on the exit value, either action or action2 (if it exists) will be executed.
Examples:
type=SingleWithScript
ptype=RegExp
pattern=interface (\S+) down
script=/usr/local/scripts/ping.sh $1
desc=Interface $1 down
action=shellcmd /usr/local/scripts/notify.sh "%s"
When "interface <ipaddress> down" line appears in input, this rule acts by calling
/usr/local/scripts/ping.sh <ipaddress>
and if the script returns 0 as its exit value, the rule calls
/usr/local/scripts/notify.sh "Interface <ipaddress> down"
(note also that SEC feeds the names of all existing contexts to the standard input of the /usr/local/scripts/ping.sh script.)
Note that context, desc, and action parameters may contain $<number> variables.
Examples:
type=SingleWithSuppress
ptype=RegExp
pattern=(\S+): [fF]ile system full
desc=File system $1 full
action=shellcmd notify.sh "%s"
window=900
When "/usr: file system full" line appears in input, this rule starts a correlation operation that calls
notify.sh "File system /usr full"
and ignores following such lines during the next 15 minutes.
Note that line "/tmp: file system full" would not be ignored, since SEC identifies correlation operations by a key that contains event description string of the composite event (given by the desc parameter). Since strings "/usr: file system full" and "/tmp: file system full" differ, SEC will start another correlation operation for the latter (see RULES AND EVENT CORRELATION OPERATIONS section for more information).
Note that context, desc, action, context2, desc2, and action2 parameters may contain $<number> variables. The pattern2 parameter may contain $<number> variables if ptype2 supports variable substitutions at runtime. If pattern2 shadows the $<number> variables of pattern, then context2, desc2, and action2 parameters may contain %<number> variables.
Examples:
type=Pair
ptype=RegExp
pattern=NFS server (\S+) not responding
desc=$1 is not responding
action=shellcmd notify.sh "%s"
ptype2=substr
pattern2=NFS server $1 ok
desc2=$1 OK
action2=shellcmd notify.sh "%s"
window=3600
When "NFS server fserv is not responding" line appears in input, the correlation operation started by this rule calls
notify.sh "fserv is not responding"
and waits for the line "NFS server fserv ok" for 1 hour, ignoring all "NFS server fserv is not responding" lines. When the line "NFS server fserv ok" appears, the correlation operation executes the shell command
notify.sh "fserv OK"
and terminates. If time (1 hour) runs out, the correlation operation will terminate without doing anything.
Note that context, desc, action, context2, desc2, and action2 parameters may contain $<number> variables. The pattern2 parameter may contain $<number> variables if ptype2 supports variable substitutions at runtime. If pattern2 shadows the $<number> variables of pattern, then context2, desc2, and action2 parameters may contain %<number> variables.
Examples:
type=PairWithWindow
ptype=RegExp
pattern=node (\S+) interface (\S+) down
desc=$1 if $2 is down
action=shellcmd notify.sh "%s"
ptype2=SubStr
pattern2=node $1 interface $2 up
desc2=$1 if $2 short outage
action2=event
window=600
When "node fserv interface 192.168.1.1 down" line appears in input, this rule starts a correlation operation that waits 10 minutes for "node fserv interface 192.168.1.1 up" line, and if that line does not arrive on time, the correlation operation executes the shell command
notify.sh "fserv if 192.168.1.1 is down"
and terminates. If the line arrives on time, the operation generates event "fserv if 192.168.1.1 short outage" and terminates.
Note that context, desc, and action parameters may contain $<number> variables.
Examples:
type=SingleWithThreshold
ptype=RegExp
pattern=user (\S+) login failure on (\S+)
desc=Repeated login failures for user $1 on $2
action=shellcmd notify.sh "%s"
window=60
thresh=3
When line "user dbadmin login failure on tty1" is observed, the rule starts a correlation operation that executes the shell command
notify.sh "Repeated login failures for user dbadmin on tty1"
if additional two such lines are observed within 1 minute. Following "user dbadmin login failure on tty1" lines will be ignored by this rule until 60 seconds have elapsed since the arrival of first line.
After changing the action parameter in the rule definition to
action=shellcmd notify.sh "%s"; reset 0 %s
the correlation operation will terminate itself after calling notify.sh, and the next matching line will start a new counting operation.
Note that context, desc, action, desc2, and action2 parameters may contain $<number> variables.
Examples:
type=SingleWith2Thresholds
ptype=RegExp
pattern=(\S+): %SYS-3-CPUHOG
desc=router $1 CPU overload
action=shellcmd notify.sh "%s"
window=300
thresh=2
desc2=router $1 CPU load normal
action2=shellcmd notify.sh "%s"
window2=3600
thresh2=0
When SYS-3-CPUHOG syslog message is received from a router, the rule starts a counting operation that executes the shell command
notify.sh "router <routername> CPU overload"
if additional SYS-3-CPUHOG syslog message is received from the router within 5 minutes. After that the correlation operation waits until no SYS-3-CPUHOG syslog messages have been received from the router during the last 1 hour, and then executes
notify.sh "router <routername> CPU load normal"
Note that the context parameter may contain $<number> variables.
Examples:
type=Suppress
ptype=RegExp
pattern=[Ff]ile system full
context=mycontext
type=Suppress
ptype=RegExp
pattern=/dev/vg01/\S+: [Ff]ile system full
The first rule suppresses all "file system full" events if the context mycontext is present. The second rule filters out all "file system full" events that concern the volume group vg01.
Examples:
type=Calendar
time=0 23 * * *
desc=NightContext
action=create %s 32400
This rule creates the context NightContext every day at 11PM. Context has a lifetime of 9 hours.
Suppose you have a configuration file my.conf with one rule in it:
type=SingleWithThreshold
ptype=RegExp
pattern=user (\S+) login failure on (\S+)
desc=Repeated login failures for user $1 on $2
action=shellcmd notify.sh "%s"
window=60
thresh=3
Suppose that SEC observes an input line "user admin login failure on tty1". This matches the pattern 'user (\S+) login failure on (\S+)', and after replacing $1 and $2 with their values, the desc parameter will yield "Repeated login failures for user admin on tty1". SEC will then form a key for the event correlation operation, using the configuration file name, the rule ID, and the event description string:
my.conf | 0 | Repeated login failures for user admin on tty1
Since the rule was the first one in the configuration file, its ID is 0. The ID for the second rule would be 1, for the third rule 2, etc. Note that because the configuration file name and the rule ID are present in the keys, event correlation operations started by different rules will not clash, even if the rules have identical values for the desc parameter.
After calculating the key, SEC will check if there already is an event correlation operation with that key. If such operation exists, the input line will be correlated by the existing operation. Otherwise, a new event correlation operation will be started.
By using appropriate event description string definitions, you can change the scope of individual event correlation operations. For instance, if you use "Repeated login failures for user $1" for the desc parameter in the above rule definition, the following three lines will now be correlated by the same event correlation operation:
user admin login failure on tty1
user admin login failure on tty5
user admin login failure on tty2
In other words, SEC will now just count login failures for different users, disregarding terminal names.
Finally, it should be noted that Single, SingleWithScript, Suppress, and Calendar rules never start event correlation operations, since they don't involve the matching of several events over a certain time window (i.e., their design allows them to complete their work almost immediately without starting an event correlation operation). Therefore, the desc parameter of these rules does not influence the number and the scope of event correlation operations.
When new data appear in SEC input files, only one line will be read at a time (even when more lines are available), after which the input buffer is updated with the line that was read. Input buffer contains the last N lines from the SEC input, in the order they were read (the value of N can be set with the -bufsize option). Updating the input buffer means that the last element (the oldest line) is removed from the end of the buffer, while the new line becomes the first element of the buffer. Note that when synthetic events have been created with the event action and new input data are also available in input files, synthetic events are always read first by SEC. After no more such events are available for reading, SEC will read new data from input files.
After the buffer has been updated (every update always adds only one and removes only one line), the rules from configuration files are processed, matching the rules against the new content of the input buffer. Even when a rule matches and its action list suggests an immediate change in the buffer (e.g., through the event action), the input buffer will _not_ be updated until all the rules have been compared with the current content of the buffer.
Rules from the same configuration file are compared with the input buffer in the order they were given in that file. When multiple configuration files have been specified, each file containing a distinct ruleset, events are processed virtually in parallel - the buffer is always processed by all rulesets. However, the order the rulesets are applied during event processing is determined by the order the files were given at SEC command line. If a -conf option specifies a pattern, SEC uses the Perl glob() function to expand the pattern, and the resulting file list is applied by SEC in the order returned by glob() (this is usually ascending ASCII order). E.g., if you have specified -conf options in the following order:
-conf=/home/risto/*.conf -conf=/home/risto/*.conf2
and directory /home/risto contains files A.conf, B.conf2, and C.conf, then SEC applies these ruleset files in the following order: A.conf, C.conf, and B.conf2. Also, note that even if A.conf contains a Suppress rule for a particular event, the event is still processed by rulesets in C.conf and B.conf2.
After the input buffer has been updated and its content has been processed by the rules, SEC handles the signals that have been received since the last check, and also reads possible output from its child processes.
When the timeout specified with the -cleantime option has expired, SEC also checks the contexts and the event correlation operations that have timers associated with them (e.g., SingleWithSuppress and PairWithWindow rules). This means that the value of -cleantime should be relatively small, in order to keep the event correlation operations accurate. By setting the -cleantime option value to 0, SEC will check the event correlation timers after reading every line, but this will consume more CPU time. The higher the value, the less CPU time will be consumed, so the value of the -cleantime option depends on your requirements. Note that if the -poll_timeout option value exceeds the value given with -cleantime, the -poll_timeout option value takes precedence (i.e., sleeps after unsuccessful polls will not be shortened).
If the -intevents command line option is given, SEC will generate internal events when it is started up, when it receives certain signals, and when it terminates normally. Inside SEC, internal event is treated as if it was a line that was read from a SEC input file. Specific rules can be written to match internal events, in order to take some action (e.g., start an external event correlation module with spawn when SEC starts up). The following internal events are supported:
SEC_STARTUP - generated when SEC is started (once the -intevents option has been given, this event will always be the first event that SEC observes)
SEC_RESTART - generated after SEC has received the SIGHUP signal and all internal data structures have been cleared (this event will be the first event that SEC observes after reloading its configuration)
SEC_SOFTRESTART - generated after SEC has received the SIGABRT signal (this event will be the first event that SEC observes after reloading its configuration)
SEC_SHUTDOWN - generated when SEC receives the SIGTERM signal, or when SEC reaches all EOFs of input files after being started with the -notail option. After generating SEC_SHUTDOWN event, SEC will sleep for 3 seconds before sending SIGTERM to its child processes (if there are any child processes that were started by SEC_SHUTDOWN, the processes will have enough time to set a signal handler for SIGTERM if needed).
Right before generating an internal event, SEC will also set up a context named SEC_INTERNAL_EVENT, in order to distinguish between true internal events and coincidents that look like internal events. The SEC_INTERNAL_EVENT context will be deleted immediately after the internal event has been processed by all rules.
If the -intcontexts command line option is given, or there is an -input option with a context specified, SEC will create an internal context each time a line is read from an input file, or a line is read that was created with the event action. The internal context will be deleted immediately after the line has been matched against all rules. For all input files that have the context name explicitly set with -input=<file pattern>=<context>, the name of the internal context is <context>. If the line was read from the input file <filename> for which there is no context name set, the name of the internal context is _FILE_EVENT_<filename>. If the line was created with the event action, the name of the internal context is _INTERNAL_EVENT. This will help the end user to write rules that will match data from one particular input source only, e.g., the rule
type=Suppress
ptype=TValue
pattern=TRUE
context=[!_FILE_EVENT_/dev/logpipe]
will pass to the following rules only those lines that were read from /dev/logpipe.
The shellcmd, spawn, pipe, and report actions involve the creation of a child process. The communication between SEC and its child processes takes place through pipes (created with Perl pipe opens like open(FH, "| mycommand") or pipe(2) system call). Note that the running time of children is not limited in any way, so long-running processes can be started from SEC. For instance, one could start a SEC agent with the spawn action that runs forever and provides SEC with additional input events. However, note that SEC sends the SIGTERM signal to all its children before termination. If some special exit procedures need to be carried out in the child process (or the child wishes to ignore SIGTERM), then the child must install a handler for the SIGTERM signal.
Note that if a rule definition includes two shellcmd actions (or other actions that call external scripts or programs), the order that these scripts or programs are executed is not determined. For instance, with the following action definition
action=shellcmd cmd1; shellcmd cmd2
cmd2 could well terminate before cmd1, or cmd2 could well start before cmd1 (e.g., when cmd1 is a complex command line and cmd2 is relatively simple, it takes more time from the shell to process and start cmd1 than cmd2).
In order to mask the variables, an extra $ or % must be prepended to them, e.g., $$1 yields $1, and not a dollar sign followed by the first value returned by the pattern. Also, all occurrences of $$ will be converted to $ when $<number> variables are substituted, and all occurrences of %% will be converted to % when %<number> or %<alnum_name> variables are substituted, so that one could use $ or % together with variable names (e.g., $$$1 will yield a dollar sign that is followed by the first value returned by the pattern).
Please note the following caveat for Pair and PairWithWindow rules when the second pattern shadows the $<number> variables of the first pattern - since both %<number> and %<alnum_name> variables may be used in the second action list and %<number> variables are substituted first (at the moment when the first pattern matches input line(s) and the event correlation operation is started), all occurrences of %% in the second action list have already been replaced with a single percent sign by the time when %<alnum_name> variables are finally substituted. (E.g., in order to express %%t in the second action list, one must actually write it as %%%%t or %%%t, since after %% to % conversion they both yield %%t.)
SEC allows the user to define patterns, context expressions, and actions which involve calls to the Perl eval() function. In addition to explicitly using %<alnum_name> variables that are global across the rules, the user can implicitly employ Perl variables created with eval() for the same purpose. E.g., when SEC has executed the following action
action=eval %a ($b = 1)
the variable $b and its value will become visible in the following context expression
context= =(++$b > 10)
(with that expression one can implement event counting implicitly). In order to avoid possible clashes with the variables inside the SEC code itself, all calls for eval() are made in the main::SEC namespace (i.e., inside the special package main::SEC). This still leaves the user with an opportunity to set the variables within the SEC code (e.g., by using the main:: prefix), and thus to alter the behavior of SEC, but this is only recommended for advanced users who have studied the code thoroughly and know what they are doing.
type=single
continue=takenext
ptype=regexp
pattern=ftpd\[(\d+)\]: \S+ \(ristov2.*FTP session opened
desc=ftp session opened for ristov2 pid $1
action=create ftp_$1
type=single
continue=takenext
ptype=regexp
pattern=ftpd\[(\d+)\]:
context=ftp_$1
desc=ftp session event for ristov2 pid $1
action=add ftp_$1 $0; set ftp_$1 1800 \
(report ftp_$1 /bin/mail root@localhost)
type=single
ptype=regexp
pattern=ftpd\[(\d+)\]: \S+ \(ristov2.*FTP session closed
desc=ftp session closed for ristov2 pid $1
action=report ftp_$1 /bin/mail root@localhost; \
delete ftp_$1
The first rule creates the context with the name ftp_<pid> when someone connects from host ristov2 with ftp. The second rule adds all logfile lines that are associated with the session <pid> to the event store of the context ftp_<pid> (before adding a line, the rule checks if the context exists). After adding a line, the rule extends context's lifetime for 30 minutes and sets the action list that will be executed when the context expires. The third rule mails collected logfile lines to root@localhost when the session <pid> is closed. Collected lines will also be mailed when the session <pid> has been inactive for 30 minutes (no logfile lines observed for that session).
Note that the logfile line that has matched the first rule will be passed to the second rule and will become the first line in the event store (the first rule has the continue parameter set to TakeNext). The second rule has also its continue parameter set to TakeNext, since otherwise no logfile lines would reach the third rule.
type=Calendar
time=59 1 * * *
desc=WAITING FOR BACKUP
action=event %s
type=PairWithWindow
ptype=SubStr
pattern=WAITING FOR BACKUP
desc=Backup not ready!
action=shellcmd sendtrap.sh "%s"
ptype2=SubStr
pattern2=BACKUP READY
desc2=Backup ready
action2=none
window=960
The first rule generates "WAITING FOR BACKUP" event every night at 1:59AM. This event is matched by the second rule, which starts a correlation operation that will wait for "BACKUP READY" event for the next 16 minutes. If the event arrives on time, no action is executed, otherwise
sendtrap.sh "Backup not ready!"
type=SingleWithScript
ptype=RegExp
pattern=node (\S+) interface (\S+) down
script=not_resp.sh $2
desc=NODE $1 IF $2 DOWN
action=event %s
type=Pair
ptype=RegExp
pattern=NODE (\S+) IF (\S+) DOWN
desc=Interface $2 is down at node $1
action=shellcmd notify.sh "%s"
ptype2=SubStr
pattern2=node $1 interface $2 up
desc2=Interface $2 is up at node $1
action2=shellcmd notify.sh "%s"
window=86400
If "node <node> interface <interface> down" event is observed, the interface is checked with the not_resp.sh script. If the interface is found to be down (not_resp.sh returns 0 as its exit code), event "NODE <node> IF <interface> DOWN" is generated which will be matched by the second rule. The second rule starts a correlation operation that calls
notify.sh "Interface <interface> is down at node <node>"
and then waits for the "node <node> interface <interface> up" event for the next 24 hours. When that event is observed, the correlation operation calls
notify.sh "Interface <interface> is up at node <node>"
Description of Error:
Disk drive (CRU: A1) has failed and can no longer be accessed. (SP Event Code 0xA07)
Probable Cause / Recommended Action:
Replace the disk module (CRU: A1).
You would like to receive an e-mail message if something happens to the disk box. You would like to use different e-mail address at night-time and also receive a report of all night events. The problem here is that useful information is scattered over 7 lines and needs to be consolidated into a single event. Consider the following rules for accomplishing this task:
type=Calendar
time=0 22 * * *
desc=night
action=create %s 36000 \
(report %s /bin/mail root@localhost)
type=Single
ptype=RegExp7
pattern=Description of Error:\n.*\n(.+)\n.*\n.*\n.*\n(.+)
context=night
desc=Error=$1 Recmnd=$2
action=shellcmd nightalarm.sh "%s"; add night %s
type=Single
ptype=RegExp7
pattern=Description of Error:\n.*\n(.+)\n.*\n.*\n.*\n(.+)
desc=Error=$1 Recmnd=$2
action=shellcmd alarm.sh "%s"
The first rule creates the context night with the lifetime of 10 hours every day at 10PM. The second rule specifies that the script nightalarm.sh must be used for sending alert messages at nights, otherwise the script alarm.sh should be used. Every night-time event is added to the context night, and collected events will be mailed to root@localhost at 8AM.
# Set up contexts NIGHT and WEEKEND for nights
# and weekends. The context NIGHT has a lifetime
# of 8 hours and the context WEEKEND 2 days
type=Calendar
time=0 23 * * *
desc=NIGHT
action=create %s 28800
type=Calendar
time=0 0 * * 6
desc=WEEKEND
action=create %s 172800
# If a router does not come up within 5 minutes
# after it was rebooted, generate event
# "<router> REBOOT FAILURE". The next rule matches
# this event, checks the router with ping and sends
# a notification if there is no response.
type=PairWithWindow
ptype=RegExp
pattern=(\S+) \d+: %SYS-5-RELOAD
desc=$1 REBOOT FAILURE
action=event %s
ptype2=RegExp
pattern2=($1) \d+: %SYS-5-RESTART
desc2=$1 successful reboot
action2=logonly
window=300
type=SingleWithScript
ptype=RegExp
pattern=(\S+) REBOOT FAILURE
script=not_responding.sh $1
desc=$1 did not come up after reboot
action=shellcmd notify.sh "%s"
# Send a notification if CPU load of a router is too
# high (two CPUHOG messages are received within 5
# minutes); send another notification if the load is
# normal again (no CPUHOG messages within last 15
# minutes). Rule is not applied at night or weekend.
type=SingleWith2Thresholds
ptype=RegExp
pattern=(\S+) \d+: %SYS-3-CPUHOG
context=!(NIGHT || WEEKEND)
desc=$1 CPU overload
action=shellcmd notify.sh "%s"
window=300
thresh=2
desc2=$1 CPU load normal
action2=shellcmd notify.sh "%s"
window2=900
thresh2=0
# If a router interface is in down state for less
# than 15 seconds, generate event
# "<router> INTERFACE <interface> SHORT OUTAGE";
# otherwise generate event
# "<router> INTERFACE <interface> DOWN".
type=PairWithWindow
ptype=RegExp
pattern=(\S+) \d+: %LINK-3-UPDOWN: Interface (.+), changed state to down
desc=$1 INTERFACE $2 DOWN
action=event %s
ptype2=RegExp
pattern2=($1) \d+: %LINK-3-UPDOWN: Interface ($2), changed state to up
desc2=$1 INTERFACE $2 SHORT OUTAGE
action2=event %s
window=15
# If "<router> INTERFACE <interface> DOWN" event is
# received from the previous rule, send a notification and
# wait for "interface up" event for the next 24 hours
type=Pair
ptype=RegExp
pattern=(\S+) INTERFACE (\S+) DOWN
desc=$1 interface $2 is down
action=shellcmd notify.sh "%s"
ptype2=RegExp
pattern2=($1) \d+: %LINK-3-UPDOWN: Interface ($2), changed state to up
desc2=$1 interface $2 is up
action2=shellcmd notify.sh "%s"
window=86400
# If ten "short outage" events have been observed
# in the window of 6 hours, send a notification
type=SingleWithThreshold
ptype=RegExp
pattern=(\S+) INTERFACE (\S+) SHORT OUTAGE
desc=Interface $2 at node $1 is unstable
action=shellcmd notify.sh "%s"
window=21600
thresh=10