Chapter 5: Running `stealth'

Now that stealth has been compiled, the construction of a policy file has been covered, and a service-account on the client has been defined, what must be done to run stealth in practice?

Here's what remains to be done:

In this chapter, these topics will be discussed.

5.1: Installing `stealth'

As stealth is mainly a system administrator's tool, it could be installed in /usr/local/bin. In that case, do (as root) from the directory where stealth was compiled/unpacked:

    install stealth /usr/local/bin
        
options given to install(1) may restrict further use of stealth.

5.2: Construct one or more policy files

Here we assume that stealth is run by root, and that root wants to store information about the host client under the subdirectory /root/stealth/client.

Stealth reports should be sent to the user admin@elsewhere, who is only interested in a short notice of changes, as the full report can always be read elsewhere. So, a support-script is developed to further filter the report generated by stealth.

As the sha1sum program on the client may be hacked, it is a good idea to transfer the client's sha1sum program to the controller first, in order to check that program locally, before trusting it to compute the sha1sums of the client's files. The same holds true for any libraries and support programs (like find) that are used intensively during integrity scans

Sha1sum checks should be performed on all setuid and setgid files on the client, and in order to be able reach all files on client, root@controller is allowed to login to the root@client account using a password-less ssh connection.

Furthermore, sha1sum checks should be performed on all configuration files, living under /etc and on the file /usr/bin/find which is used intensively to perform the checks.

The required policy file is constructed as follows, per section:

5.2.1: the DEFINE directives


    DEFINE  SSHCMD  /usr/bin/ssh root@client -T -q exec /bin/bash --noprofile
    DEFINE  EXECSHA1 -xdev -perm +u+s,g+s \( -user root -or -group root \) \ 
                    -type f -exec /usr/bin/sha1sum {} \;
        
The first DEFINE defines the ssh command to use: an ssh-connection will be made to the root account at the client.

The second DEFINE shows the arguments for find(1) when looking for all root setuid or setgid normal files. For all these files the sha1sum(1) program should be run.

5.2.2: the USE directives


    USE BASE        /root/stealth/client
    USE EMAIL       admin@elswhere
    USE MAILER      /root/bin/stealthmail
    USE MAILARGS    "Client STEALTH report"
    USE SSH         ${SSHCMD}
        

5.2.3: the commands

First, we'll copy the client's sha1sum program to the controller. In practice, this should also include the shared object libraries that are used by sha1sum, as they might have become corrupted as well.

5.2.3.1: Obtain the client's sha1sum program

First, the sha1sum program is copied to a local directory


    GET /usr/bin/sha1sum /root/tmp
        
This command must succeed.

5.2.3.2: Check the integrity of the client's sha1sum program

Next, we'll check the received sha1sum program, using our own:


    LABEL \nCheck the client's sha1sum program
    LOCAL CHECK LOG = local/sha1 /usr/bin/sha1sum /root/tmp/sha1sum
        
The LABEL command will write the label to the report file just before the output of the sha1sum program is generated.

The LOCAL command will check the sha1sum of the program copied from the client. The report is written on the file /root/stealth/client/local/sha1. If this fails, the program will not continue, but will alert admin@elsewhere that the check failed. This is of course rather serious, as it indicates that either the controller's sha1sum is behaving unexpectedly or that the client's sha1sum program has changed.

The sha1sum program may have changed due to a normal upgrade. If so, admin@elsewhere will know this, and can (probably) ignore the warning. The next time stealth is run, the (now updated) SHA1 value is used, and it should again match the obtained SHA1 value from the copied sha1sum program.

5.2.3.3: Check the client's /usr/bin/find command

The client will use it's find command intensively: find is a great tool for producing files having almost any conceivable combination of characteristics. Of course, the client's find command itself must be ok, as well as the client's sha1sum program. Now that we know that the client's sha1sum program is ok, we can use it to check the client's /usr/bin/find program.

Note that the controller itself will not suffer any processing load here: only the client itself is taxed for checking the intergrity of its own files:


    LABEL \nchecking the client's /usr/bin/find program
    CHECK LOG = remote/binfind /usr/bin/sha1sum /usr/bin/find
        

5.2.3.4: Check the client's setuid/setgid files

Having checked the client's sha1sum and find programs, sha1 checksum checks should be performed on all setuid and setgid files on the client. For this we activate the sha1sum program on the client. In order to check the setuid/setgid files, the following command is added to the policy file:


    LABEL \nsuid/sgid/executable files uid or gid root on the / partition     
    CHECK LOG = remote/setuidgid /usr/bin/find / ${EXECSHA1}
        

5.2.3.5: Check the configuration files in the client's /etc/ directory

Finally, the client's configuration files are checked. Some of these files change so frequently that we don't want them to be checked. E.g., /etc/adjtime, /etc/mtab. To check the configuration file, do:


    LABEL \nconfiguration files under /etc
    CHECK LOG = remote/etcfiles                         \ 
          /usr/bin/find /etc -type f -not -perm +6111   \ 
            -not -regex "/etc/\(adjtime\|mtab\)"        \ 
            -exec /usr/bin/sha1sum {} \;
        

5.2.4: The complete `policy' file

Here is the complete policy file that we've constructed so far:

    DEFINE  SSHCMD  /usr/bin/ssh root@client -T -q exec /bin/bash --noprofile
    DEFINE  EXECSHA1 -xdev -perm +u+s,g+s \( -user root -or -group root \) \ 
                    -type f -exec /usr/bin/sha1sum {} \;

    USE BASE        /root/stealth/client
    USE EMAIL       admin@elswhere
    USE MAILER      /root/bin/stealthmail
    USE MAILARGS    "Client STEALTH report"
    USE SSH         ${SSHCMD}

    USE DD          /bin/dd
    USE DIFF        /usr/bin/diff
    USE PIDFILE     /var/run/stealth-
    USE REPORT      report
    USE SH          /bin/sh

    GET /usr/bin/sha1sum /root/tmp

    LABEL \nCheck the client's sha1sum program
    LOCAL CHECK LOG = local/sha1 /usr/bin/sha1sum /root/tmp/sha1sum

    LABEL \nchecking the client's /usr/bin/find program
    CHECK LOG = remote/binfind /usr/bin/sha1sum /usr/bin/find
 
    LABEL \nsuid/sgid/executable files uid or gid root on the / partition     
    CHECK LOG = remote/setuidgid /usr/bin/find / ${EXECSHA1}

    LABEL \nconfiguration files under /etc
    CHECK LOG = remote/etcfiles                         \ 
          /usr/bin/find /etc -type f -not -perm +6111   \ 
            -not -regex "/etc/\(adjtime\|mtab\)"        \ 
            -exec /usr/bin/sha1sum {} \;




5.3: Running `stealth' for the first time

When stealth is now run, it will create its initial report files under root/stealth/client.

The first time stealth is run, it is usually run `by hand':


    stealth policy
        
this will show all executed commands on the standard output, and will initialize the reports. Running stealth this way for the just constructed policy file results in the following output (lines were wrapped to improve readability):

    GET /usr/bin/sha1sum /root/tmp
    LABEL \nCheck the client's sha1sum program
    LOCAL CHECK LOG = local/sha1 /usr/bin/sha1sum /root/tmp/sha1sum
    LABEL \nchecking the client's /usr/bin/find program
    CHECK LOG = remote/binfind /usr/bin/sha1sum /usr/bin/find
    LABEL \nsuid/sgid/executable files uid or gid root on the / partition     
    CHECK LOG = remote/setuidgid /usr/bin/find / -xdev -perm +u+s,g+s 
            \( -user root -or -group root \) -type f 
            -exec /usr/bin/sha1sum {} \;
    LABEL \nconfiguration files under /etc
    CHECK LOG = remote/etcfiles                         /usr/bin/find /etc 
            -type f -not -perm +6111   -not -regex "/etc/\(adjtime\|mtab\)"
            -exec /usr/bin/sha1sum {} \;
    LOCAL /usr/bin/scp -q root@client:/usr/bin/sha1sum /root/tmp
    LABEL \nCheck the client's sha1sum program
    LOCAL CHECK LOG = local/sha1 /usr/bin/sha1sum /root/tmp/sha1sum
    LABEL \nchecking the client's /usr/bin/find program
    CHECK LOG = remote/binfind /usr/bin/sha1sum /usr/bin/find
    LABEL \nsuid/sgid/executable files uid or gid root on the / partition     
    CHECK LOG = remote/setuidgid /usr/bin/find / -xdev -perm +u+s,g+s 
            \( -user root -or -group root \) -type f 
            -exec /usr/bin/sha1sum {} \;
    LABEL \nconfiguration files under /etc
    CHECK LOG = remote/etcfiles                        /usr/bin/find /etc 
            -type f -not -perm +6111   -not -regex "/etc/\(adjtime\|mtab\)"
            -exec /usr/bin/sha1sum {} \;
    

This all produces the following output:

5.3.1: The mailed report

The /root/bin/stealthmail is called with the following arguments:


    "Client STEALTH report" admin@elswhere
        

The contents of the mailed report now is (the date will of course change, the next time stealth is run):


    STEALTH (1.21) started at Mon Nov 24 10:50:30 2003
        
    Check the client's sha1sum program
    Initialized report on local/sha1
    
    checking the client's /usr/bin/find program
    Initialized report on remote/binfind
    
    suid/sgid/executable files uid or gid root on the / partition     
    Initialized report on remote/setuidgid
    
    configuration files under /etc
    Initialized report on remote/etcfiles
        

5.3.2: Files under /root/stealth/client

Under /root/stealth/client the following entries are now available:

This completes the information created by stealth during its first run.

5.4: Running `stealth' again: all files unaltered

When stealth is run again, it will update its report files under root/stealth/client. If nothing has changed, the log-files will remain unaltered. The new run will, however, produce some new info on the file /root/client/report:

    STEALTH (1.21) started at Mon Nov 24 10:50:30 2003
    
    Check the client's sha1sum program
    Initialized report on local/sha1
    
    checking the client's /usr/bin/find program
    Initialized report on remote/binfind
    
    suid/sgid/executable files uid or gid root on the / partition     
    Initialized report on remote/setuidgid
    
    configuration files under /etc
    Initialized report on remote/etcfiles
    
    
    STEALTH (1.21) started at Mon Nov 24 10:54:35 2003
        
Note that just one extra line was added: a timestamp showing the date/time of the last run. The systems administrator may reduce/remove the report file every once in a while to reclaim some disk space.

5.5: Running `stealth' again: modifications have occurred

Basically, three kinds of modifications are possible: additions, modifications, and removals. Here we'll show the effect all these changes have on stealth's output.

For the example, the following changes were made to the client's files:

Next, stealth was once again run, producing the following output:

5.6: Failing LOCAL commands

If the client's sha1sum program itself is altered, a serious situation has developed. In that case, further actions by stealth would be suspect, as their results might easily be currupted. Checks will proceed, but a warning is generated on the report file (and in the mail sent to admin@elsewhere:


STEALTH (1.21) started at Mon Nov 24 10:54:35 2003

Check the client's sha1sum program
MODIFIED: /root/tmp/sha1sum
    < fc62fc774999584f1e29e0f94279a652  /root/tmp/sha1sum
    > 45251e259bfaf1951658a7b66c328c52  /root/tmp/sha1sum

*** BE CAREFUL *** REMAINING RESULTS MAY BE FORGED

configuration files under /etc
REMOVED: /etc/motd.org
    > 945d0b8208e9861b8f9f2de155e619f9  /etc/motd.org
MODIFIED: /etc/motd
    < 945d0b8208e9861b8f9f2de155e619f9  /etc/motd
    > 7f96195d5f051375fe7b523d29e379c1  /etc/motd
        
(The report shows the removal of the previously added file motd.org, and the modification of motd. These are real, as the original motd file, modified earlier, was restored at this point).

5.7: Automating `stealth' runs using `cron'

In order to automate the execution of stealth, a file /etc/cron.d/stealth could be created, containing a line like (assuming stealth lives in /usr/bin):

    2,17,32,47 * * * *  root    test -x /usr/bin/stealth && \ 
                                /usr/bin/stealth -q /root/stealth/client.pol
        
This will start stealth 2 minutes after every hour. Alternate schemes are left to the reader to design.

In general, randomizing events makes it harder to notice them. stealth may start its tasks at a random point in time if its -i flag (for random interval) is used. This flag expects an argument in seconds (or in minutes, if at least an m is appended to the interval specification). Somewhere between the time stealth starts and the specified interval the scan will commence. For example, the following two commands have identical effects: the scan is started somewhere between the moment stealth was started and 5 minutes:


    stealth -i 5min -q /root/stealth/client.pol
    stealth -i 300  -q /root/stealth/client.pol
        
When the -d flag is given, the -i flag has no effect.

As another alternative, stealth my be started specifying the --keep-alive pidfile option. Here, pidfile is the name of a file that will contain the process id of the stealth process running in the background. For example:

    
    stealth --keep-alive /var/run/stealth -i 300  -q /root/stealth/client.pol
        
Now, cron(1) may be used to restart this process at indicated times:

    2,17,32,47 * * * *  root    test -x /usr/bin/stealth && \ 
                                   /usr/bin/stealth --rerun /var/run/stealth
        

As yet another alternative, the cron-job may activate a script performing stealth's rerun, starting another stealth run if necessary. The advantage of such an approach is that stealth is automatically started after, e.g., a reboot. The following script expects two arguments (both of which must be absolute paths). The first argument is the path to the pidfile to use, the second argument is the path to the policy file to use. The script is found in the distribution package as /usr/share/doc/stealth/usr/bin/stealthcron:

#!/bin/bash

PROG=`basename $0`
STEALTH=/usr/bin/stealth

testAbsolute()
{
    echo $1 | grep "^/" > /dev/null 2>&1 && return

    echo "\`$1' must be absolute path"
    exit 1
}

case $# in
    (2)
        testAbsolute $1
        testAbsolute $2

        if [ -x ${STEALTH} ] ; then
            ${STEALTH} --rerun $1
            [ $? -eq 0 ] || ${STEALTH} --keep-alive $1 -q $2
        fi
    ;;
            
    (*)
    echo "
$PROG by Frank B. Brokken (f.b.brokken@rug.nl)
Usage: $PROG [sleep] pidfile configfile
where:
    pidfile:    absolute path to pidfile to be used by ${STEALTH}
    configfile: absolute path to configuration file to be used by ${STEALTH}

    calls $STEALTH} --rerun pidfile. 
    If that fails, ${STEALTH} --keep-alive pidfile -q configfile is started.
"
    exit 1
    ;;
esac
The script could be called from /etc/cron.d/stealth using a line like

22  8  * * * root test -x /usr/bin/stealthcron && /usr/bin/stealthcron
        /var/run/stealth.target /usr/share/stealth/target.pol
        
Note that the command should be on a single line. It was spread out here over two lines to enhance readability. Also note that the directory /var/run is usually not writable to non-root users. If stealth is used by a non-root user another directory should be used to store stealth.target in.

5.8: Report File Rotation

When stealth performs integrity scans it will append information to the report file. This file will therefore eventually grow to a large size, and the systems manager controlling stealth might want to rotate the report file every once in a while (e.g., using a program like logrotate(1), also see the upcoming section 5.8.2). In order to ensure that no log-rotation takes place while stealth is busy performing integrity scans (thus modifying the report file) the options --suppress and --resume were implemented. Both options require the process-ID file of currently active stealth process as their argument.

For example, if a stealth process was once started using the command


    stealth -q --keep-alive /var/run/stealth.small --repeat 900 \ 
                    /var/stealth/policies/small.pol
        
then the --suppress and --resume commands for this process should be formulated as:

    stealth --suppress /var/run/stealth.small
    stealth --resume /var/run/stealth.small
        
The stealth process identified in the files provided as arguments to the --suppress and --resume options is called the targeted stealth process below.

The --suppress option has the following effect:

Now that the report file will no longer be affected by the targeted stealth process, log-rotation may take place. E.g., a program like logrotate(1) allows its users to specify a command or script just before log-rotation takes place, and `stealth --suppress pidfile' could be specified nicely in such a pre-rotation section.

The --resume option has the following effect:

Note that, once --suppress has been issued, all commands except --resume and --terminate are ignored by the targeted stealth process. While suppressed, the --terminate command is acknowledged as a `emergency exit' which may or may not interfere with, e.g., an ongoing log-rotation process. The targeted stealth process should not normally be terminated while it is in its suppressed mode. The normal way to terminate a stealth process running in the background is:

5.8.1: Status file cleanup

Whenever stealth is run and it encounters a modified situation the already existing status file that is used to summarize that particular situation is saved and a new status file is created. Eventually, this will result in many status files. While report files can be rotated, it is pointless to rotate old status files, since they never are modified. Instead status files exceeding a certain age could be removed and more recent files might be zipped to conserve space. In stealth's binary distribution the file /usr/share/doc/stealth/usr/bin/stealthcleanup is provided which can be used to perform this cleanup. The script expects one argument: a resource file defining the following shell variables: Here is the stealthcleanup script as it is found in the binary distribution's /usr/share/doc/stealth/usr/bin directory:
#!/bin/bash

usage()
{
    echo "
Usage: $0 rc-file
Where:
    rc-file: resource file defining:
            \`directories' - one or more directories containing status files
            \`gzdays'      - number of days status files may exist before they
                             are compressed
            \`rmdays'      - number of days gzipped status files may exist
                             before they are removed. 
"
    exit 1
}


error()
{
    echo "$*" >&2
    exit 1
}

[ $# == 1 ] || usage

# now source the configuration file
. $1           

for x in $directories
do
    cd $x || error "\`$x' must be a directory"

    /usr/bin/find ./ -mtime +$rmdays -type f -regex '.*[0-9]+-[0-9]+\.gz' \
        -exec /bin/rm {} \;

    /usr/bin/find ./ -mtime +$gzdays -type f -regex '.*[0-9]+-[0-9]+' \
        -exec /bin/gzip {} \;
done

exit 0




Assuming that the status files are written in /var/stealth/target/local and /var/stealth/target/remote; that status file should be compressed when older than 2 days and removed after 30 days, the resource file is:
directories="
    /var/stealth/target/local
    /var/stealth/target/remote
    "

rmdays=30
gzdays=3
Furthermore assuming that the resourcefile is installed in /etc/stealth/cleanup.rc and the stealthcleanup script itself in /usr/bin/stealthcleanup, the stealthcleanup script could be called as follows:

    /usr/bin/stealthcleanup /etc/stealth/cleanup.rc
        
Note that stealthcleanup may be called whether or not there are active stealth processes, as stealth does not use status files anymore once they have been written.

5.8.2: Using `logrotate' to control report- and status files

A program like logrotate(1) allows its users to specify a command or script immediately following log-rotation, and `stealth --resume pidfile' could be specified nicely in such a post-rotation section.

Here is an example of a specification that can be used with logrotate(1). Logrotate (on Debian systems) keeps its configuration files in /etc/logrotate.d, and assuming there is a host target, whose report file is /var/stealth/target/report, the required logrotate(1) specification file (e.g., /etc/logrotate.d/target could be:

/var/stealth/target/report {
    weekly
    rotate 12
    compress
    missingok
    prerotate
        /usr/bin/stealth --suppress /var/run/stealth.target
    endscript
    postrotate
        /usr/bin/stealth --resume /var/run/stealth.target
    endscript 
}
Using this specification file, logrotate(1) will Note thet stealth --resume xxx will always start with another file integrity scan.