One of the worst nightmares of every system administrator is to be unaware of intruders. We tend to believe that our servers have not been violated until some kind of evidence, usually of the worst kind, catches our attention.

Well, the backup software Bacula can be, quite surprising in the first place, our allied.

Let’s start by clarifying that I’m not talking about recovering corrupted files after a break in, which is of course part of the game. Bacula has more to offer.

Bacula is in the best position to survey what’s going on across a large installation of servers. Each day Bacula contacts each server and gets informations about files. And what if those information could include ownership, permissions and a hash signature like MD5 or SHA1? Saving that, Bacula could be able to check if something has changed across the night.

Well, the good news are: Bacula already does that and implements a family of jobs called “Verify jobs” to do precisely that. The steps to benefit from this technology are three:

1. configure Bacula
2. take a master scan of all your files on all your server, to be considered the right situation
3. schedule once per day a check on each server to verify that files still match the master scan

Let’s review each step.

Configure Bacula

Configuring for Verify jobs involves declaring a FileSet, a Schedule and a JobDefs and then define as many jobs as you wish.

The Schedule is the easiest part:

Schedule {
        Name = "VerifyCycle"
        Run = Level=Catalog sun-sat at 5:05
}

Just be sure to schedule verify jobs not at the same time of backup jobs, to keep the load down and avoid hogging the network.

Than define a FileSet, including all the files you want to keep under monitoring:

FileSet {
        Name = "Intrusion Set"
        Include {
                Options {
                        verify=pins5
                        signature=MD5
                }
                File = /boot
                File = /bin
                File = /sbin
                File = /usr/bin
                File = /lib
        }
}

My FileSet is actually much longer and includes a lot of single files and directories from /etc. The signature option tells bacula to compute an MD5 hash of each file, while the verify option states to use that signature (5) and to check also permissions (p), inodes (i), number of links (n) and size (s).

Now we create a job template (JobDefs in Bacula lingo):

JobDefs {
        Name = "DefaultVerifyJob"
        Type = Verify
        Level = Catalog
        Messages = Standard
        Pool = "Default"
        Schedule = "VerifyCycle"
}

Note the Type set to Verify and the Schedule set to VerifyCycle, as we defined it a moment ago. Now we are ready to setup as many verify jobs as we need. I’ll use two hosts as examples and assume the Client entry for each of them are already in place, since it’s the very same directive Bacula should use to identify the clients for backup purposes.

Job {
        Name = "verify:host1"
        Client = "host1.strumentiresistenti.org"
        FileSet = "Intrusion Set"
        JobDefs = "DefaultVerifyJob"
}

Job {
        Name = "verify:host2"
        Client = "host2.strumentiresistenti.org"
        FileSet = "Intrusion Set"
        JobDefs = "DefaultVerifyJob"
}

Don’t forget to restart Bacula.

Preparing to check the master scan

So far we have informed Bacula that we plan to execute two verify jobs on host1 and host2, using the FileSet called “Intrusion Set” every night at 5:05. But Bacula needs one more information: what to consider a healthy situation!

So, before starting to verify our hosts, we need to make a scan for each one and mark it as the master scan. Let’s use bconsole to do it:

# bconsole
Connecting to Director 192.168.122.130:9101
1000 OK: bacula.strumentiresistenti.org Version: 5.0.1 (24 February 2010)
Enter a period to cancel a command.
*run job=verify:host1.strumentiresistenti.org level=InitCatalog yes

Do the same for each host you configured.

Testing: run a verify job

Now Bacula knows how to check your hosts every night at 5:05 AM. But you are a true sysadmin and don’t want to wait tomorrow to know if your configuration is right. So, let’s start a verify job, using once again bconsole:

# bconsole
Connecting to Director 192.168.122.130:9101
1000 OK: bacula.strumentiresistenti.org Version: 5.0.1 (24 February 2010)
Enter a period to cancel a command.
*run job=verify:host1.strumentiresistenti.org yes

That’s it! Just remove the level= parameter (which defaults to Catalog) and your verify job will be scheduled. After it has completed, you can check the output in bconsole using messages command. If you have configured Bacula to send you an email for each job (and you definitely should!) you’ll receive the same output in your mailbox. Just remember to check it out every day when you begin your work day.

The output you’ll get

Now let’s take a look at the output Bacula generates on each verify job. Look in the mail for a message from Bacula or use bconsole to read the message ringbuffer with command messages (did you know you can use just m to read all the messages? Bacula has shortcuts)

You should find a section like the following one:

  Build:                  x86_64-pc-linux-gnu ubuntu 10.04
  JobId:                  283
  Job:                    verify:host1.strumentiresistenti.org.2012-05-09_05.05.01_02
  FileSet:                Intrusion Set
  Verify Level:           Catalog
  Client:                 host1.strumentiresistenti.org
  Verify JobId:           267
  Verify Job:             
  Start time:             09-May-2012 05:08:03
  End time:               09-May-2012 05:08:20
  Files Examined:         9,705
  Non-fatal FD errors:    9
  FD termination status:  OK
  Termination:            Verify OK

The Verify OK termination is telling you that everything is fine. If something has changed, the OK is replaced with Verify Differences and that precisely what you must do. Look for lines like:

09-May 05:07 bacula.strumentiresistenti.org JobId 279: File: /root/.ssh/known_hosts
09-May 05:07 bacula.strumentiresistenti.org JobId 279:       st_size  differ. Cat:
  2210 File: 2652
09-May 05:07 bacula.strumentiresistenti.org JobId 279:       MD5 digest differs.
  File=Miaep7k+EteZ6fVGTsN/eQ Cat=pfIHd8qqfIU9mUUz3nHIwQ

As you can see, the file /root/.ssh/known_hosts has changed. Some one has made a new SSH connection somewhere! An intruder? No, wait, you did it yesterday to move some files across two servers with scp. Well, now Bacula will keep sending you a Verify Differences every night about host1. How can you tell Bacula that everything is fine?

Simple: just reinitialize the Catalog for host1 with bconsole command:

* run job=verify:host1.strumentiresistenti.org level=InitCatalog yes

Exactly as you did the first time.

That’s all, folks. I hope you’ll find usefull this guide. See you on next post.