What could make one’s Saturday morning better, than a slight hangover? Well idling through a few mail folders, reading a few log messages and discovering that your server has been hacked for over three days is not the thing, for sure.
Being a bit on the paranoid side, I have a few tools up and running, that monitor my filesystems, for changes that were not made by me.
One such tool is tiger which gave me lines like:
# Checking listening processes NEW: --WARN-- [lin003w] The process `bash' is listening on socket 7171 (TCP on every interface) is run by angel. NEW: --WARN-- [lin003w] The process `oidentd' is listening on socket 113 (TCP on every interface) is run by oident.
# Verifying system specific password checks... NEW: --FAIL-- [lin005f] Installed file `/usr/bin/pstree' checksum differs from installed package 'psmisc'. NEW: --WARN-- [lin001w] File `/lib/libproc.a' does not belong to any package. NEW: --WARN-- [lin001w] File `/lib/libproc.so.2.0.6' does not belong to any package. NEW: --WARN-- [lin001w] File `/lib/lidps1.so' does not belong to any package. NEW: --WARN-- [lin001w] File `/sbin/ttyload' does not belong to any package. NEW: --WARN-- [lin001w] File `/sbin/ttymon' does not belong to any package. NEW: --WARN-- [lin001w] File `/usr/sbin/ttyload' does not belong to any package.
# Checking listening processes NEW: --WARN-- [lin003w] The process `bd' is listening on socket 60001 (TCP on every interface) is run by sdb.
This I just ignored, being absent minded at times, I tend to forget when I install new binaries. Altough I clearly remembered, that I haven’t touched the server for quite a time.
But there were other reports as well, from RKHunter that some files were changed on the file system and a new user was added. It turned out, that a day later the guy activated himself, and installed SH4 and SH5 rootkits on the system.
RKHunter and Tiger reported the changes. A few commands were changed, to hide some processes and files. The rootkit is a bit screwed up, as the top command doesn’t work at all.
Since I also keep regular backups of the system I wanted to quickly remove the backdoors, I tried to overwrite the changed files on the system. The changed commands had a few attributes set, so they were read only. To overwrite them with the original files first I had to chattr them.
Unfortunately these rootkits tend to restore themselves from the demons they spawn, which I couldn’t find until the files were restored. Since I had all the information from the security programs it was quite easy to write a script to erase the rootkits and restore the original files.
SRC="/path-to-archive" for i in /bin/ls /bin/netstat /bin/ps /usr/bin/find /usr/bin/md5sum /usr/bin/pstree /usr/bin/top /sbin/ifconfig; do echo $i chattr -sia $i rm $i cp ${SRC}/$i $i done chattr -sia /lib/lidps1.so /etc/sh.conf /dev/srd0 /dev/shm/bd /dev/shm/nou rm /lib/lidps1.so /etc/sh.conf /dev/srd0 /dev/shm/bd /dev/shm/nou chattr -sia -R /lib/libsh.so /usr/lib/libsh /dev/.raw rm -r /lib/libsh.so /usr/lib/libsh /dev/.raw rm /sbin/ttyload /sbin/ttymon /usr/sbin/ttyload
When this was ran, I was able to find and kill the processes from the backdoors. There were strange processes named bash, ttymon, /usr/sbin/httpd, /usr/sbin/apache/log and suchlike, I didn’t recognize. After killing the processes, I’ve re ran the script, to make sure there were no application hooks that recreated the files. Then I went through the entire proc table again with a fine comb.
Post-mortem
It turned out, that the hacker came in through the mldonkey server, that I used to download some torrents. It was ran in a chrooted environment to start with, but this didn’t seem to matter. So much for chroot. The attacker first used a kernel exploit to gain root priviledges. Then he created an account named “angel”and uploaded and installed the rootkits. He also uploaded what seemed to me like an IRC bot, and spread it liberally around the machine. (I found about 5 copies, but I didn’t count)
The kernel vulnerability existed, because even though the new patched kernel was already installed, the system was not restarted to use it. After the reboot the kernel exploit is hopefully gone, but the mldonkey server remains vulnerable. Mldonkey is disabled on the server for now.
Great article. I had this issue and was wondering why even if I was using chattr I couldn’t delete the files. By reading your article I realized that the i attribute was applied to the directories.
Thank you for this article, had same problem as MVU, didn’t figure that with chattr -R i can remove their /usr/lib/libsh.