After spending hours on the net looking for a solution I came over a Swedish forum, which gave away the solution.

1. Reboot the box, either by power cycling or from the remote
2. Press the Menu on the remote when the boot starts
3. Change the output to the resolution your TV can do
4. save and reboot

As I’m not very good at interactive HTML mock-ups I decided to go with Wicket, which seemed to be useful and easy when a friend showed me. The mock-up progressed well, but upon deployment all Hell broke loose.

I chose Wicket, because I hate most web frameworks already, Struts is way to old and twisted for anything proper, I have not found a single open source JSF implementation that works properly, and I had nothing concrete against Wicket yet.

I also wanted to try Maven, since I hate it’s conception so much, it seemed like a good time to try, after all we had two days on our hands, why not waste it properly? I still hate it, but only on principle, otherwise it to me it”s a lot like the apt of java. And my world is Debian based.

So the mock-up went well. I’ve spent hours figuring out how to do a simple menu that shows selected items with the <wicket:link> feature, with mouseovers. I managed to get it working, to realize it would never work on IE. Then I reimplemented it, then again. Finally I threw it all away and added a JavaScript onload and some brute-force to get it working.

function fireOnLoad() {
  // highlight current menu
  var menuTable = window.document.getElementById('menuTable');
  for (var i=0; i<menuTable.rows[2].cells.length; i++) {
    var curM = menuTable.rows[2].cells[i];
    if(curM.children[1].tagName !='A') {
      var curClassName = curM.className + 'Selected';
      curM.className = curClassName ;
      curM.children[0].style.display='block';
      menuTable.rows[1].cells[i].children[0].style.display='block';
    }
  }
  return true;
}

For the HTML code like

<table id="menuTable" class="menuTable" border="0">
<tbody>
<tr>
<td class="subMenuItemContainer"></td>
</tr>
<tr>
<td class="subMenuPointer">
    <img style="display:none" onclick="window.location.href='../img/submenu.gif';return false;" src="../img/submenu.gif" alt="" /></td>
...</tr>
<tr class="menuItemContainer">
<td class="menuItem" onmouseover="return menuRollOver(this)" onmouseout="return menuRollOut(this)">
<div class="hiddenSubMenu">
<ul class="subMenu">
	<li><a href="../pages/AgencyList">Keresés</a></li>
</ul>
</div>
<a href="../pages/AgencyList">Ügynökségek</a></td>
...</tr>
</tbody></table>

It’s quite ugly, but adequate for the current situation.

After the pages were ready and tested on localhost I decided to move it to our server for presentation. The server runs Debian and has an Apache Tomcat 5.5 running on it for ages. It turned out however, that somehow the project requires Tomcat 6. Not wanting to go over all the project’s config, I went for the server upgrade. The Tomcat had no applications that were to be effected, so it seemed like an easy choice.

Install from apt went smoothly, the application deployed easily. There was a strange problem though.

Every time I clicked a link it either didn’t display any images or it displayed some, or none at all. Refreshing the page always changed something. I suspected, it must be an issue with the Wicket filter, but could not put my hand on it.

Trying to isolate the problem I tried loading a single image from the application. It turned out, that a security exception is thrown every time I try to reload a resource that was previously loaded. Turning of security in Tomcat packaged for Debian did not work out of the box, so I tried to find a solution. I found the page on Java security in Wicket but adding the values on the page just resulted in another error.

I decided to add all rules found on the Wicket WIKI, but that wasn’t enough. So I was adding the rules one-by-one to come up with my version of the security policy for 1.4

grant {
// For substitution of one object for another during serialization
// or deserialization. This is used in ReplaceObjectOutputStream,
// which is used for page versioning (undoing changes).
//permission java.io.SerializablePermission "enableSubstitution";

// For FilePageStore's custom serialization
//permission java.io.SerializablePermission "enableSubclassImplementation";

// For crypted URL functionality (see WebRequestWithCryptedUrl).
//permission java.security.SecurityPermission "insertProvider.SunJCE";

// The following was required to get Wicket, at least the examples, to work at all
permission java.lang.reflect.ReflectPermission
   "suppressAccessChecks";
permission java.lang.RuntimePermission
   "accessClassInPackage.org.apache.tomcat.util.http";
permission java.util.PropertyPermission
   "org.apache.tomcat.util.http.FastHttpDateFormat.CACHE_SIZE",
   "read";
};

As it turned out the problem is gone now. At least until I hit another security restriction.

This seemed like such an easy task to do, what could be hard in creating a disk image with CloneZilla, copy it to a server, set up a virtual machine there with kvm, restore the image and redirect all traffic to this computer instead of the one in our office. We estimated it could be done in two to three hours tops, and we get home around 7PM.

We missed a few points though. The machine had a faulty CD drive, so booting CloneZilla was not so easy. We went for the network boot option, but something just didn’t work out on the PXE boot. So we switched the CD drive to a working one, well at least it used to work a few years ago… But not anymore. So we decided to scrap the swap partition boot CloneZilla from there, and create our image. It worked! Almost. Except that CloneZilla didn’t quite identify the disk partition types, and didn’t see the RAID device at all. It turned out, that the utility somehow runs in user mode and starting the partimage utility with sudo is an acceptable workaround. We then had the disk image on a different server, where we tried to restore it into a 10GB virtual disk image. As it turns out, CloneZilla is unable to restore images that are bigger then the target partition even if there is less data in it.

So we went on a quest for a parted that can shrink our 100GB partition to 10GB. Well, if there was one… As we had no intention of booting from the non-working CD we popped in a disk from our jukebox server with Ubuntu on it. No wonder it worked flawlessly, until I learned that a RAID 1 device with ext3 on it is not easy to shrink. So I decided to break the RAID block, remove the incompatible ext3 flags, and resize the first partition to 10GB. Then I used partimage to create the backup, it was created with about 300MB/min.

The backup was quickly copied over to the eagerly waiting server, that was to host the virtual machine. We fired up the VM with a 10GB disk image and CloneZilla iso mounted as a CD, and went out to restore the image to our partition. We didn’t even try to use the CloneZilla UI, except for mounting the host with sshfs to access our backup image. The setup was a breeze, except, that the restore speed was only about 50MB/min. No worries, it’s a small application after all, we don’t need a huge server for it anyway. It was also 1 AM already, so we didn’t pay much attention to detail anymore.

We redirected all the network traffic, to the new virtual interface and tested it while waiting for the restore to complete. During redirect, we fired a command, that killed the network adapter under us, so we were desperately trying to reach the hosting provider, to reboot our server. We went through all numbers listed on their homepage. Good thing they have a webcam in their console room, so we could see the admin passing through on the way to our server, obviously not in a good mood! Suffice to say that was the most fun we had all evening!

After the restore was complete and some minor fixes (make the image bootable, install the grub loader, change the root device in the loader AND the fstab as well, change network configuration, to match the setup, AND the /etc/hosts AND /etc/resolv.conf, that are easily overlooked at 3 in the morning) we had the image booting up already. And oh boy, was it slow? Painfully so! The machine doesn’t support HW virtualization, and SW virtualization just doesn’t cut it.

So we grit our teeth and moved the whole bunch of stuff back to where it was, rebuilt the RAID array, and redirected everything where it was originally. Also we were not happy at all. We went home at 4 AM, when the streets are empty, and the bars were about to close.

I could not believe, that a P4 running at 3GHz should be so slow. So today I took the backup from yesterday, installed VirtualBox, and restored the image to a new virtual machine. It took about 2 hours altogether. I’m not saying that VirtualBox is in any way superior to kvm, but I can show that in our case it’s ten times as fast. I’m sure we made mistakes in the deployment and there might be ways to reach this speed with kvm as well.

No matter how much trouble we went through, it was sure an interesting night, we learned a lot about virtualization and image cloning, that might soon become handy.

Being a  bit on the paranoid side, I have a few tools up and running, that monitor my filesystems, for changes that were not made by me.

One such tool is tiger which gave me lines like:

 # Checking listening processes
NEW: --WARN-- [lin003w] The process `bash' is listening on socket 7171 (TCP on every interface) is run by angel.
NEW: --WARN-- [lin003w] The process `oidentd' is listening on socket 113 (TCP on every interface) is run by oident.

# Verifying system specific password checks...
NEW: --FAIL-- [lin005f] Installed file `/usr/bin/pstree' checksum differs from installed package 'psmisc'.
NEW: --WARN-- [lin001w] File `/lib/libproc.a' does not belong to any package.
NEW: --WARN-- [lin001w] File `/lib/libproc.so.2.0.6' does not belong to any package.
NEW: --WARN-- [lin001w] File `/lib/lidps1.so' does not belong to any package.
NEW: --WARN-- [lin001w] File `/sbin/ttyload' does not belong to any package.
NEW: --WARN-- [lin001w] File `/sbin/ttymon' does not belong to any package.
NEW: --WARN-- [lin001w] File `/usr/sbin/ttyload' does not belong to any package.

# Checking listening processes
NEW: --WARN-- [lin003w] The process `bd' is listening on socket 60001 (TCP on every interface) is run by sdb.

This I just ignored, being absent minded at times, I tend to forget when I install new binaries. Altough I clearly remembered, that I haven’t touched the server for quite a time.

But there were other reports as well, from RKHunter that some files were changed on the file system and a new user was added. It turned out, that a day later the guy activated himself, and installed SH4 and SH5 rootkits on the system.

RKHunter and Tiger reported the changes. A few commands were changed, to hide some processes and files.  The rootkit is a bit screwed up, as the top command doesn’t work at all.

Since I also keep regular backups of the system I wanted to quickly remove the backdoors, I tried to overwrite the changed files on the system. The changed commands had a few  attributes set, so they were read only. To overwrite them with the original files first I had to chattr them.

Unfortunately these rootkits tend to restore themselves from  the demons they spawn, which I couldn’t find until the files were restored. Since I had all the information from the security programs it was quite easy to write a script to erase the rootkits and restore the original files.

SRC="/path-to-archive"

for i in /bin/ls /bin/netstat /bin/ps /usr/bin/find /usr/bin/md5sum /usr/bin/pstree /usr/bin/top /sbin/ifconfig; do
    echo $i
    chattr -sia $i
    rm $i
    cp ${SRC}/$i $i
done

chattr -sia /lib/lidps1.so /etc/sh.conf /dev/srd0  /dev/shm/bd /dev/shm/nou
rm /lib/lidps1.so /etc/sh.conf /dev/srd0  /dev/shm/bd /dev/shm/nou

chattr -sia -R /lib/libsh.so /usr/lib/libsh /dev/.raw
rm -r /lib/libsh.so /usr/lib/libsh /dev/.raw

rm /sbin/ttyload /sbin/ttymon /usr/sbin/ttyload

When this was ran, I was able to find and kill the processes from the backdoors. There were strange processes named bash, ttymon, /usr/sbin/httpd, /usr/sbin/apache/log and suchlike, I didn’t recognize.  After killing the processes, I’ve re ran the script, to make sure there were no application hooks that  recreated the files. Then I went through the entire proc table again with a fine comb.

Post-mortem

It turned out, that the hacker came in through the mldonkey server, that I used to download some torrents. It was ran in a chrooted environment to start with, but this didn’t seem to matter. So much for chroot. The attacker first used a kernel exploit to gain root priviledges. Then he created an account named “angel”and uploaded and installed the rootkits. He also uploaded what seemed to me like an IRC bot, and spread it liberally around the machine. (I found about 5 copies, but I didn’t count)

The kernel vulnerability  existed, because even though the new patched kernel was already installed, the system was not restarted to use it. After the reboot the kernel exploit is hopefully gone, but the mldonkey server remains vulnerable. Mldonkey is disabled on the server for now.

Tisztelt HR Manager!

Láttam állásajánlatukat, elolvastam a reklámszövegüket weblapon.
6 éve J2SE alkalmazásokat fejlesztek és egy kicsit unom, ezért a tavasszal J2EE be is belekóstoltam, pont a Websphere-el.

CV-t nem küldenék egyelőre, mivel önöknek is van elég CV -jük mástól én meg küldtem eleget másova.

Arra lennék kíváncsi, hogy egy ledolgozott hónap után mennyit tudnék én hazavinni, ha az önök cégénél dolgoznék. kezdetben mennyi lenne és 1 év után mennyi lenne? Mennyi időközönként igazítják a fizetést a nyújtott teljesítményhez?

Tisztelettel:
MXXXé SXXXXXXs

Dear HR Manager!

I’ve seen your job ad and read your banner on the website.
I’m developing J2SE applications for over 6 years and a bit bored with it, so I’ve gave J2EE a try, with Websphere.

I would not send my CV for now, as I’m sure you have enough CVs from other people and I’ve sent enough of mine to others.

I would be interested in how much I could take home after a month of work if I worked at your company. how much in the begining and how much after a year. What periodicity do you use to match the wage to the actual performance?

Best regards:
MXXXé SXXXXXXs

INSERT INTO B (ID, COL) VALUES (SELECT C.nextval, COL FROM A)

Nice and easy. This seems that Oracle only evaluates a the sequence operations in a statement once, thus when you take the next value from a sequence you will always get the same value within the statement. This will either result in a constraint violation (if you have one defined) or table A’s ID column filled with the same number.

So solutions like:

INSERT INTO B (ID, COL) VALUES (SELECT (SELECT C.nextval FROM DUAL), COL FROM A)

would not work either, as the expression is still evaluated once.

I see two solutions to this really simple problem, either create a trigger on table B for this excercise, which I consider an overkill, or use a LOOP. Oracle states on it’s documentation, that the loop content on every iteration.