Category Archives: KVM

LAMP with https causes issues with slow requests


So I ran into a really interesting issue the other day.

I was developing an app in PHP and was preparing it for https. In this project I use sessions and cookies with the secure flag set and when I enabled https I noticed that all requests to the server took a very long time. One request took about 8-10 seconds and I could not figure out why.

I was using these versions on CentOS 7 running on the main machine.

This project is a little special because I am running the mysql-server on Remnux in a virtual machine in KVM on the main machine. Remnux is running this version of mysql-server:

I doubt you would have the issue if the mysql server ran on the same machine, I.e localhost. On this scenario the machine used and the Remnux instance where the database is running is using

So, as you may understand I am using as the database in my php code.

I started to debug with strace by attaching to the httpd process:

This will print a lot of nice stuff when you suspect something is really strange. For instance, below you see the web server reading a file called dbfunctions.php.

This is how a connection to the mysql server looks like. You can see the connect() and setsocketopt() functions being called to establish a connection. It was also here I noticed a delay.

It can be useful to troubleshoot delays.

I also found a thread on stack overflow which suggested that the cause was DNS recursive requests. So I fired up tcpdump with this command

And I saw that the machine was trying to do reverse lookups of 192.168.x.x which took forever (Going out to google)

I had a discussion with a friend and after trying nginx I realised that the problem was not apache or nginx. He suggested I try this setting in the mysql-server on Remnux.

I edited the file /etc/mysql/my.cnf and added the setting “skip-name-resolve”. Here is a snippet from my configuration file on Remnux:

After a restart of everything the issue was gone.

I hope I can help someone else by making this documented and the steps to troubleshoot public. This was really a very strange issue and took some time to troubleshoot.

Good luck!

Compiling QEMU and libvirtd from source to be more sneaky when doing malware analysis


This is another post related to malware analysis and QEMU/KVM. I did a bit more research and found different older articles describing how to make pafish happy and how to evade malware that are aware of virtual machines.

Below is a screenshot from the output of pafish on Windows 7.

The Windows 7 system is running on a KVM host with the following kernel:

As you can see, it looks pretty good and we may be able to win over most malware, but there are still stuff that we want to remove. For instance, check out the device manager

As you can see there are devices called QEMU which indicates that this is not a laptop. Note that pafish did not detect this, but still, we should fix it. Our goal today is to make it say something else. Before we continue there are a lot of posts that I have used for reference (Check the end of this guide), but I wanted to start fresh and make a guide for everyone trying to do this on.. You guessed it, CentOS 7.

Before we start let me just explain a little. When you install qemu/kvm on CentOS 7 using yum it will be called qemu-kvm, but when you compile it will be called qemu-system-x86_64. This is important to understand. It is still the same, but it is called different depending on if it is compiled or not. Read more here
Also, make sure that you have the kvm module loaded. My laptop for this guide is an old laptop running a AMD CPU. Check with lsmod if the proper modules are loaded. For intel it should say kvm_intel.

1. Install a fresh CentOS 7 minimal.
I installed it with Gnome Desktop as I am using an old laptop. And update it:

After we finish it will look something like this (2015-06-25)

2. Install some more packages and development tools
Grab a coffee while you wait.

Before we continue we can do some tweaking to compile a little bit faster. Depending on how many cores your system has, you can change the jobs parameter for make. Execute the following command:

This would give me the value of “make -j2”. Yea, this is an old laptop..

3. Cloning QEMU

4. Edit drivers and compiling
Now the fun stuff starts. We are going to rename the device from QEMU HARDDISK to something else. Make sure you are in the cloned qemu folder that we just cloned from the git repo (step 4)

Find the driver name (As seen in the device manager)

These are the files that you need to edit. I will replace it with “WDC WD20EARS” that should simulate a 2TB disk from Western Digital (according to a Google search).

Also, let us not forget the DVD drive. Let us call it Toshiba DVD-ROM (because it was the only thing that popped up in my head)

And there are some more places that we need to edit:

Time to build! Make sure you are root so you can install it.

Fix a sympbolic link to make virt-manager happy.

5. Time to compile libvirt

If you want to know more about compiling libvirt, and the arguments I am using with autogen you can read more here before you continue:

Go ahead and start!

You should see something like this in /var/log/messages

6. Install virt-manager

You can now start virt-manager by executing ./virt-manager in the same directory. Or, if you do not want to use virt-manager and create the virtual machine, you can just continue to step 7.

7. Install the first virtual machine
Let us install a Windows XP SP0 and see whats what. I am using an old iso image which I placed in /root/

When the installation is complete, go into the device manager and voilá:

Device Manager after QEMO modifications

Each time you want to modify something in the qemu code you can just rerun the make and make install steps and you will update the binary and drivers will be changed accordingly.

8. Changing The BIOS
This is the last step if you really would like to be sneaky. When you execute dmidecode it will also tell you that this is QEMU and not something else.

Notice QEMU after Manufacturer on the devices

QEMU uses seabios so we will start by performing a clone of the latest source and compile it.

The completed bios file will be located in the folder out and is called “bios.bin”. The next step is to execute it with some parameters set (“-k en” means english keyboard)

Notice the parameter

This is where I specify the bios file we compiled.

The other parameters:

Is where I tell it to set Dell as a manufacturer and not QEMU. And your final product after running dmidecode should be something like this:

dmidecode after the parameter changes

You can also add the bios file using virsh edit and set the following.

However, you still require the smbios arguments. You could also add the “-smbios type” parameters in the domain xml for the vm.

That was all!

References and links

Using KVM and guestfish for dynamic malware analysis on Windows


After the news that FireEye was affected by the VENOM vulnerability it got me thinking that how hard can it be to do this in KVM. This article will give you a start on how to do sort of dynamic analysis, or at least get you going. I will probably update it later on when I do more.


This guide requires that you have already installed a CentOS 7.1 system with KVM support and a Windows XP machine installed. This machine is named “winxp” and will be used further down in the guide. The system has to be configured with IDE drives and qcow2 format. Make sure you have two disks, one disk called “windows_xp.qcow2” mounted at C: and one called “windows_xp_data.qcow2” mounted at E: on the machine. The size of the disks should be at least 80GB or so per disk. Other things like memory should be fine at 2GB.

Let’s kick off! Make sure you have a snapshot on the vm called “ready”. We will use this as default state before starting the machine. So to sum up.

1. Install a VM Windows XP guest with 2gb of ram and qcow2 disks so that we can use snapshots
2. Make sure it has two disks, one at C: at 80GB and one at E: at 80GB. These qcow2 images are saved in the folder /vm/
3. Create a snapshot called “ready” when you have installed it.
4. Done!

Let’s start!

Everything we configure from now on will be done on the KVM host. That is the mother machine running the guest machines (the Windows XP system is a guest)

1. Install the epel repo. You will need this for the packages.

2. Install some packages from epel. They are required to mount the NTFS volume offline

3. The is an issue on current (2015-06-08) CentOS 7.1 with winsupport. You have to manually download the rpm and install it using yum localinstall. This is used to be able to mount NTFS drives.
Read more here:
Download here

4. Create the folder /root/files/

5. That should be it. Test it with the script in the next step.

Before you run it you have to know what it does. As you cannot insert files on a live system without (potentially) damaging the filesystem we will make sure the guest is shut down first. We will use destroy since we do not want to wait for a graceful shutdown. After the machine is shut down, we mount the disk called “windows_xp_data.qcow2” which is our E: on the winxp system and then upload all files in the folder “/root/files” to E:\files\ on the Windows XP system.

6. The startup script on the vm host.

7. When the machine now has booted you should find the files on E:\files. What you can do now is to implement a python application running at startup on the Windows XP machine that looks in the folder and then uses something like a custom cfg file to perhaps first install a older version of adobe and then launch your pdf file in files. As you can manipulate the filesystem before you boot the machine you may be able remove some or all traces of QEMU or KVM to perhaps “dodge” the vm awareness in some malware.

Good luck!

Converting ova images to qcow2

The ova format is an open format for virtual machines. In this small guide I will explain how you can convert it to qcow2 format and run it in KVM.

In my example I am using LogPoint which is a SIEM product that you can download a trial of here

The file you download will be called logpoint.ova. Once you have it you can check out the file format using the file command:

You can also have a look at the contents of the tar archive

The next step is to extract the files. Just run

Time for some conversion!

The next step is to logon with ssh to your KVM server and start virt-manager and import the image.

In virt-manager choose new and then import existing image. If you get a black screen on the console try to change the video to QXL and restart it. As LogPoint is based on Ubuntu you can change that while in the guide.

Good luck!