Category Archives: CentOS

LAMP with https causes issues with slow requests


So I ran into a really interesting issue the other day.

I was developing an app in PHP and was preparing it for https. In this project I use sessions and cookies with the secure flag set and when I enabled https I noticed that all requests to the server took a very long time. One request took about 8-10 seconds and I could not figure out why.

I was using these versions on CentOS 7 running on the main machine.

This project is a little special because I am running the mysql-server on Remnux in a virtual machine in KVM on the main machine. Remnux is running this version of mysql-server:

I doubt you would have the issue if the mysql server ran on the same machine, I.e localhost. On this scenario the machine used and the Remnux instance where the database is running is using

So, as you may understand I am using as the database in my php code.

I started to debug with strace by attaching to the httpd process:

This will print a lot of nice stuff when you suspect something is really strange. For instance, below you see the web server reading a file called dbfunctions.php.

This is how a connection to the mysql server looks like. You can see the connect() and setsocketopt() functions being called to establish a connection. It was also here I noticed a delay.

It can be useful to troubleshoot delays.

I also found a thread on stack overflow which suggested that the cause was DNS recursive requests. So I fired up tcpdump with this command

And I saw that the machine was trying to do reverse lookups of 192.168.x.x which took forever (Going out to google)

I had a discussion with a friend and after trying nginx I realised that the problem was not apache or nginx. He suggested I try this setting in the mysql-server on Remnux.

I edited the file /etc/mysql/my.cnf and added the setting “skip-name-resolve”. Here is a snippet from my configuration file on Remnux:

After a restart of everything the issue was gone.

I hope I can help someone else by making this documented and the steps to troubleshoot public. This was really a very strange issue and took some time to troubleshoot.

Good luck!

Compiling QEMU and libvirtd from source to be more sneaky when doing malware analysis


This is another post related to malware analysis and QEMU/KVM. I did a bit more research and found different older articles describing how to make pafish happy and how to evade malware that are aware of virtual machines.

Below is a screenshot from the output of pafish on Windows 7.

The Windows 7 system is running on a KVM host with the following kernel:

As you can see, it looks pretty good and we may be able to win over most malware, but there are still stuff that we want to remove. For instance, check out the device manager

As you can see there are devices called QEMU which indicates that this is not a laptop. Note that pafish did not detect this, but still, we should fix it. Our goal today is to make it say something else. Before we continue there are a lot of posts that I have used for reference (Check the end of this guide), but I wanted to start fresh and make a guide for everyone trying to do this on.. You guessed it, CentOS 7.

Before we start let me just explain a little. When you install qemu/kvm on CentOS 7 using yum it will be called qemu-kvm, but when you compile it will be called qemu-system-x86_64. This is important to understand. It is still the same, but it is called different depending on if it is compiled or not. Read more here
Also, make sure that you have the kvm module loaded. My laptop for this guide is an old laptop running a AMD CPU. Check with lsmod if the proper modules are loaded. For intel it should say kvm_intel.

1. Install a fresh CentOS 7 minimal.
I installed it with Gnome Desktop as I am using an old laptop. And update it:

After we finish it will look something like this (2015-06-25)

2. Install some more packages and development tools
Grab a coffee while you wait.

Before we continue we can do some tweaking to compile a little bit faster. Depending on how many cores your system has, you can change the jobs parameter for make. Execute the following command:

This would give me the value of “make -j2”. Yea, this is an old laptop..

3. Cloning QEMU

4. Edit drivers and compiling
Now the fun stuff starts. We are going to rename the device from QEMU HARDDISK to something else. Make sure you are in the cloned qemu folder that we just cloned from the git repo (step 4)

Find the driver name (As seen in the device manager)

These are the files that you need to edit. I will replace it with “WDC WD20EARS” that should simulate a 2TB disk from Western Digital (according to a Google search).

Also, let us not forget the DVD drive. Let us call it Toshiba DVD-ROM (because it was the only thing that popped up in my head)

And there are some more places that we need to edit:

Time to build! Make sure you are root so you can install it.

Fix a sympbolic link to make virt-manager happy.

5. Time to compile libvirt

If you want to know more about compiling libvirt, and the arguments I am using with autogen you can read more here before you continue:

Go ahead and start!

You should see something like this in /var/log/messages

6. Install virt-manager

You can now start virt-manager by executing ./virt-manager in the same directory. Or, if you do not want to use virt-manager and create the virtual machine, you can just continue to step 7.

7. Install the first virtual machine
Let us install a Windows XP SP0 and see whats what. I am using an old iso image which I placed in /root/

When the installation is complete, go into the device manager and voilá:

Device Manager after QEMO modifications

Each time you want to modify something in the qemu code you can just rerun the make and make install steps and you will update the binary and drivers will be changed accordingly.

8. Changing The BIOS
This is the last step if you really would like to be sneaky. When you execute dmidecode it will also tell you that this is QEMU and not something else.

Notice QEMU after Manufacturer on the devices

QEMU uses seabios so we will start by performing a clone of the latest source and compile it.

The completed bios file will be located in the folder out and is called “bios.bin”. The next step is to execute it with some parameters set (“-k en” means english keyboard)

Notice the parameter

This is where I specify the bios file we compiled.

The other parameters:

Is where I tell it to set Dell as a manufacturer and not QEMU. And your final product after running dmidecode should be something like this:

dmidecode after the parameter changes

You can also add the bios file using virsh edit and set the following.

However, you still require the smbios arguments. You could also add the “-smbios type” parameters in the domain xml for the vm.

That was all!

References and links

Using KVM and guestfish for dynamic malware analysis on Windows


After the news that FireEye was affected by the VENOM vulnerability it got me thinking that how hard can it be to do this in KVM. This article will give you a start on how to do sort of dynamic analysis, or at least get you going. I will probably update it later on when I do more.


This guide requires that you have already installed a CentOS 7.1 system with KVM support and a Windows XP machine installed. This machine is named “winxp” and will be used further down in the guide. The system has to be configured with IDE drives and qcow2 format. Make sure you have two disks, one disk called “windows_xp.qcow2” mounted at C: and one called “windows_xp_data.qcow2” mounted at E: on the machine. The size of the disks should be at least 80GB or so per disk. Other things like memory should be fine at 2GB.

Let’s kick off! Make sure you have a snapshot on the vm called “ready”. We will use this as default state before starting the machine. So to sum up.

1. Install a VM Windows XP guest with 2gb of ram and qcow2 disks so that we can use snapshots
2. Make sure it has two disks, one at C: at 80GB and one at E: at 80GB. These qcow2 images are saved in the folder /vm/
3. Create a snapshot called “ready” when you have installed it.
4. Done!

Let’s start!

Everything we configure from now on will be done on the KVM host. That is the mother machine running the guest machines (the Windows XP system is a guest)

1. Install the epel repo. You will need this for the packages.

2. Install some packages from epel. They are required to mount the NTFS volume offline

3. The is an issue on current (2015-06-08) CentOS 7.1 with winsupport. You have to manually download the rpm and install it using yum localinstall. This is used to be able to mount NTFS drives.
Read more here:
Download here

4. Create the folder /root/files/

5. That should be it. Test it with the script in the next step.

Before you run it you have to know what it does. As you cannot insert files on a live system without (potentially) damaging the filesystem we will make sure the guest is shut down first. We will use destroy since we do not want to wait for a graceful shutdown. After the machine is shut down, we mount the disk called “windows_xp_data.qcow2” which is our E: on the winxp system and then upload all files in the folder “/root/files” to E:\files\ on the Windows XP system.

6. The startup script on the vm host.

7. When the machine now has booted you should find the files on E:\files. What you can do now is to implement a python application running at startup on the Windows XP machine that looks in the folder and then uses something like a custom cfg file to perhaps first install a older version of adobe and then launch your pdf file in files. As you can manipulate the filesystem before you boot the machine you may be able remove some or all traces of QEMU or KVM to perhaps “dodge” the vm awareness in some malware.

Good luck!

Using winexe on CentOS 7 to execute commands on remote Windows 7 machines

winexe is an application for Linux that you can use to execute remote commands on Windows machines. Similar to how a meterpreter would work or how you use psexec on Windows.
The reason for using it could be that you want to script certain actions on remote machines, such as executing powershell scripts and gather information for forensic purposes etc. I was interested in it because of the last example.

In this guide I will show you how to do that and build a scenario for you and show you how one could use it to collect remote information on your endpoints. As usual I use CentOS 7 with the latest updates as per the date of this post.

Download winexe from here. If you download the current source (as per this date) it is called “” with sha1sum cbbebd80c935a8408448f3a92c04b85ea19a8b64

Or, clone it using git:

Make sure you have all the packages installed. You will need the epel repo for some packages as well.

3. Unzip the package downloaded in step 1 (I downloaded it to /root/) and navigate into the source folder. Or if you used git simple go into the folder and then in the source folder.

4. Build it

5. If everything works good you will get the output

Time for the fun stuff. Now you will have your winexe binary in the build directory. Navigate into the folder build

Now. Before we continue you need to set up the Windows machine. There are some issues with Windows 7 and later as described here. I had to add this key to my machine to make it work:

Start cmd as administrator and paste it in. That should be everything.

I have already uploaded a powershell script to my machine. It will run some commands and output this to a text file locally on the system. The local user I have created is called “bob” with the password of “secretpassword”. The system I am targeting is a system called windows.domain.local. Note that it is not joined to a active directory domain, it is just a domain I use for testing. You could do it with kerberos and an active directory account as well. The command line for executing the powershell script is:

The –uninstall parameter removes the service that winexe installs after it is done.
The other parameters you can read about here

To just run a simple ipconfig you can do like this

Or maybe drop yourself into a shell:


Guide for setting up yubikey OTP ssh access with CentOS 7


Today I will show you how to use the yubikey and set up authentication on CentOS 7 from scratch.
I have bought a yubikey 2 standard and will configure it to be used as OTP device when logging on via SSH. This is very cheap solution which offers a high level of security for home users.

The system I am using has just been installed and it is standard CentOS 7 with just default selections when installing. System is also updated and current to this day.

Let’s start!

Begin by installing some packages

Clone the yubico-c library and install it.

If you get the error messsage

You need to install asciidoc

Clone the yubikey-personalization library and install it

Clone the yubico-c-client and install it

Clone the yubico-pam module and install it

Fix the symlink for the pam module

NOTE! Before you continue, this is where you can lock yourself out and create a box that is not accessible. You should be logged on with root on another terminal so that you can revert back if it does not work. Console access will still work, so if you use KVM or something similar, you should be able to access it through that.

Start by making sure your yubikey is configured as shown in this guide:
YubiKey YubiCloud Configuration

You should also make sure that the OTP function is working by going here after you have configured it:

Your server will need to be able to contact the yubikey API service to validate the auth, so make sure you have internet connection to the box as well.
When it is working, move on!

Change SELinux to run in permissive mode. People have been reporting issues with SELinux and yubikey so I recommend that you put it in permissive mode. If you don’t want to spend a couple of hours troubleshooting.

Edit the file /etc/sysconfig/selinux and set the parameter SELINUX to permissive as shown below.

This will run at boot next time and change it to permissive. Now run the command to change mode to permissive in this session

Get the status. You can see the row “Current mode”, should say “permissive”

Create a user, I have picked the name ‘test’. Make sure you change the password as well.

Edit the file /etc/pam.d/sshd

Make sure it contains the row:

as seen below in the file:

Crete the authfile mentioned above.

Add your user and yubikey id separated with ‘:’. This is how my /etc/yubipasswd file looks like:

Open a new terminal and login as test

ssh test@your-yubikey-configured-system

When you are promted for the password, write your password and press your yubikey so that it becomes one long string. Imagine:

[root@localhost test]# ssh test@localhost
test@localhost’s password: password + yubikey OTP token

The yubikey will press enter by default and this should log you on! Voila!

Other information
If you have multiple configurations on your yubikey, i.e Configuration Slot 1 and Slot 2 you may have used the wrong Slot. Slot 1 is pressed for 1 second and the other is above 2 seconds. (Or something like that). I use Configuration Slot 1 because then you dont have to press it for a longer period of time.

If you have installed a default CentOS 7 you probably do not need to change other SSH configuration files such as /etc/ssh/sshd_config

And /etc/pam.d/system-auth by adding “try_first_pass” to the row

You can also try installing the packages via epel repo. Packages are called:

The pam library I think you will have to install manually as previously shown.

Links related to this guide

Good luck!

Putting together a small systemd service script and controlling python


This time I will demonstrate how to create a simple systemd service file so that you can control your applications with systemctl in CentOS 7. As you know systemctl will be replacing init so it is time to see what you can do with it.

Imagine the following sample script called It will not do anything but print hello world and then sleep for one second.

Make sure you set the permissions so that it is executable. I will be placing this script in the folder /usr/local/bin/hello/

The next step is to create a small bash script that will execute the python script. In this script you can do more stuff if needed. I will call it

Make sure it is also located in /usr/local/bin/hello/

Now you want to be able to control that running a service command, i.e “service hello start” or
“systemctl start hello.service”

Start by navigating to the systemd folder and create the hello.service file. This is where we will tell systemctl what will happen.

Edit the hello.service file and make sure you point ExecStart to the bash script called

Now, there are a few things you need to know. The parameter “ExecStart” is what going to happen when you tell systemctl to “start”. There are also “ExecStop” and “ExecReload”. Those will be used for? You guessed it, when stopped and reloaded. There are also nice stuff such as “ExecStartPre” and “ExecStartPost” to execute commands before and after the main app has started.

More info can be found at Red Hat

However, in our service file above we only need the settings I have provided.
So now it is time to execute it:

As you see I am using the old service command but CentOS 7 is redirecting to systemctl, that is fine. Both will work. The status command also shows that it is running.

We can also confirm by checking what is running

If you change any of the contents of the hello.service file you will be given the following message:

That is fine, just execute “systemctl daemon-reload”

To enable this on startup you can just do the classic chkconfig hello on and systemctl will take over:

If you have a python application that may involve mongodb or some other service like mysql you can edit your service file to look like this

This will mean that syslog, network and the mongodb service is required to start before your app is.

Have fun!

Converting ova images to qcow2

The ova format is an open format for virtual machines. In this small guide I will explain how you can convert it to qcow2 format and run it in KVM.

In my example I am using LogPoint which is a SIEM product that you can download a trial of here

The file you download will be called logpoint.ova. Once you have it you can check out the file format using the file command:

You can also have a look at the contents of the tar archive

The next step is to extract the files. Just run

Time for some conversion!

The next step is to logon with ssh to your KVM server and start virt-manager and import the image.

In virt-manager choose new and then import existing image. If you get a black screen on the console try to change the video to QXL and restart it. As LogPoint is based on Ubuntu you can change that while in the guide.

Good luck!

Telldus Tellstick Duo on CentOS 7

The Telldus Tellstick Duo is a tiny usb device to control rf devices such as lightbulbs, thermo meters and a lot of other stuff. The software to control the device is open source and works fine on Ubuntu/Debian, OS X and Windows. People have also documented how to get it to work with CentOS 6. But if you are like me and want to run the latest CentOS version then it is a little bit more complicated. In this guide I will show you how to compile and get it to run on the latest version of CentOS 7.

1. Begin by plugging in your tellstick duo in your machine, if you tail /var/log/messages you will see the following when you plug it in:

2. The next step is to install some repos and libraries so that we can compile the latest version of the software from telldus. Note, if this guide fails you can try to do “yum groupinstall Development Tools” instead.

3. Install the epel repo

4. Install all of the libftdi, libusb and libconfuse libraries

5. Download and unpack the telldus software. At the time of writing the latest package is 2.1.2

NOTE ! depending on your version of Tellstick Duo, version 2.1.2 of the software may not work. If you run in to issues such as error messages saying “TellStick not found” when you run tdtool. try 2.1.1 instead.

If you then run into issues when compiling 2.1.1 with make, you may need to edit the file “telldus-core-2.1.1/common/Socket_unix.cpp”. Make sure you include unistd.h so that it looks like this in the top of the file:


6. Now when you are supposed to run “cmake .” it will fail:

This is because of two things. It cannot find the and the FTDI_LIBRARY is not found, this can be fixed. First download the

Run the following cmake command to point to the library:

This is how it looks like:

7. Almost done. Time to make it. Run this command:

All should be good and you can move on to installing it.

8. Install it:

9. Fix some symlinks and run ldconfig:

10. Done! Start the daemon with this command

Note. You have to restart the daemon each time you change the tellstick.conf file. You can use kill or whatever you like to do this.

11. Configure your devices in the file /etc/tellstick.conf . This is what it looks like by default:

12. Once you have set up your devices according to the documentation you can control them using “tdtool”

Other notes

For me as I have an old version of the Tellstick Duo I have to use version 2.1.1 of the software. I get error messages such as:

You can fix this by doing what is documented here:

exFAT on CentOS 7

I recently decided to swap out my Apple Mac Mini as a media center (I use Plex) and run Plex media server on Linux instead. Plex has come a long way and now works fine on CentOS 6 or later.

However, some of my media was on a exFAT ( formatted USB drive and I wanted that to work on CentOS 7. A google search found a good solution and I wanted to share that with you in this small guide.

1. Start with adding some repos that you will need. As this guide is for CentOS 7 I have chosen the repos that fit that.

2. Download the latest epel release from this location (at the time of writing it is epel-release-7-5.noarch.rpm)

3. Download and install

3. Download the latest nux repo from this location (at the time of writing it is nux-dextop-release-0-1.el7.nux.noarch.rpm):

4. Download and install:

5. Install the exFAT packages

6. Plugin your disk and get the partition that you want to mount. For instance:

Produces this output:

7. Mount it:

8. Done! it will now be mounted at /media/

Setup mail client on CentOS 7


A lot of people often need to be able to send e-mail from a server at home. For instance, you may be running a Raspberry Pi that is monitoring something or doing something fun such as filming or taking pictures of your neighbor. I wrote this small guide as it is probably the easiest way of doing it for CentOS 7.

1. To start with, you probably have an ISP with an e-mail server, such as This will be needed later in step 4 of the guide.

2. Make sure postfix is installed. Or else, yum install postfix. At the time of writing this is what I am running.

[root@serv1 ~]# yum list installed|grep postfix
postfix.x86_64 2:2.10.1-6.el7 @anaconda

3. Backup your /etc/postfix/ file to /etc/postfix/

4. Edit the file /etc/postfix/ and change the following settings:

myorigin =
mydestination =, localhost
relayhost =

Where can be something like or whatever the name is of your own domain.

5. Restart postfix, service postfix restart

6. Try it by doing something like this, if you want to monitor your RAID

/usr/sbin/mdadm --detail /dev/dm-1 > /root/RAID_STATUS.log
/bin/mail -s "RAID STATUS" "" < /root/RAID_STATUS.log