LAMP with https causes issues with slow requests

Hey,

So I ran into a really interesting issue the other day.

I was developing an app in PHP and was preparing it for https. In this project I use sessions and cookies with the secure flag set and when I enabled https I noticed that all requests to the server took a very long time. One request took about 8-10 seconds and I could not figure out why.

I was using these versions on CentOS 7 running on the main machine.

This project is a little special because I am running the mysql-server on Remnux in a virtual machine in KVM on the main machine. Remnux is running this version of mysql-server:

I doubt you would have the issue if the mysql server ran on the same machine, I.e localhost. On this scenario the machine used 10.1.1.2 and the Remnux instance where the database is running is using 192.168.10.250.

So, as you may understand I am using 192.168.10.250 as the database in my php code.

I started to debug with strace by attaching to the httpd process:

This will print a lot of nice stuff when you suspect something is really strange. For instance, below you see the web server reading a file called dbfunctions.php.

This is how a connection to the mysql server looks like. You can see the connect() and setsocketopt() functions being called to establish a connection. It was also here I noticed a delay.

It can be useful to troubleshoot delays.

I also found a thread on stack overflow which suggested that the cause was DNS recursive requests. So I fired up tcpdump with this command

And I saw that the machine was trying to do reverse lookups of 192.168.x.x which took forever (Going out to google)

I had a discussion with a friend and after trying nginx I realised that the problem was not apache or nginx. He suggested I try this setting in the mysql-server on Remnux.

http://dev.mysql.com/doc/refman/5.7/en/server-options.html#option_mysqld_skip-name-resolve

I edited the file /etc/mysql/my.cnf and added the setting “skip-name-resolve”. Here is a snippet from my configuration file on Remnux:

After a restart of everything the issue was gone.

I hope I can help someone else by making this documented and the steps to troubleshoot public. This was really a very strange issue and took some time to troubleshoot.

Good luck!

Various code snippets in C++ (Mostly WINAPI)

Hey,

I thought I would share some code snippets used in various projects of mine. The code I will be sharing can be compiled with Visual Studio 2015 or later (probably).

The first one is a piece of code to get any of the KNOWNFOLDERID‘s in Windows.

If you are using shellcode from meterpreter and you are going to bundle this into another executable, you can execute the shellcode by using this code.

The last snippet is code that can be used to convert a string in hex to the char values.

Good luck!

Raspberry Pi as tor2web service running apache

Hey,

In this article I will describe how to set up a Raspberry Pi, install tor and access it using tor2web.

Start by installing RASPBIAN JESSIE LITE (Minimal image based on Debian Jessie) from the raspberry pi web . I will be using this version:

Version:November 2015
Release date:2015-11-21
Kernel version:4.1

I will walk you through everything until you have a raspberry pi online with a hidden tor2web url that you can access from anywhere.

1 – Change root password
The default password is “raspberry” and user is “pi”. Make sure you change this.

2 – Updating the Pi
Once you have connected the Pi to the network make sure you update it.

apt-get update -y; apt-get upgrade -y

3 – Installing apache2

I will refer to Raspberry Pi documentation for how to install apache.
We will also configure it to only listen on localhost, as we do not need it to listen on the public interface, we will only have ssh there. This is also they way tor hidden services work, but more on that later!

This is optional. Install php, mysql and php-pdo

4 – Configure apache
We only want it to listen on localhost because we want to use it on tor. So let’s change that. Modify the file /etc/apache2/ports.conf so that it looks like this. Make sure the line is changed from “Listen 80” to “Listen 127.0.0.1:8080”.

5 – Installing tor

6 – Configure tor
We will now create the directories and configure the hidden service. If you want to read more check the reference documentation here

Edit the file /etc/tor/torrc

Time to start, “service tor start”

Your hostname will now be visible in this file:

Your website will now be available by accessing a url such as

https://xxxxxxxxxxxxxxx.onion.to/

Replace the x’s with your hostname found in the hostname file. This works because of the “.to” that is added to the onion domain. More information here https://tor2web.org/

To make tor start on boot, enter the following command

References and additional reading

  • http://httpd.apache.org/docs/2.4/ssl/ssl_howto.html
  • https://www.digitalocean.com/community/tutorials/how-to-create-a-ssl-certificate-on-apache-for-ubuntu-14-04
  • https://www.torproject.org/docs/tor-hidden-service.html.en
  • https://www.torproject.org/docs/debian.html.en
  • https://tor2web.org/
  • Compiling QEMU and libvirtd from source to be more sneaky when doing malware analysis

    Hello,

    This is another post related to malware analysis and QEMU/KVM. I did a bit more research and found different older articles describing how to make pafish happy and how to evade malware that are aware of virtual machines.

    Below is a screenshot from the output of pafish on Windows 7.

    The Windows 7 system is running on a KVM host with the following kernel:

    As you can see, it looks pretty good and we may be able to win over most malware, but there are still stuff that we want to remove. For instance, check out the device manager

    As you can see there are devices called QEMU which indicates that this is not a laptop. Note that pafish did not detect this, but still, we should fix it. Our goal today is to make it say something else. Before we continue there are a lot of posts that I have used for reference (Check the end of this guide), but I wanted to start fresh and make a guide for everyone trying to do this on.. You guessed it, CentOS 7.

    Before we start let me just explain a little. When you install qemu/kvm on CentOS 7 using yum it will be called qemu-kvm, but when you compile it will be called qemu-system-x86_64. This is important to understand. It is still the same, but it is called different depending on if it is compiled or not. Read more here
    Also, make sure that you have the kvm module loaded. My laptop for this guide is an old laptop running a AMD CPU. Check with lsmod if the proper modules are loaded. For intel it should say kvm_intel.

    1. Install a fresh CentOS 7 minimal.
    I installed it with Gnome Desktop as I am using an old laptop. And update it:

    After we finish it will look something like this (2015-06-25)

    2. Install some more packages and development tools
    Grab a coffee while you wait.

    Before we continue we can do some tweaking to compile a little bit faster. Depending on how many cores your system has, you can change the jobs parameter for make. Execute the following command:

    This would give me the value of “make -j2”. Yea, this is an old laptop..

    3. Cloning QEMU

    4. Edit drivers and compiling
    Now the fun stuff starts. We are going to rename the device from QEMU HARDDISK to something else. Make sure you are in the cloned qemu folder that we just cloned from the git repo (step 4)

    Find the driver name (As seen in the device manager)

    These are the files that you need to edit. I will replace it with “WDC WD20EARS” that should simulate a 2TB disk from Western Digital (according to a Google search).

    Also, let us not forget the DVD drive. Let us call it Toshiba DVD-ROM (because it was the only thing that popped up in my head)

    And there are some more places that we need to edit:

    Time to build! Make sure you are root so you can install it.

    Fix a sympbolic link to make virt-manager happy.

    5. Time to compile libvirt

    If you want to know more about compiling libvirt, and the arguments I am using with autogen you can read more here before you continue:
    https://libvirt.org/compiling.html

    Go ahead and start!

    You should see something like this in /var/log/messages

    6. Install virt-manager

    You can now start virt-manager by executing ./virt-manager in the same directory. Or, if you do not want to use virt-manager and create the virtual machine, you can just continue to step 7.

    7. Install the first virtual machine
    Let us install a Windows XP SP0 and see whats what. I am using an old iso image which I placed in /root/

    When the installation is complete, go into the device manager and voilá:

    Device Manager after QEMO modifications

    Each time you want to modify something in the qemu code you can just rerun the make and make install steps and you will update the binary and drivers will be changed accordingly.

    8. Changing The BIOS
    This is the last step if you really would like to be sneaky. When you execute dmidecode it will also tell you that this is QEMU and not something else.

    Notice QEMU after Manufacturer on the devices

    QEMU uses seabios so we will start by performing a clone of the latest source and compile it.

    The completed bios file will be located in the folder out and is called “bios.bin”. The next step is to execute it with some parameters set (“-k en” means english keyboard)

    Notice the parameter

    This is where I specify the bios file we compiled.

    The other parameters:

    Is where I tell it to set Dell as a manufacturer and not QEMU. And your final product after running dmidecode should be something like this:

    dmidecode after the parameter changes

    You can also add the bios file using virsh edit and set the following.

    However, you still require the smbios arguments. You could also add the “-smbios type” parameters in the domain xml for the vm.

    That was all!

    References and links

    Using KVM and guestfish for dynamic malware analysis on Windows

    Hey,

    After the news that FireEye was affected by the VENOM vulnerability it got me thinking that how hard can it be to do this in KVM. This article will give you a start on how to do sort of dynamic analysis, or at least get you going. I will probably update it later on when I do more.

    Prerequisites

    This guide requires that you have already installed a CentOS 7.1 system with KVM support and a Windows XP machine installed. This machine is named “winxp” and will be used further down in the guide. The system has to be configured with IDE drives and qcow2 format. Make sure you have two disks, one disk called “windows_xp.qcow2” mounted at C: and one called “windows_xp_data.qcow2” mounted at E: on the machine. The size of the disks should be at least 80GB or so per disk. Other things like memory should be fine at 2GB.

    Let’s kick off! Make sure you have a snapshot on the vm called “ready”. We will use this as default state before starting the machine. So to sum up.

    1. Install a VM Windows XP guest with 2gb of ram and qcow2 disks so that we can use snapshots
    2. Make sure it has two disks, one at C: at 80GB and one at E: at 80GB. These qcow2 images are saved in the folder /vm/
    3. Create a snapshot called “ready” when you have installed it.
    4. Done!

    Let’s start!

    Everything we configure from now on will be done on the KVM host. That is the mother machine running the guest machines (the Windows XP system is a guest)

    1. Install the epel repo. You will need this for the packages.

    2. Install some packages from epel. They are required to mount the NTFS volume offline

    3. The is an issue on current (2015-06-08) CentOS 7.1 with winsupport. You have to manually download the rpm and install it using yum localinstall. This is used to be able to mount NTFS drives.
    Read more here:
    https://www.centos.org/forums/viewtopic.php?f=48&t=52437
    Download here
    http://people.redhat.com/~rjones/libguestfs-winsupport/7/7.1/x86_64/

    4. Create the folder /root/files/

    5. That should be it. Test it with the script in the next step.

    Before you run it you have to know what it does. As you cannot insert files on a live system without (potentially) damaging the filesystem we will make sure the guest is shut down first. We will use destroy since we do not want to wait for a graceful shutdown. After the machine is shut down, we mount the disk called “windows_xp_data.qcow2” which is our E: on the winxp system and then upload all files in the folder “/root/files” to E:\files\ on the Windows XP system.

    6. The startup script on the vm host.

    7. When the machine now has booted you should find the files on E:\files. What you can do now is to implement a python application running at startup on the Windows XP machine that looks in the folder and then uses something like a custom cfg file to perhaps first install a older version of adobe and then launch your pdf file in files. As you can manipulate the filesystem before you boot the machine you may be able remove some or all traces of QEMU or KVM to perhaps “dodge” the vm awareness in some malware.

    Good luck!

    Using winexe on CentOS 7 to execute commands on remote Windows 7 machines

    winexe is an application for Linux that you can use to execute remote commands on Windows machines. Similar to how a meterpreter would work or how you use psexec on Windows.
    The reason for using it could be that you want to script certain actions on remote machines, such as executing powershell scripts and gather information for forensic purposes etc. I was interested in it because of the last example.

    In this guide I will show you how to do that and build a scenario for you and show you how one could use it to collect remote information on your endpoints. As usual I use CentOS 7 with the latest updates as per the date of this post.

    1.
    Download winexe from here. If you download the current source (as per this date) it is called “winexe-winexe-waf-b787d2a2c4b1abc3653bad10aec943b8efcd7aab.zip” with sha1sum cbbebd80c935a8408448f3a92c04b85ea19a8b64

    Or, clone it using git:

    2.
    Make sure you have all the packages installed. You will need the epel repo for some packages as well.

    3. Unzip the package downloaded in step 1 (I downloaded it to /root/) and navigate into the source folder. Or if you used git simple go into the folder and then in the source folder.

    4. Build it

    5. If everything works good you will get the output

    6.
    Time for the fun stuff. Now you will have your winexe binary in the build directory. Navigate into the folder build

    7.
    Now. Before we continue you need to set up the Windows machine. There are some issues with Windows 7 and later as described here. I had to add this key to my machine to make it work:

    Start cmd as administrator and paste it in. That should be everything.

    8.
    I have already uploaded a powershell script to my machine. It will run some commands and output this to a text file locally on the system. The local user I have created is called “bob” with the password of “secretpassword”. The system I am targeting is a system called windows.domain.local. Note that it is not joined to a active directory domain, it is just a domain I use for testing. You could do it with kerberos and an active directory account as well. The command line for executing the powershell script is:

    The –uninstall parameter removes the service that winexe installs after it is done.
    The other parameters you can read about here

    To just run a simple ipconfig you can do like this

    Or maybe drop yourself into a shell:

    References
    http://www.aldeid.com/wiki/Winexe#Common_options
    http://sourceforge.net/p/winexe/winexe-waf/ci/master/tree/#
    http://ss64.com/nt/cmd.html

    Guide for setting up yubikey OTP ssh access with CentOS 7

    Hi!

    Today I will show you how to use the yubikey and set up authentication on CentOS 7 from scratch.
    I have bought a yubikey 2 standard and will configure it to be used as OTP device when logging on via SSH. This is very cheap solution which offers a high level of security for home users.

    The system I am using has just been installed and it is standard CentOS 7 with just default selections when installing. System is also updated and current to this day.

    Let’s start!

    1.
    Begin by installing some packages

    2.
    Clone the yubico-c library and install it.

    If you get the error messsage

    You need to install asciidoc

    3.
    Clone the yubikey-personalization library and install it

    3.
    Clone the yubico-c-client and install it

    4.
    Clone the yubico-pam module and install it

    5.
    Fix the symlink for the pam module

    NOTE! Before you continue, this is where you can lock yourself out and create a box that is not accessible. You should be logged on with root on another terminal so that you can revert back if it does not work. Console access will still work, so if you use KVM or something similar, you should be able to access it through that.

    Start by making sure your yubikey is configured as shown in this guide:
    YubiKey YubiCloud Configuration

    You should also make sure that the OTP function is working by going here after you have configured it:
    https://demo.yubico.com/

    Your server will need to be able to contact the yubikey API service to validate the auth, so make sure you have internet connection to the box as well.
    When it is working, move on!

    6.
    Change SELinux to run in permissive mode. People have been reporting issues with SELinux and yubikey so I recommend that you put it in permissive mode. If you don’t want to spend a couple of hours troubleshooting.

    Edit the file /etc/sysconfig/selinux and set the parameter SELINUX to permissive as shown below.

    This will run at boot next time and change it to permissive. Now run the command to change mode to permissive in this session

    Get the status. You can see the row “Current mode”, should say “permissive”

    7.
    Create a user, I have picked the name ‘test’. Make sure you change the password as well.

    8.
    Edit the file /etc/pam.d/sshd

    Make sure it contains the row:

    as seen below in the file:

    9.
    Crete the authfile mentioned above.

    Add your user and yubikey id separated with ‘:’. This is how my /etc/yubipasswd file looks like:

    Open a new terminal and login as test

    ssh test@your-yubikey-configured-system

    When you are promted for the password, write your password and press your yubikey so that it becomes one long string. Imagine:

    [root@localhost test]# ssh test@localhost
    test@localhost’s password: password + yubikey OTP token

    The yubikey will press enter by default and this should log you on! Voila!

    Other information
    If you have multiple configurations on your yubikey, i.e Configuration Slot 1 and Slot 2 you may have used the wrong Slot. Slot 1 is pressed for 1 second and the other is above 2 seconds. (Or something like that). I use Configuration Slot 1 because then you dont have to press it for a longer period of time.

    If you have installed a default CentOS 7 you probably do not need to change other SSH configuration files such as /etc/ssh/sshd_config

    And /etc/pam.d/system-auth by adding “try_first_pass” to the row

    You can also try installing the packages via epel repo. Packages are called:

    The pam library I think you will have to install manually as previously shown.

    Links related to this guide
    https://demo.yubico.com/
    http://www.yubico.com/wp-content/uploads/2013/07/YubiKey_YubiCloud_Configuration.pdf
    https://developers.yubico.com/yubico-pam/Yubikey_and_SSH_via_PAM.html
    https://developers.yubico.com/yubikey-personalization/
    https://github.com/Yubico/yubico-pam#readme

    Good luck!

    Putting together a small systemd service script and controlling python

    Hello!

    This time I will demonstrate how to create a simple systemd service file so that you can control your applications with systemctl in CentOS 7. As you know systemctl will be replacing init so it is time to see what you can do with it.

    Imagine the following sample script called hello.py. It will not do anything but print hello world and then sleep for one second.

    Make sure you set the permissions so that it is executable. I will be placing this script in the folder /usr/local/bin/hello/

    The next step is to create a small bash script that will execute the python script. In this script you can do more stuff if needed. I will call it run.sh

    Make sure it is also located in /usr/local/bin/hello/

    Now you want to be able to control that running a service command, i.e “service hello start” or
    “systemctl start hello.service”

    Start by navigating to the systemd folder and create the hello.service file. This is where we will tell systemctl what will happen.

    Edit the hello.service file and make sure you point ExecStart to the bash script called run.sh

    Now, there are a few things you need to know. The parameter “ExecStart” is what going to happen when you tell systemctl to “start”. There are also “ExecStop” and “ExecReload”. Those will be used for? You guessed it, when stopped and reloaded. There are also nice stuff such as “ExecStartPre” and “ExecStartPost” to execute commands before and after the main app has started.

    More info can be found at Red Hat

    However, in our service file above we only need the settings I have provided.
    So now it is time to execute it:

    As you see I am using the old service command but CentOS 7 is redirecting to systemctl, that is fine. Both will work. The status command also shows that it is running.

    We can also confirm by checking what is running

    If you change any of the contents of the hello.service file you will be given the following message:

    That is fine, just execute “systemctl daemon-reload”

    To enable this on startup you can just do the classic chkconfig hello on and systemctl will take over:

    If you have a python application that may involve mongodb or some other service like mysql you can edit your service file to look like this

    This will mean that syslog, network and the mongodb service is required to start before your app is.

    Have fun!

    Setup a OpenVPN server on FreeBSD 10.1

    Hey,

    I just wanted to recommend this guide if you are interested in setting up a VPN server at home or in the cloud.

    https://www.digitalocean.com/community/tutorials/how-to-configure-and-connect-to-a-private-openvpn-server-on-freebsd-10-1

    I recommend you using at least 4096 key size and AES 256. If you go any higher it will take a lot of time to generate the keys, and you will get bad performance and timeouts (even with only one client) on the standard $5 per month server at DigitalOcean. More CPU cores will enable you to run stronger crypto!

    Good luck!

    Converting ova images to qcow2

    The ova format is an open format for virtual machines. In this small guide I will explain how you can convert it to qcow2 format and run it in KVM.

    In my example I am using LogPoint which is a SIEM product that you can download a trial of here

    The file you download will be called logpoint.ova. Once you have it you can check out the file format using the file command:

    You can also have a look at the contents of the tar archive

    The next step is to extract the files. Just run

    Time for some conversion!

    The next step is to logon with ssh to your KVM server and start virt-manager and import the image.

    In virt-manager choose new and then import existing image. If you get a black screen on the console try to change the video to QXL and restart it. As LogPoint is based on Ubuntu you can change that while in the guide.

    Good luck!