docker firewall blocking at the source port

Today I installed my brand new docker host system, started some images and as usual trying to setup iptables to restrict the ports to specific source servers and ports.

Hours later I learned that this was kinda not so easy.

  • You cannot use the INPUT chain as Docker works with the standard bridge in the FORWARD chain.
  • The DOCKER-USER chain (which is a part of the FORWARD chain) see packets after they have been redirected to the destination port, so if we have multiple containers running at port 8000 internally it get’s very complex to find the correct container, not to mention new containers starting up.

One example:

I have a docker image that runs a web-server listening inside the container on port 8000 which is mapped to the host port 80 as in this config:

ports:
– “80:8000”

This means my host is listening on port 80 and that get’s redirected to the Docker container on port 8000.

Now the DOCKER-USER chain sees only port 8000 where I could drop some other source IP.

But I want to selectively open ports and let not configured ones dropped by default, even if somebody spin’s up new containers.

The docker-documentation and some other sources disable the iptables feature of docker and manually configure the firewall, which is okay but not that what I wanted as the sources allow general inter-container communication or have a VERY complex rulesystem that I don’t want to explain the my Administrator colleagues of that system.

 

The solution to this is hidden within the iptables flow diagram:

The PREROUTING chain of the nat table is the first one that get touched and where the source port of 80 is redirected to the container on the destination port 8000.

So we tag the SYN packets we want to drop in the PREROUTING chain of the mangle table with the MARK target and later at the DOCKER-USER chain of the filter table we will drop them accordingly.

My final ruleset looks like that:

 

#!/bin/bash

#Flush and setup:
/sbin/iptables -F DOCKER-USER
/sbin/iptables -t mangle -F PREROUTING

##############
# Docker conf
###############
#packets in NAT table are only traversed with the first packet!
#MARK1: allowed packet
#MARK2: drop this in FORWARDING (DOCKER-USER)Chain
#Mark everything with 0x2, so block everything
/sbin/iptables -t mangle -A PREROUTING -i ens3 -m state –state NEW -j MARK –set-mark 2
###
# Mark the packets we want to allow with 0x1 (which are previously also mark’ed with 0x2)
###
/sbin/iptables -t mangle -A PREROUTING -i ens3 -m tcp -p tcp –dport 80 -s xxx.xxx.xxx.xxx -j MARK –set-mark 1
/sbin/iptables -t mangle -A PREROUTING -i ens3 -m tcp -p tcp –dport 443 -j MARK –set-mark 1

###
# Drop the packets marked with 0x2
###

/sbin/iptables -A DOCKER-USER -m mark –mark 2 -j DROP
/sbin/iptables -A DOCKER-USER -j RETURN

This ruleset will allow port 80 from a specific IP address and port 443 from every source IP. Everything else exposed by docker containers is not reachable for anybody – or at least they cannot open a TCP connection to it because the initial SYN packets are dropped away.

And the best is I can start as much containers as I want on port 8000 and don’t have to fiddle around with the internal docker ports.

Hope that helps somebody ;)

Advertisements

Gitlab “file changed as we read it”

Months after the OSCE and my shiny new OSCP certification, a refactor of the infrastructure and a current build-up of a “Security Operation Center”, I’m fully back at work.

My desk is fullblown with management documents and other shit I don’t want to read, instead I’m looking at my screen after I builded a Docker Infrastructure including a GitLab server:

root@7845ae79a54c:/# gitlab-rake gitlab:backup:create
Dumping database …
Dumping PostgreSQL database gitlabhq_production … [DONE]
done
Dumping repositories …
done
Dumping uploads …
tar: ./-/….. file changed as we read it
Backup failed

Hmm, great :) Not only that the gitlab server is no more than a day old, the backup won’t work. Grr.

Long Story short, after a lot of googling my problem was a GlusterFS mount on my docker host which is bound into the Docker container.

The fix is to issue i issued the following commands on my docker host:

gluster volume info
gluster volume set dockergfs cluster.consistent-metadata on

And after a host reboot (or remount just as you like), surprise surprise the backup works now!

 

If I find more time between my family, my private projects and my daily business I’ll write a review of the OSCP course if there is interest :)

Transparently sniffing and modify traffic with scapy in-path

I decided to play a little bit with scapy and tried to find something in the internet that helped me to sniff and modify traffic without ARP-Storming the whole network.

I want to connect a box to my computer eth0 and the network to eth1, so that I’m between the box and the network.

Note: This one does not defeat SSL right now, but that could be easily done with all the tools out there.

The idea is to transparently forward all traffic and only pick out specific packets for modification with scapy.

I did do this with an combination of bridged interface, iptables NFQueue and scapy.

First we gonna create a bridge between eth0 and eth1 to transparently forward all traffic:

ifconfig eth0 0.0.0.0
ifconfig eth1 0.0.0.0
brctl addbr br0
brctl addif br0 eth0 eth1
ifconfig br0 up

Second, we need to setup iptables to send the traffic we want to change to an NFQueue.
Important is that this line is executed within the startup of the scapy script, so the actual modding only happens if the script is running.
It would be bad if the script is not running and the packets don’t traverse..

The line we need to set in the scapy script is:

iptables -A FORWARD -m physdev –physdev-in eth1  -s 192.168.1.10 -j NFQUEUE

Third, our scapy program catches those packets, modify them if they are of interest for us and forwards them accordingly.
Important is to command scapy that it should also forward traffic we do not want to be modified (e.g. other SNMP calls in this example):

https://github.com/devBioS/scapy-mitm

What I actually do is to modify a response of our target system to the monitoring system.
I choose the SNMP response of an HP ProLiant power-supply check that checks if the power supplies are OK or not (or disconnected):

The “original” says that both power-supplies are degraded or in other words they have a problem:

# snmpwalk -v 1 -c public -O eq 192.168.1.10 1.3.6.1.4.1.232.6.2.9.3.1.4
iso.3.6.1.4.1.232.6.2.9.3.1.4.1.1 3
iso.3.6.1.4.1.232.6.2.9.3.1.4.1.2 3

# ./check_psu -H 192.168.1.10
PSU 1 DEGRADED, PSU 2 DEGRADED

after starting the scapy script, it will happily transform to something much better:

#./scapymod.py
Adding iptable rules :
iptables -A FORWARD -m physdev –physdev-in eth1 -s 192.168.1.10 -p udp -j NFQUEUE
[+] Running Main loop
Got a packet ! source ip : 192.168.1.10 dest: 192.168.1.60
Modding…
Old status: 2 (<ASN1_OID[‘.1.3.6.1.4.1.232.6.2.9.3.1.4.1.1’]>)
New status: 2 (OK)
Got a packet ! source ip : 192.168.1.10 dest: 192.168.1.60
Modding…
Old status: 2 (<ASN1_OID[‘.1.3.6.1.4.1.232.6.2.9.3.1.4.1.2’]>)
New status: 2 (OK)

and in the console:

# snmpwalk -v 1 -c public -O eq 192.168.1.10 1.3.6.1.4.1.232.6.2.9.3.1.4
iso.3.6.1.4.1.232.6.2.9.3.1.4.1.1 2
iso.3.6.1.4.1.232.6.2.9.3.1.4.1.2 2

# ./check_psu -H 192.168.1.10
PSU 1 OK, PSU 2 OK

And this all by just carefully crafting and modifying packets

Have Phun :)

ESXi 5: Expanding datastore by extending local array

Recently i extended my RAID 5 array by replacing one-by-one hdd’s with bigger ones and waiting for each to rebuild.

All the instructions are executed on a HP DL360 G6 with an SmartArray P410i controller with only one logicaldisk.

It is highly recommended that you backup all your VM’s before executing a single command – everything worked fine for me but one error in a command could leas to a complete data loss of everything!

Continue reading “ESXi 5: Expanding datastore by extending local array”

SSL Funambol on Android without adding Certificates

Recently I wrote a blog entry about how to add a root certificate to your rooted android keystore for using SSL funambol.
This works perfectly as long as you have root access to the device.

Sometimes you come into the situation where you cannot root the device, eg. in a corporate environment or if you just don’t want to crack a new device just to make the funambol client working, like me for now.

I’ve got a new Motorola Xoom and needed funambol to sync my contacts and calendar entries.

After asking Mr. Google there are only 2 ways till Android 4.x is ready:

  • Using http without ssl
  • Using ssl and recode the funambol client to accept all cert’s

I decided to use the 2nd solution – this also refreshes my java a little bit :-)

Of course i want to share everything with you – if you’re too lazy to read all the stuff you can point your Android client here to install my compiled Funambol 10.0.8 client without the certificate check:

For Android < 4.0 this binary works:

Direct download with android: funambol-android-10.0.8_Tasks-devBioS.apk

Mirror 1: funambol-android-10.0.8_Tasks-devBioS.apk

Since ICS (>= 4.0) people having problems syncing Calender. This is because they made changes in the Calender API wich renders the Calendar Sync unusable, you can use this binary (which only works on ICS and greater):
Direct download with android: funambol-android-10.1.3_Tasks-devBioS.apk

UPDATE 2012-04-03: I activated the task sync feature also and re-uploaded the binary.
If anyone is intrested in the source on how to activate it, drop me a line and i will update the post.

Continue reading “SSL Funambol on Android without adding Certificates”

Adding Root Certificates to Android Phone with root access

Yeah, after being really busy with my real life, here is another intresting trick for you:

How to get some more root certificates on an android phone where you have root acces (or, at least, you can start & use root explorer).

Some background info:
I use a funabol community server to keep my phone’s and outlook’s in sync and recently my colleague sven did a great job converting my HTC HD2 with winmobile to Android 2.x (kudo’s to him! thanks!).

Update on 2011-12-27: I changed the client to allow self-signed certificates: here

The challenge is that if you use funabol with self signed ssl certificates you need to get those recognized by android which is a really complicated task if you don’t know how.  But here we go:

What you need before (and what i don’t describe):

Our steps include:

  1. Export the certificate out of the funambol java keystore
  2. Get the cacert.bks from the android device
  3. modify the cacerts.bks of android
  4. reboot and finished

Step 1 – Export the funambol certificate

  • Execute  “%JAVA_HOME%\bin\keytool -export -alias tomcat -file myroot.cer”  (in-detail like here)
  • copy the myroot.cer to the SD-Card of the android device (or download to another computer)

Step 2

  • insert the SD-Card to the android device, startup root explorer and navigate to /etc/security/
  • copy the file cacert.bks
  • navigate to /sd-card and paste the file
  • insert to another computer
  • Go and execute the Portecle Keytool and open the cacerts.bks from your SD-Card
  • When promtes for a password, just hit enter
  • go to Tools -> Import Key Pair   and select your myroot.cer, give it any name you want
  • save the cacerts.bks
  • re-insert the SD-Card to android device
  • open up root explorer, head to /sd-card, copy, paste to /etc/security/
  • make sure root explorer show “mounted as r/w” in the header of the program.
  • set permissions of the newly copied cacert.bks to rw-r–r– (owner,group,other: read     owner:write)

Step 3

  • double check if the permissions of cacerts.bks are set correctly to rw-r–r–
  • restart the phone
  • funambol sync should now complete.

Have phun to be in sync!

P.S. Keep in touch! the next xbee’s blogpost’s are half-way written, but i really don’t have time ATM… sry

QuickFix: IE8 hangup opening intranet pages

Something that i always hate is when there is no update and a product stop’s working so i cannot do my work anymore.

In this case i (am forced to) use Internet Explorer to open up several intranet pages that allowed me to do my administration work.

The problem sums like this:

  • Google, and everything else on “the internet” works seemless
  • opening something internal just hang up IE8 for a couple of minutes and then just stop loading without an error
  • Same in FireFox with IETab installed
  • same in IE 7 + IE 8 64 bit and 32 bit

The solution is as easy as stupid:

In “Internet Options”  -> “Security” the “Local Intranet” is set up with disabled “Protected Mode”.
Enabling it fixed
the Problem instantly for IE8, IE7 and Firefox with IETab.