January 2024

Why this site?

Getting my home server to its current glory has been a winding journey of discovery. This is a recap of what I have achieved and some breadcrumbs for others (or my future self) to replicate the setup.

TLDR

If you need a public IP address that you can use for your hosted services but your home internet setup doesn’t give you one directly then you need another solution. Setup a VPS as a dumb router, forwarding packets based on port numbers and IP addresses. The clever reverse proxy logic is performed later on your server and on your local network. All you need is Wireguard, dnsmasq and a way to manage iptables to set everything up. When done correctly your services are none the wiser but you get to have the best of both worlds: local traffic stays local and services that need a public IP are connected to the VPS.

What’s it all about?

The problem to solve:

So as a little background, in an earlier version of this setup | was using the nginx reverse proxy to redirect packets from the VPS to my local server (using its stream function). This worked but had a major downside in that nginx changed the source and destination IP address on every packet. This meant that all my packets from outside of my network looked like they were originating from the VPS (on its Wireguard interface). The advantage here is that no routing is required, the packets are edited with the destination on my local machine and at the same time the return address is the VPS so all the connections can still send data both ways.

I wanted to actually see the true source IP address, and in a way that I didn’t need to configure each of my services and the nginx reverse proxy to send the originating IP tucked away in the custom proxy headers.

The problem is that if the packets have their original IP source address then as soon as they make it onto my local network they won’t know to go back through the Wireguard tunnel and will instead try and return through my default gateway (192.168.0.1). This will probably break most TCP handshakes and so two way connections are impossible.

I hear you throw your hands up and exclaim: “but how will the packets find their way home?” They need to go back the way they came in! One solution is to simply set the wireguard connection as the default connection for the entire server. A better solution is to only change the default gateway exactly where it is needed and leave the majority of connections unchanged. So we must configure default routes for a few docker containers.

Example connections:

So I have a few scenarios that I need to define so that we can understand the motivations for all the different configs later. I had the goal of being able to use my hosted services at home and when I am out (or on another network) seamlessly. In addition I have some services that I only want to use on my local network or over the Wireguard network (so they should also seamlessly work on either). One last requirement: if I am connected to the Wireguard network then all my normal internet browsing should also be safe behind the VPN.

So what does all this look like?

Below I have a list of what I want to connect to (destination) and where I want to connect from (source) and then a table of how the different combinations work together.

Connection Destinations:

1. (Sub) Domain either http or https (e.g. mail.myowndomain.com or photos.myowndomain.com)
2. Local machine + port number (e.g. server:1883 or server:8008)
3. External website (via VPS IP) (e.g. google.com or ifconfig.net)

Connection Source

1. Local (e.g. on my wifi)
2. External (e.g. on my cell phone)
3. Wireguard (either but with Wireguard activated on the device)

A Table!

So the result is that because I have subdomains for my most frequently used or ‘always connected’ services, I don’t use the Wireguard app on my phone or my laptop very much. I only need to manually activate the Wireguard connection if I want to hide my traffic from my ISP or the local network or on the rare occasion that I want to access something that I would normally access when I am at home (like a samba share or my Zigbee manager and MQTT bridge for example). Using Nextcloud or Pairdrop works as expected. I have access at home or away without having to think about it and any links that I share with friends work fine because the service has its own subdomain. In addition local access is blazing fast because it is entirely local!

Network Map

OK now we get objective let’s look at some network layouts etc.

Another Table!

The goal here is for us to set a series of rules so that packets within our networks are sent to the correct destination but the challenge is that for the most part we want to manipulate the packet destination and source address as little as possible.

Its a diagram!

This is the logical network diagram for how connections will traverse from the internet to our services.

The ports that I have open are in two groups: 80 and 443 for http and https respectively. 25, 465, 993 are for my mail server.

                         Public Internet
                               │
                123.123.123.123│ 80,443,25,465,993
                    ┌----------┴-----------┐
                    │                      │
                    │         VPS          │
                    │                      │
                    └----------┬-----------┘
                    10.10.10.1 │
                               │
                    10.10.10.2 │
                    ┌----------┴-----------┐
                    │                      │
                    │      Wireguard       │
                    │       (Docker)       │
                    └----------┬-----------┘
                   172.0.30.50 │
                               │
                ┌--------------┴----------------┐
      172.0.30.3│ 80,443             172.0.30.2 │ 25,465,993
     ┌----------┴-----------┐        ┌----------┴-----------┐
     │                      │        │                      │
     │    Reverse-Proxy     │        │      Mailserver      │
     │       (Docker)       │        │       (Docker)       │
     └----------------------┘        └----------------------┘
    

The Network of Networks

DNS

This took me a while to understand the importance of hosting my own DNS but once it was setup everything just works in the background magically. In this setup we will be connecting from one of three networks, the local network, the Wireguard network or the public internet. The public internet has its own DNS servers (that we will need to config) but then we still need two of our own DNS servers for the local and Wireguard networks. I use Pihole on my home network but I used dnsmasq on the VPS because once it is setup I don’t intend to change it at all. I run Pihole as a docker container and I have set the default DNS on my router to point to it (but I also left a public DNS as a fallback).

Secure Tunnel

Our VPS will act as the central connection point for all the Wireguard connections. See the guide for setting up a default connection. The other Wireguard node will be running in a Docker container and will route traffic from specific Docker containers. Additionally other devices will connect to to the VPS via their own Wireguard clients (e.g. my laptop and mobile)

Containers!

Running services in Docker has many benefits (and some cons) and I don’t intend to go into them here but I do need to describe how the virtual networks in docker work alongside the Wireguard overlay network, my actual local network and finally my ISPs network. By using Docker we create a dedicated network so that our Docker services can reach the Wireguard container and so that other services are not impacted.

Default Gateways

The last piece of the packet flow is getting the packets to return to their correct source. This is done by editing the default IP route within each container that we want to connect to the Wireguard network (and effectively use the VPS public IP). Unfortunately the only way to do this is to run commands within each docker container after they have been started. Luckily if we chose our containers carefully (or possibly build them ourselves…..) we can simply add a startup script that runs the necessary commands. This way if we reboot the host server or if the docker container is disrupted then at least we don’t need any manual intervention. The startup script does two things: it redirects default traffic to the Wireguard Docker container and it makes an exception for traffic on the local network (192.168.0.0/24), sending local traffic to the default docker network gateway.

Routing with ‘ip tables’

The routing at each stage is easy to comprehend:

1. Allow connections on designated ports
2. Allow ipv4 packet routing across interfaces
3. Forward connections on specific ports to an ip address
4. When packets are sent back, masquerade our outward facing address on the return address

Here is what that looks like:

From Internet to mail.myowndomain.com

A connection is being made from IP: 210.210.210.210 to my domain at IP: 123.123.123.123.

The packet will look like this:

    INBOUND PACKET  
    FROM               TO  
    210.210.210.210  123.123.123.123  
    25               25  
    

Then the VPS receives the packet and according to the iptables rule the packet is forwarded to 10.10.10.2.

Now the packet looks like this:

    INBOUND PACKET  
    FROM               TO  
    210.210.210.210    10.10.10.2  
    25               25  
      

Now the Wireguard Docker container will receive this packet and according to its own iptables rules will forward the packet to the Mailserver container.

The packet is updated:

    INBOUND PACKET  
    FROM               TO  
    210.210.210.210    172.0.30.2  
    25               25  
    

At this point the Mailserver container will accept the packet and begin negotiating a proper connection so it will need to send a packet back to the sender.

The packet will look like this:

    OUTBOUND PACKET  
    FROM               TO  
    172.0.30.2      210.210.210.210  
    25               25  
    

But wait! 210.210.210.210 is not on our network so this packet will need to go to a gateway. The Mailserver will not send this packet to the network gateway of 172.0.30.1 but instead it will send this packet to the Wireguard Docker container. The packet is still unchanged however at the Wireguard Docker container the default gateway is 10.10.10.1 and our iptables masquerade rule will change the source address.

The packet becomes:

    OUTBOUND PACKET  
    FROM               TO  
    10.10.10.2      210.210.210.210  
    25               25  
    

Now the packet is routed through the VPS and again an iptables rule will masquerade the source IP.

The packet finally looks like this when it leaves the VPS:

    OUTBOUND PACKET  
    FROM               TO  
    123.123.123.123   210.210.210.210  
    25               25  
      

and finally ………

How to

I borrowed steps form a few other How-To guides (links at the bottom) so anything that is copied all the credit goes to the original authors. At the same time I want this guide to be complete and not depending on external links.

VPS

I setup the cheapest Linode instance running Debian. I also setup a new user with sudo privileges so that I am not logging in as root. If you decided to follow along as the root user then ‘sudo’ is not required.

VPS - Wireguard

First we need to install wireguard using the apt package manager.

~$ sudo apt update

~$ sudo apt install wireguard

Next we will make a new directory in our home directory creatively called ‘Wireguard’. Then we use the ‘wg’ command to generate the private and public key pair and save them into appropriately name files in the Wireguard directory. Next we need to reduce the permissions of the privatekey file to keep the private key a secret (maybe this step is unnecessary as hopefully the VPS is only accessed by us). Finally we make a new file called wg0.conf and also restrict its permissions, as it will contain a copy of the private key. Then open it with our text editor nano.

~$ cd ~

~$ mkdir wireguard

~$ cd wireguard

~/wireguard$ sudo wg genkey | tee privatekey | wg pubkey > publickey

~/wireguard$ sudo chmod 600 privatekey

~/wireguard$ sudo touch wg0.conf

~/wireguard$ sudo chmod 600 wg0.conf

~/wireguard$ sudo nano wg0.conf

Next we can paste this template into the config file and fill in all the details:

GNU nano              wg0.conf

    [Interface]
    Address = 10.10.10.1/24
    ListenPort = 51820
    PrivateKey = SERVER_PRIVATE_KEY
    MTU = 1500
    

We can print the private key onto our screen so that we can copy-paste it into our wg0.conf with “sudo cat privatekey” After we are happy with wg.conf we can copy it into its final destination so that Wireguard will use it when it is running. This seems like an extra step but it means that we can edit the Wireguard config file but the changes will only be applied when we decided to update the actual config file at “/etc/wireguard/wg0.conf” Finally we use the command ‘wg-quick’ to launch Wireguard and also enable systemctl to launch Wireguard after a reboot.

~/wireguard$ sudo cp ~/wireguard/wg0.conf /etc/wireguard/wg0.conf

~/wireguard$ sudo wg-quick up wg0

~/wireguard$ sudo systemctl enable wg-quick@wg0

We will come back to wg0.conf in the future when we have the other end of the tunnel setup on out local server. Next we need to update a config file “/etc/sysctl.conf” to allow packet forwarding. We open the config file with nano.

~/wireguard$ sudo nano /etc/sysctl.conf

Find the line that looks like this: “#net.ipv4.ip_forward=1” and make it look like this “net.ipv4.ip_forward=1”

GNU nano              /etc/sysctl.conf

    # Uncomment the next line to enable packet forwarding for IPv4
    net.ipv4.ip_forward=1
    

After saving /etc/sysctl.conf update the config with the following command.

~/wireguard$ sudo sysctl -p

OK let’s move on to our local server and get the other end of the Wireguard connection going!

Local Server - Docker

I assume that we already have Docker installed with Docker Compose.

Here are some useful commands that we will use but that are just universally useful when using Docker containers:

~$ docker compose up -d

This launches all the containers in our compose.yml file (assuming its in the same directory)

~$ docker exec -it [container_name] sh

This connects us to a bash command prompt inside the container

~$ docker exec -it [container_name] curl ifconfig.net

This will return your public IP address, this is helpful to check that the packet routing is working as depending on which container you run this in the result will be different.

Local Server - Pihole

Not much of a guide. Setting it up with Docker is trivial once you are used to setting up compose configs.

GNU nano              compose.yml

    services:
      pihole:
        container_name: pihole
        image: pihole/pihole:latest
        restart: unless-stopped
        ports:
          - 53:53/tcp
          - 53:53/udp
          - 8080:80/tcp
        environment:
          TZ: America/New_York
          FTLCONF_LOCAL_IPV4: 192.168.0.xx
          PIHOLE_DNS_: 9.9.9.9;1.1.1.1
          IPv6: false
        volumes:
          - ~/pihole/etc-pihole:/etc/pihole
          - ~/pihole/etc-dnsmasq.d:/etc/dnsmasq.d
    

Local Server - Wireguard

For the local Wireguard docker container I went with linuxserver.io so the setup will have a few differences to our “bare metal” install on the VPS.

We setup the compose file as follows:

GNU nano              compose.yml

    services:
      wireguard:
        image: lscr.io/linuxserver/wireguard
        container_name: wireguard
        cap_add:
          - NET_ADMIN
          - SYS_MODULE
        environment:
          - PUID=1000
          - PGID=1000
          - TZ=America/New_York
        volumes:
          - /path/to/app/data/wireguard-client:/config
          - /lib/modules:/lib/modules
        networks:
          default:
            ipv4_address: 172.0.30.50
        sysctls:
          - net.ipv4.conf.all.src_valid_mark=1
        restart: unless-stopped

    networks:
      wgnet:
        name: wgnet
        external: true
        ipam:
          config:
            - subnet: 172.0.30.0/16
    

We configure the Wireguard config file like this:

GNU nano              wg0.conf

    [Interface]
    PrivateKey = CLIENT_PRIVATE_KEY
    Address = 10.10.10.2/32
    DNS = 10.10.10.1

    PostUp  = iptables -t nat -A POSTROUTING -o wg0 -j MASQUERADE
    PreDown = iptables -t nat -D POSTROUTING -o wg0 -j MASQUERADE

    PostUp  = iptables -t nat -A PREROUTING -p tcp -d 10.10.10.2 -m multiport --dports 25,465,993 -j DNAT --to-destination 172.0.30.2
    PreDown = iptables -t nat -D PREROUTING -p tcp -d 10.10.10.2 -m multiport --dports 25,465,993 -j DNAT --to-destination 172.0.30.2

    PostUp  = iptables -t nat -A PREROUTING -p tcp -d 10.10.10.2 -m multiport --dports 80,443 -j DNAT --to-destination 172.0.30.3
    PreDown = iptables -t nat -D PREROUTING -p tcp -d 10.10.10.2 -m multiport --dports 80,443 -j DNAT --to-destination 172.0.30.3

    [Peer]
    PublicKey = SERVER_PUBLIC_KEY
    AllowedIPs = 0.0.0.0/0
    Endpoint = 123.123.123.123:51820
    PersistentKeepalive = 25
    

Launch the Wireguard container like this:

~$ docker compose up -d wireguard

Now we need to go back to the Wireguard config on the VPS to save the client public key from the docker Wireguard container so that the Wireguard connection can be made.

GNU nano              wg0.conf

    [Interface]
    Address = 10.10.10.1/24
    ListenPort = 51820
    PrivateKey = SERVER_PRIVATE_KEY

    [Peer]
    PublicKey = CLIENT_PUBLIC_KEY
    AllowedIPs = 10.10.10.2/32
    

To update the Wireguard docker container just restart the container:

~$ docker restart wireguard

To update the Wireguard daemon running on the VPS like this:

~$ sudo wg-quick down wg0 ~$ sudo cp ~/wireguard/wg0.conf /etc/wireguard/wg0.conf ~$ sudo wg-quick up wg0

Docker - Gateway Script

ip rules

We need to run three commands to get the routing setup within our container:

~$ ip route del default

~$ ip route add default via $WIREGUARD_GATEWAY

~$ ip route add $LAN_SUBNET via $WGNET_GATEWAY

You can test that this works by using the ‘docker exec -it’ command to manually run the commands and then test that the routing is working. I found that for my nginx container I also needed to install iproute2 to get this to work. In the end I put this into a script that uses environment variables taken from the compose.yml file.

GNU nano              network-setting.sh

    #!/bin/bash

    if [ -n "$WIREGUARD_GATEWAY" ] && [ -n "$LAN_SUBNET" ] && [ -n "$WGNET_GATEWAY" ]; then
        echo "**** Setting wireguard as default route except for lan access ****"
        apt update -y && apt install iproute2 -y && apt autoremove -y && apt clean -y && apt purge -y
        ip route del default
        ip route add default via $WIREGUARD_GATEWAY
        ip route add $LAN_SUBNET via $WGNET_GATEWAY
    else
        echo "**** Wireguard route environment variables not set, skipping..... ****"
    fi
    

VPS - iptables

This can get complicated really quickly and is essentially the secret sauce making all this work properly!

GNU nano              rules.v4

    *filter
    # Drop incoming and forwarding packets; allow outgoing packets
    :INPUT DROP [0:0]
    :FORWARD ACCEPT [0:0]
    :OUTPUT ACCEPT [0:0]

    :UDP - [0:0]
    # Open wireguard port
    -A UDP -p udp --dport 51820 -j ACCEPT
    # Allow all UDP from wireguard network
    -A UDP -p udp -s 10.10.10.0/24 -j ACCEPT

    :TCP - [0:0]
    # Allow all TCP from wireguard network
    -A TCP -p tcp -s 10.10.10.0/24 -j ACCEPT

    :ICMP - [0:0]
    # Allow all ICMP
    -A ICMP -p icmp -j ACCEPT
    # allow all ICPM message forwarding
    -A FORWARD -p icmp -j ACCEPT

    # open public TCP ports for forwarding
    -A FORWARD -d 123.123.123.123 -p tcp --syn -m multiport --dports 80,443,25,465,993  -m conntrack --ctstate NEW -j ACCEPT
    # allow established connections to be forwarded between interfaces 
    -A FORWARD -i eth0 -o wg0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
    -A FORWARD -i wg0 -o eth0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
    # allow in and out on wireguard
    -A FORWARD -i wg0 -j ACCEPT
    -A FORWARD -o wg0 -j ACCEPT
    # Acceptance policy
    -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
    # Allow local loopback
    -A INPUT -i lo -j ACCEPT
    # Drop a packet if it is invalid
    -A INPUT -m conntrack --ctstate INVALID -j DROP

    # Pass traffic to protocol-specific chains
    # Allow only new TCP connections established with new SYN packets
    -A INPUT -p udp -m conntrack --ctstate NEW -j UDP
    -A INPUT -p tcp --syn -m conntrack --ctstate NEW -j TCP
    -A INPUT -p icmp -m conntrack --ctstate NEW -j ICMP

    # Reject anything that reached this point
    -A INPUT -p udp -j REJECT --reject-with icmp-port-unreachable
    -A INPUT -p tcp -j REJECT --reject-with tcp-reset
    -A INPUT -j REJECT --reject-with icmp-proto-unreachable
    # Commit the changes
    COMMIT

    *raw
    :PREROUTING ACCEPT [0:0]
    :OUTPUT ACCEPT [0:0]
    COMMIT

    *nat
    :PREROUTING ACCEPT [0:0]
    # Route tcp packets for ports
    -A PREROUTING -d 123.123.123.123 -p tcp -m multiport --dports 80,443,25,465,993  -j DNAT --to-destination 10.10.10.2
    :INPUT ACCEPT [0:0]
    :OUTPUT ACCEPT [0:0]
    :POSTROUTING ACCEPT [0:0]
    # all outgoing packets to internet have our public ip
    -A POSTROUTING -o eth0 -j MASQUERADE
    COMMIT

    *security
    :INPUT ACCEPT [0:0]
    :FORWARD ACCEPT [0:0]
    :OUTPUT ACCEPT [0:0]
    COMMIT

    *mangle
    :PREROUTING ACCEPT [0:0]
    :INPUT ACCEPT [0:0]
    :FORWARD ACCEPT [0:0]
    :OUTPUT ACCEPT [0:0]
    :POSTROUTING ACCEPT [0:0]
    COMMIT
    

VPS - dnsmasq

We need to run our own DNS server on the VPS

Install dnsmasq and edit the config file:

~$ sudo apt install dnsmasq

~$ sudo nano /etc/dnsmasq.conf

Edit the config as shown below and save:

GNU nano              dnsmasq.conf

    domain-needed
    bogus-priv
    no-resolv

    #explicitly define host-ip mappings
    address=/server/10.10.10.2

    #default nameservers: (quad9 and cloudflare)
    server=9.9.9.9
    server=1.1.1.1
    

Then when we are done editing:

~$ sudo systemctl restart dnsmasq

Tests:

If everything is routing correctly the following test commands should produce the following results:

From a terminal on your server (so obviously on your local network):

Before you go….

Work to be done

The iptables rules are still not 100% perfect and I think that there is probably a mix of redundant rules and maybe some loopholes that would ideally be fixed at some point. MTU set at 1500 is a very hacky solution and should not be there but I have not found a clean way for the connections to negotiate correctly using ICMP.

Acknowledgements

Links to tutorials:

1. How to set up a Wireguard VPN
2. routing docker traffic through Wireguard
3. iptables port forwarding