How to Reverse Proxy Properly With Caddy and Docker

2023-03-18 (Sat)
5-minute read

Warning: This post is over 365 days old. The information may be out of date.

This is a tutorial on how to properly set up Caddy running in a Docker container.

The prerequisites

A server with the Docker service enabled. Let’s (host)name the server timmy.
Some services you want to expose to the web
A custom Docker network that houses these services (let’s say proxynetwork here)
(optional) Some services you don’t want to expose to the web
(optional) A VPN service such as Wireguard to access those services

Why reverse proxy?

Because typing service.timmy.example.com is a whole lot better than 192.168.1.200:4343 or timmy:4343.

Especially once you have more than two or three services to access. Good luck remembering all the port numbers!

Part 1 - The outer layer

First off, install Caddy with the host networking option. (I’ll explain why later.) Then in the configuration, map the domains according to your needs:

# Set up logging
(logs) {
    log {
        output file /data/logs/access.log
    }
}

# Hide the services from search engines
(no_robots_txt) {
    respond /robots.txt 200 {
        body "User-agent: *
Disallow: /
"
        close
    }
}

# Main page
timmy.example.com {
    # You can either reverse proxy to a service...
    reverse_proxy 192.168.1.200:8080
    # Or respond like this (uncomment the following line)
    #respond "Hello World!"

    import no_robots_txt
    import logs
}

# Different services
*.timmy.example.com {
    import logs

    @plex host plex.timmy.example.com
    handle @plex {
        # `172.18.0.1` is the Docker host IP
        reverse_proxy 172.18.0.1:32400
        import no_robots_txt
    }
}

Pretty standard-fare stuff. Once you have the container running, Caddy will take care of things like SSL certificate management with Let’s Encrypt, assuming timmy.example.com is pointing to the public IP of the server, and *.timmy.example.com is a CNAME to timmy.example.com.

If that’s all you wanted to do, you can stop here! But what if you have some internal services you want to access over VPN? Then read on.

Part 2 - Internal services

This is the tricky part, because we want to restrict the people that can access internal services to those that are connected over VPN only.

Let’s set up the internal Caddy server first. Create the following Caddyfile:

# Set up logging
(logs) {
    log {
        output file /data/logs/access.log
    }
}

:80 {
    @main host timmy.private.example.com
    handle @main {
        reverse_proxy foo:8080
    }

    @jenkins host jenkins.timmy.private.example.com
    handle @jenkins {
        reverse_proxy jenkins:8080
    }
}

For this to work, timmy.private.example.com must point to the private IP inside the VPN network, and *.timmy.private.example.com must be a CNAME of timmy.private.example.com. You can publish those to the public DNS – they’re internal addresses anyway so it’s not like people can access them.

Now, you might be wondering why we are using :80 for the server block, instead of something like timmy.private.example.com or *.timmy.private.example.com. And that’s because since this is an internal Caddy server that gets reverse-proxied-to by another Caddy server, we don’t want it automatically generating SSL certificates or what not. In fact, having multi-layered HTTPS in a reverse proxy setup is generally a Bad Idea™, which is why we will only listen on the standard HTTP port 80.

Also, you may have noticed that we are using hostnames to refer to services, like jenkins. This only works if you use the aforementioned custom Docker network, so you need to make sure that both the service (Jenkins, for example) and the internal Caddy server is on the same network! In this case, I would create the internal Caddy server on the proxynetwork, and forward port 80 to 680 on the host machine.

Great, so now all we have to do is join those two together!

Part 3 - Joining the two servers

On the outer Caddy server, add the following to the Caddyfile:

# Reverse proxy to internal Caddy
timmy.private.example.com {
        import logs

        reverse_proxy 172.18.0.1:680
}

*.timmy.private.example.com {
        import logs

        reverse_proxy 172.18.0.1:680
}

Now you should be able to access the services. All set, right?

Not so fast!

Not so secure

You’d think that since timmy.private.example.com points to a private IP address inside the VPN network, people on the Internet cannot access those services.

But see the host keyword? Caddy checks what host the client is visiting and forwards them to the appropriate blocks. So what does Caddy check? It checks the Host header embedded into every HTTP/S request.

Do you see the problem yet? HTTP/S requests are made by clients, and their contents can be arbitrary.

Try a simple test: disconnect from your VPN network, open up the hosts file on your operating system, and add an entry that points timmy.private.example.com to the public IP of your server. Then visit timmy.private.example.com. What do you see?

Because this check can easily be bypassed with something so simple as a hosts file, it is NOT secure, so we have to do something about it.

Part 4 - Hardening the outer shell

Back at the outer Caddy server layer, add the following snippet:

(block_nonvpn) {
    @notVPN not remote_ip 10.10.0.0/24
    handle @notVPN {
        respond "Access denied" 403
    }
}

You will need to replace 10.10.0.0/24 with the actual IP range of your VPN network.

You might be tempted to not respond at all, and terminate the session immediately upon figuring out that a public IP is trying to access a private server block:

(block_nonvpn) {
    @notVPN not remote_ip 10.10.0.0/24
    handle @notVPN {
        abort
    }
}

However, this is not recommended, as you can set up something like fail2ban to find abusers with the 403 entries in the access log, and ban them from the network and reduce load.

Of course, for this to work, the outer Caddy layer must be in the host network mode. In my experiments, any other network mode causes Caddy to recognize all clients as originating from 172.18.0.1, or the internal IP address of the Docker network layer that points to the Docker machine host.

Then, add the snippet to the private services, like so:

# Reverse proxy to internal Caddy
timmy.private.example.com {
        import logs
        import block_nonvpn

        reverse_proxy 172.18.0.1:680
}

*.timmy.private.example.com {
        import logs
        import block_nonvpn

        reverse_proxy 172.18.0.1:680
}

If you try the hosts file trick again, you should see that it no longer works!

Conclusion

I hope this helps with configuring Caddy for reverse-proxying to different Docker services!

Thanks to

@fvbommel on the Caddy forums for the security advice