Troubleshooting Supabase Self-Hosted: A Journey Through Docker DNS and Service Discovery

Introduction

When setting up a self-hosted Supabase instance, you might encounter various challenges, particularly around service discovery and networking. This blog post documents our journey through troubleshooting a Supabase setup, specifically focusing on issues with Kong (the API gateway) and service resolution.

The Initial Setup

Our setup involved a self-hosted Supabase instance using Docker Compose, with services including:

Kong (API Gateway)
Supabase Studio
Authentication Service
PostgreSQL Database
Storage Service
Analytics
and more…

The Challenge

The main issue we encountered was that Kong was unable to properly route requests to the Supabase Studio and other internal services. The complexity was increased by our use of Tailscale VPN for remote access. The symptoms included:

“An invalid response was received from the upstream server” errors
Kong logs showing attempts to resolve services to external IP addresses (specifically Tailscale IPs)
Services appearing healthy but not accessible through Kong

Investigation Process

1. Initial Diagnostics

We started by checking Kong’s logs, which revealed that it was trying to resolve internal service names to an external IP address (124.170.73.156) instead of using Docker’s internal DNS system. This was particularly interesting because the IP was associated with our Tailscale network.

2. Configuration Analysis

The investigation led us to examine several key configuration files:

kong.yml: Contains service definitions and routing rules
docker-compose.yml: Defines the service network configuration

3. DNS Resolution Testing

We performed direct service accessibility tests using:

docker exec supabase-kong wget -qO- http://172.20.0.13:3000/

This helped confirm that the services were actually accessible within the Docker network.

Key Findings

Docker DNS Initialization: The most important discovery was that Docker’s internal DNS system needs time to properly initialize and stabilize. Initial connection failures don’t necessarily indicate a configuration problem.
Service Resolution: While it’s possible to hardcode IP addresses using extra_hosts in Docker Compose, it’s better to let Docker’s built-in DNS handle service resolution.
Container Dependencies: The depends_on conditions in the Docker Compose file ensure services start in the correct order, but they don’t guarantee that services are fully initialized and ready to accept connections.

The Solution

The solution turned out to be simpler than expected:

Allow sufficient time for all services to fully initialize
Let Docker’s internal DNS handle service resolution naturally
Ensure all services are on the same Docker network (default network in this case)

Best Practices Learned

Patience During Startup: Give the system time to fully initialize before concluding there’s an issue.
Trust Docker DNS: Rather than hardcoding IP addresses, trust Docker’s built-in DNS resolution system.
Check Service Health: Use Docker’s healthcheck mechanisms to ensure services are truly ready.
Monitor Logs: Keep an eye on logs during startup to understand the initialization process.

Additional Complexity: VPN Considerations

An important factor in our troubleshooting journey was the presence of Tailscale VPN in our setup. This added an extra layer of networking complexity:

DNS Resolution Conflicts: Tailscale’s networking layer can sometimes interfere with Docker’s internal DNS resolution, especially when services are trying to communicate within the Docker network.
IP Resolution: We observed that some services were attempting to resolve hostnames to Tailscale IP addresses (in the 100.x.x.x range) instead of using Docker’s internal networking.
Network Priority: The presence of multiple network interfaces (Docker bridge network and Tailscale) can sometimes lead to confusion in service discovery and routing.

Mitigating VPN Impact

To ensure smooth operation with VPN services like Tailscale:

Network Isolation: Ensure Docker services primarily use Docker’s internal networking for inter-service communication.
DNS Priority: Configure Kong’s DNS resolver explicitly to prefer Docker’s DNS (127.0.0.11).
Network Interface Priority: Be aware of how your system’s network interface priority might affect service resolution.

Configuration Tips

Here are some key configurations that help ensure proper service communication:

# Kong service configuration
kong:
  environment:
    KONG_DNS_RESOLVER: 127.0.0.11  # Docker's internal DNS
    KONG_DNS_ORDER: LAST,A,CNAME
    KONG_DNS_NO_SYNC: "off"
    KONG_DNS_STALE_TTL: 0
    KONG_DNS_NOT_FOUND_TTL: 0
    KONG_DNS_ERROR_TTL: 0
  dns:
    - 127.0.0.11  # Ensure Docker DNS is primary
  dns_search: ""  # Prevent external DNS search

Conclusion

While initial service resolution issues in a self-hosted Supabase setup might seem daunting, they often resolve themselves once the system has had time to properly initialize. Understanding Docker’s networking and DNS resolution system is key to troubleshooting such issues effectively.

Remember that complex systems like Supabase require proper initialization time, and what might appear as a configuration issue could simply be a matter of waiting for all services to fully start up and establish their connections.