Fix: Geth admin_startWS Socket Leak and WebSocket Port Collision

DevOps engineers and node operators managing private EVM networks, layer-2 sequencers, or automated RPC clusters frequently use Geth’s administrative JSON-RPC methods (admin_startWS and admin_stopWS) to dynamically adjust API permissions and network interfaces on live nodes without triggering a complete process restart.

However, an unhandled concurrency bug in go-ethereum’s internal HTTP/WebSocket server lifecycle leads to severe socket exhaustion:

[ERROR] Failed to start WebSocket endpoint: 
    listen tcp 0.0.0.0:8546: bind: address already in use
    at AdminAPI.StartWS (node/node.go:342:15)
[CRITICAL] RPC Daemon crashed: Secondary WebSocket listener collided with orphaned server instance.

The issue arises because Geth’s single-WebSocket guarantee is enforced strictly per internal httpServer instance rather than globally across the node process. When reconfiguring endpoints, the previous WebSocket listener goroutine is not fully closed before the new listener attempts to bind to port 8546, creating an orphaned socket leak that prevents external dApps and trading bots from establishing real-time subscriptions.

If your node is also experiencing query timeouts and goroutine starvation near the chain tip, consult our analysis on Geth eth_getLogs Goroutine Leak and FilterMaps Livelock.

Architectural Breakdown: Geth Node Server Lifecycle

In node/node.go and rpc/server.go, Geth exposes administrative endpoints to spin up HTTP, IPC, and WebSocket interfaces:

The Core Lifecycle Flaw

1. Instance Isolation: When admin_startWS is called, Geth creates a new httpServer wrapper. If an existing httpServer was already running, Geth attempts an in-flight handoff.
2. Graceful Shutdown Blocking: In Go’s standard net/http package, server.Shutdown(ctx) waits for active WebSocket upgrade connections to finish their payloads. Because Web3 clients maintain persistent, open-ended eth_subscribe streams, Shutdown blocks until its context times out.
3. OS Socket State: If admin_startWS is dispatched before the operating system transitions the TCP socket from FIN_WAIT_2 / TIME_WAIT to CLOSED, the POSIX bind() syscall fails with EADDRINUSE.

Step-by-Step Resolution Protocol

To dynamically restart or reconfigure Geth WebSocket endpoints without risking port collision or orphaned process leaks, execute the following operational sequence.

1. Diagnose Dangling Sockets via Linux CLI

Before invoking admin_startWS, inspect whether any existing processes or threads hold open handles on port 8546:

# Check all processes listening on port 8546
sudo ss -tulpn | grep :8546

# Inspect file descriptor status for the Geth PID
PID=$(pgrep -f "geth.*--mainnet")
sudo lsof -Pan -p $PID -i :8546

If the socket displays CLOSE_WAIT or TIME_WAIT, the previous server instance has not released its kernel descriptor.

2. Implement Safe Graceful Teardown and Verification Script

Use this Python or Bash automation script that wraps the IPC call with an explicit socket verification barrier before triggering admin_startWS:

#!/usr/bin/env python3
# scripts/safe_restart_ws.py
import time
import socket
import json
import urllib.request

GETH_IPC_URL = "http://127.0.0.1:8545"

def send_rpc(method, params=[]):
    payload = json.dumps({"jsonrpc": "2.0", "method": method, "params": params, "id": 1}).encode()
    req = urllib.request.Request(GETH_IPC_URL, data=payload, headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=10) as res:
        return json.loads(res.read().decode())

def is_port_bound(host="127.0.0.1", port=8546):
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
        return s.connect_ex((host, port)) == 0

def safe_rebind_ws(new_host="0.0.0.0", new_port=8546, apis="eth,net,web3"):
    print("[*] Initiating admin_stopWS...")
    send_rpc("admin_stopWS")

    # Poll operating system until port 8546 is genuinely released
    print("[*] Awaiting OS socket release...")
    max_wait = 15
    start = time.time()
    while is_port_bound(new_host if new_host != "0.0.0.0" else "127.0.0.1", new_port):
        if time.time() - start > max_wait:
            raise TimeoutError(f"[-] Port {new_port} remained bound after {max_wait}s!")
        time.sleep(0.5)

    print("[+] Port cleanly unbound. Dispatching admin_startWS...")
    result = send_rpc("admin_startWS", [new_host, new_port, "*", apis])
    
    if result.get("result") is True:
        print(f"[SUCCESS] WebSocket endpoint cleanly bound to {new_host}:{new_port} with APIs: {apis}")
    else:
        print(f"[-] Failed to start WebSocket: {result}")

if __name__ == "__main__":
    safe_rebind_ws()

3. Configure Systemd Socket Reuse Flags

If you manage Geth via systemd, ensure your service unit does not leave abandoned worker threads alive on reload:

# /etc/systemd/system/geth.service
[Unit]
Description=Go Ethereum Node
After=network.target

[Service]
Type=simple
User=ethereum
Restart=on-failure
RestartSec=5s
KillMode=process
KillSignal=SIGTERM
TimeoutStopSec=60s

# Ensure file descriptor limits accommodate high WebSocket connection counts
LimitNOFILE=65536

ExecStart=/usr/local/bin/geth \
  --mainnet \
  --ws \
  --ws.addr "0.0.0.0" \
  --ws.port 8546 \
  --ws.api "eth,net,web3" \
  --ws.origins "*"

[Install]
WantedBy=multi-user.target

Technical Diagnostic Matrix

Frequently Asked Questions

Q: Why doesn’t Geth reuse the port using SO_REUSEPORT?

While SO_REUSEPORT permits multiple processes to bind to the identical port simultaneously, using it in an RPC daemon can cause incoming requests to be load-balanced randomly between the decommissioned old server instance and the newly configured server, leading to inconsistent API responses.

Q: Does this issue affect the standard HTTP RPC port (8545)?

Yes. admin_startHTTP and admin_stopHTTP utilize identical underlying httpServer abstractions. The issue is more pronounced on WebSockets because persistent socket connections hold connections open significantly longer than transient HTTP POST requests.

Q: Can I run WebSockets and HTTP on the same port in Geth?

Yes. Modern Geth versions support unified RPC over a single port by specifying --http and --http.api "eth,net,web3" and using standard HTTP upgrade headers to initiate WebSocket subscriptions over port 8545.