unable to setup HTTP3/QUIC with nginx 1.25.3 on a reverse proxy

Question

Now that nginx supports QUIC and HTTP/3 protocols starting from 1.25.0, i wanted to give it a try. I compiled the same using boringssl as follows:

wget https://nginx.org/download/nginx-1.25.3.tar.gz

tar xzf nginx-1.25.3.tar.gz

cd /opt
git clone https://boringssl.googlesource.com/boringssl
cd /opt/boringssl
cmake -GNinja -B build
ninja -C build

cd /opt/nginx-1.25.3/
./configure --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock \
    --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp \
    --user=nginx --group=nginx --with-compat --with-file-aio --with-threads \
    --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module \
    --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module \
    --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module \
    --with-http_sub_module --with-http_v2_module --with-http_v3_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module \
    --with-cc-opt='-g -O2 -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC -I../boringssl/include' \
    --with-ld-opt='-Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie -L../boringssl/build/ssl -L../boringssl/build/crypto'
make && make install

this does compile nginx with BoringSSL:

# nginx -V
nginx version: nginx/1.25.3
built by gcc 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04) 
built with OpenSSL 1.1.1 (compatible; BoringSSL) (running with BoringSSL)
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-http_v3_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC -I../boringssl/include' --with-ld-opt='-Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie -L../boringssl/build/ssl -L../boringssl/build/crypto'

Then, i've the following configuration for the quic/http server:

server {
    listen 443 ssl default_server;
    # listen [::]:443 ssl reuseport;
    listen [::]:443 quic default_server reuseport;
    # listen [::]:443 quic reuseport;
    http2 on;
    http3 on;
    http3_hq on;
    quic_retry on;
    
    # SSL configuration
    ssl_certificate         /etc/ssl/lp-playback.studio/fullchain;
    ssl_certificate_key     /etc/ssl/lp-playback.studio/key;
    ssl_protocols       TLSv1.3;
    # ssl_client_certificate /etc/ssl/cloudflare.crt;
    # ssl_verify_client on;
    ssl_early_data on;
    ssl_stapling on;
    ssl_stapling_verify on;

    server_name bgw-2.lp-playback.studio;

    #root /var/non-existing-path-just-testing;
    #index index.html index.htm index.nginx-debian.html;

    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
    proxy_buffering off;
    proxy_http_version 1.1;
    proxy_read_timeout 600s;
    # add_header Alt-Svc 'quic=":$server_port"; ma=2592000; v="46,43,39"';    
    add_header Alt-Svc 'h3=":$server_port"; ma=86400' always;
    add_header quic-status $http3 always;
    add_header x-quic 'h3' always;

    location / {
        proxy_pass http://localhost:8080;
    }
}

and the TLS certs were generated using acme.sh client. However, when trying to use the curl with http3 support to verify, it returns with an http2 connection:

➜ docker run --rm ymuski/curl-http3 curl -I -X GET --http3 'https://bgw-2.lp-playback.studio/'                                                                             (master⇂1|♽1…35⚑2)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0   185    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
HTTP/2 415 
date: Fri, 24 Nov 2023 08:18:50 GMT
content-length: 185
access-control-allow-credentials: true
access-control-allow-headers: *
access-control-allow-methods: GET, POST, OPTIONS, HEAD
access-control-allow-origin: *
access-control-expose-headers: *
access-control-max-age: 600
access-control-request-headers: *
access-control-request-method: GET
cache-control: no-cache, no-store, must-revalidate
expires: 0
pragma: no-cache
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
content-security-policy: object-src 'none'; frame-ancestors 'self'; form-action 'self'; block-all-mixed-content; sandbox allow-forms allow-same-origin allow-scripts allow-popups allow-downloads; base-uri 'self';
alt-svc: h3=":443"; ma=86400
x-quic: h3
access-control-allow-origin: *
timing-allow-origin: *
access-control-allow-methods: GET,POST,OPTIONS,PUT,DELETE
access-control-allow-credentials: true
access-control-allow-headers: Origin,Content-Type,Accept,Authorization,X-Packet-Service-Type,X-Packet-URI,X-User-Address,XD-User-Address,X-Request-From,X-Response-Format
access-control-max-age: 1728000

I am now stumped at why the http3 connection is not being established at all. if i get a verbose curl request, i see the following being logged:

*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
*   Trying 34.18.75.164:443...
* Connected to bgw-2.lp-playback.studio (34.18.75.164) port 443
* ALPN: offers h2,http/1.1
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
.
.

the ALPN offer seems to suggest only h2 and http1.1 are supported? did i miss something when compiling nginx? is the http/server block in nginx-conf misconfigured? does nginx not support h3 for reverse proxy (that'd be weird)?

open to any suggestions for the same.

Answer 1

It was an issue with the nginx configuration, specifically the following:

server {
    listen 443 ssl default_server;
    listen [::]:443 quic default_server reuseport;

the quic protocol was configured to listen on IPv6 ([::]:443)!

Changing that to just the following:

server {
    listen 443 ssl default_server;
    listen 443 quic default_server reuseport;

worked. the domain is working w/ http3 now:

➜ docker run --rm ymuski/curl-http3 curl -I -X GET --http3 'https://bgw-2.lp-playback.studio/'
HTTP/3 415 
server: nginx/1.25.3
date: Mon, 27 Nov 2023 03:42:47 GMT
content-length: 185
access-control-allow-credentials: true
access-control-allow-headers: *
access-control-allow-methods: GET, POST, OPTIONS, HEAD
access-control-allow-origin: *
access-control-expose-headers: *
access-control-max-age: 600
access-control-request-headers: *
access-control-request-method: GET
cache-control: no-cache, no-store, must-revalidate
expires: 0
pragma: no-cache
alt-svc: h3=":443"; ma=86400
quic-status: h3
x-quic: h3