Protocols, Ports and Authentication

A provider's dashboard hands over one line: gate.example-proxy.com:7000:USERNAME:PASSWORD. Whether that line works in curl, in Python or in a browser depends on three things it does not say: the protocol, the meaning of the port and the way the client proves who it is.

HTTP proxies and the CONNECT tunnel

An HTTP proxy handles two kinds of traffic in two different ways. For a plain HTTP page, the client sends the whole request to the proxy, full URL included. The proxy reads it, fetches the page and returns it. The proxy can log, cache or rewrite everything.

For an HTTPS page, the client asks for a tunnel instead. It sends one short request with the CONNECT method, naming a host and a port. The proxy opens a TCP connection to that host and answers 200. From then on it copies bytes in both directions without understanding them.

CONNECT example.com:443 HTTP/1.1
Host: example.com:443
Proxy-Authorization: Basic VVNFUk5BTUU6UEFTU1dPUkQ=

HTTP/1.1 200 Connection established

[TLS handshake between client and example.com]
[encrypted requests and responses]

The first three lines are the only part the proxy can read. The TLS handshake that follows runs between the client and the website, end to end. This is why a proxy whose URL starts with http:// carries HTTPS sites without trouble. The scheme describes how the client talks to the proxy, not what travels inside.

What the proxy can and cannot see

The difference between the two modes decides how much trust a proxy operator needs. On plain HTTP the operator sees everything. On HTTPS the operator sees where the traffic goes and how much of it there is.

Two practical rules follow. Never send a login or an API key to a plain HTTP address through a proxy run by an unknown operator. And be wary of any proxy that asks the client to install a certificate or switch off TLS verification. That request means the proxy intends to decrypt the traffic.

HTTPS proxies, SOCKS5 and the HTTP/3 gap

HTTPS proxies

Look again at the CONNECT exchange. The credentials sit in a Proxy-Authorization header, encoded in Base64. That is an encoding, not encryption. Anyone on the same coffee-shop Wi-Fi can decode it, along with the hostname.

An HTTPS proxy closes that gap. The client first opens a TLS connection to the proxy itself, then sends CONNECT inside it. The proxy URL starts with https://. Not every provider offers a TLS port and not every client supports one, so test before relying on it.

SOCKS5

SOCKS5 works one level lower. The client says "connect to this host on this port" and then sends raw bytes. The proxy never parses HTTP, so it adds no headers and carries any TCP protocol: web pages, mail, a database session. SOCKS5 proxies have three properties worth knowing.

• No encryption of its own. SOCKS5 passwords cross the network in clear text. HTTPS inside the tunnel is still protected.
• UDP on paper. The protocol defines a UDP relay. Many commercial proxies do not switch it on, so ask before building on it.
• A DNS choice. The client can resolve the hostname itself or let the proxy do it. In URL form, socks5:// resolves locally and socks5h:// resolves at the proxy.

For ordinary HTTPS scraping the website cannot tell the two protocols apart. HTTP proxies have the widest tool support and make the sensible default. SOCKS5 earns its place when the traffic is not HTTP or the tool speaks nothing else.

The HTTP/3 caveat

HTTP/3 runs on QUIC, and QUIC runs on UDP. A CONNECT tunnel carries TCP only. A browser behind an HTTP proxy therefore falls back to HTTP/2 or HTTP/1.1. Scripts do the same. Sites accept this, because many office networks block QUIC too. A newer standard for proxying UDP exists, but it should not be assumed in a commercial proxy product.

Two ways to authenticate

A paid proxy must know which customer to bill. It learns that in one of two ways. With username and password, the client sends credentials on every new connection. With IP allowlisting, the customer registers a source IP in the dashboard, and the proxy accepts anything arriving from it.

Credentials travel with the code, so they work from a laptop, a server or a container that started ten seconds ago. They also carry settings: most providers read country and session options from the username, as Chapter 4 shows. The cost is a secret that can leak into logs and repositories.

An allowlist keeps secrets out of the code and suits tools that cannot send proxy credentials. Selenium driving Chrome is the usual example. The weakness is the IP itself. An allowlist breaks the moment the source IP changes, and it lets everyone behind the same office IP spend the plan.

Browsers add one trap. Chrome has long been unable to send a username and password to a SOCKS5 proxy. In a Chromium-based browser, use an HTTP proxy with credentials, or a SOCKS5 port opened by allowlist.

One status code saves hours of confusion. A 407 Proxy Authentication Required comes from the proxy and means wrong or missing credentials. A 403 that arrives inside the tunnel comes from the website and is a different problem.

Anatomy of a proxy URL

Most libraries want the dashboard's four fields rearranged into a URL: http://USERNAME:[email protected]:7000. Each part has one job.

The port deserves a second look. By convention Squid listens on 3128, many HTTP proxies use 8080 and SOCKS uses 1080. Commercial gateways ignore convention and give ports meaning: one for rotating IPs, a range for sticky sessions, another for SOCKS5. Speaking HTTP to a SOCKS port produces a hang or a garbled error, never a clear message. The password causes the other classic failure. An @ or : inside it must be written as %40 or %3A.

Curl is the fastest way to test a URL before it goes into code. The first command below prints the exit IP. The second shows the CONNECT exchange and any 407. ProxyLook's proxy checker also writes a curl test command for any IP and port pasted into it.

# HTTP proxy, HTTPS target. Prints the exit IP as JSON.
curl -x http://USERNAME:[email protected]:7000 \
     --max-time 30 https://httpbin.org/ip

# Verbose mode: shows the CONNECT request and the proxy's reply.
curl -v -x http://USERNAME:[email protected]:7000 \
     --max-time 30 -o /dev/null https://example.com

The SOCKS5 form differs in one flag. Use the provider's SOCKS5 port where it has a separate one.

# SOCKS5 with remote DNS: the proxy resolves the hostname.
curl --socks5-hostname gate.example-proxy.com:7000 \
     --proxy-user USERNAME:PASSWORD \
     --max-time 30 https://httpbin.org/ip

# The same request written as a proxy URL.
curl -x socks5h://USERNAME:[email protected]:7000 \
     --max-time 30 https://httpbin.org/ip

DNS leaks

A scraper in Chicago uses a German SOCKS5 proxy, written as socks5://. Before each request, the machine asks its local DNS server for the target's address. Only then does it hand the proxy an IP to connect to. That lookup is a DNS leak.

The leak has three costs. The local network and its DNS provider learn every hostname the job touches. The answer is tuned for Chicago, so a site that steers visitors by DNS hands a German exit its nearest US servers. And in a browser, a site can compare the resolver's location with the IP's location and notice the gap.

• HTTP and HTTPS proxies: the hostname goes to the proxy, which resolves it. No action needed.
• curl: use --socks5-hostname or a socks5h:// URL, never plain --socks5.
• Python requests: run pip install "requests[socks]" and write the scheme as socks5h://.
• Firefox: tick "Proxy DNS when using SOCKS v5" in the connection settings.
• Verify: capture traffic on port 53 during a run. Target hostnames there mean a leak.