WebSockets and Real-Time Data

Data That Arrives, Not Data You Fetch#

Some data has no pages at all. A market ticker, a live auction, a sports score widget, a delivery tracker: the page loads once and then the server keeps pushing updates. Reloading the page every few seconds to catch those changes is slow, heavy and still misses events between reloads. The better approach is to do what the page does and subscribe to the same stream. Most of these streams are WebSockets; some are Server-Sent Events. Consuming them is less about parsing and more about keeping a long-lived connection healthy and never silently losing data.

How a WebSocket Starts#

A WebSocket begins as an ordinary HTTP GET with an upgrade request. If the server agrees, it replies 101 Switching Protocols, and from then on the same TCP (and TLS) connection carries small messages called frames in both directions until one side closes it. The protocol details are in the networking lesson on WebSockets; for scraping, the handshake matters because it carries your access contract.

GET /ws?v=2 HTTP/1.1
Host: stream.example.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
Sec-WebSocket-Version: 13
Origin: https://www.example.com
Cookie: session=...

HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=

Your client library generates the key and checks the accept value. What you must reproduce yourself is everything else: the exact URL and query parameters, any cookie or token, sometimes the Origin header (servers use it to reject cross-site connections), and any Sec-WebSocket-Protocol subprotocol the page requested.

Find the Stream in DevTools#

Open the Network tab, filter by WS (Chrome lists them under "Socket" in some versions), and reload. Click the connection and open the Messages panel. Read it in order, because the first few frames usually define the conversation:

1. Auth frame. Some feeds want a token sent as the first message rather than in the URL.
2. Subscribe frame. Names the channels and symbols, for example {"op": "subscribe", "args": ["trades:ABC"]}. Get it exactly right or you will receive nothing but heartbeats.
3. Heartbeats. Small periodic frames like {"type": "ping"} or a bare 2. Note who sends them and what the reply looks like.
4. Data frames. The payload you want. Note whether they carry a sequence number or message id; you will need it.

If frames appear as binary, the app uses a binary encoding such as MessagePack or Protocol Buffers, or compresses payloads itself. MessagePack decodes with msgpack.unpackb(frame); Protocol Buffers need the message schema, which is usually recoverable from the app's JavaScript as covered in XHR and API reverse engineering.

Frames that look like 0{...}, 40, 42["price",{...}] belong to Socket.IO, which wraps its own protocol inside WebSocket (with numeric prefixes for open, ping, pong and events). Do not hand-roll it; use the python-socketio client, which speaks those prefixes and heartbeats for you.

A Robust Consumer in Python#

The websockets library is the standard asyncio choice. Its connect() can be used as an async iterator that reconnects automatically with backoff whenever the connection drops, which removes most of the hand-written retry logic scrapers used to carry.

import asyncio
import json
from websockets.asyncio.client import connect
from websockets.exceptions import ConnectionClosed

URL = "wss://stream.example.com/ws?v=2"
SUBSCRIBE = {"op": "subscribe", "args": ["trades:ABC", "trades:XYZ"]}

async def receiver(queue: asyncio.Queue) -> None:
    async for ws in connect(
        URL,
        origin="https://www.example.com",   # some servers reject unknown origins
        additional_headers={"Cookie": "session=..."},
        ping_interval=20,        # protocol-level ping every 20 s
        ping_timeout=20,         # drop the connection if no pong in 20 s
        max_size=4 * 1024 * 1024,
    ):
        try:
            await ws.send(json.dumps(SUBSCRIBE))
            async for frame in ws:
                await queue.put(frame)
        except ConnectionClosed as exc:
            print("connection closed:", exc, "- reconnecting")
            continue

Non-obvious lines:

• ping_interval and ping_timeout handle protocol-level keepalive. They detect a dead connection (for example after a laptop sleeps or a NAT entry expires) that would otherwise hang forever waiting for the next frame.
• Many feeds also use application-level heartbeats in JSON. Those are not handled by the library. If the server sends {"type": "ping"} and expects {"type": "pong"}, you must answer it in the loop, or the server closes the connection after its timeout.
• max_size raises the default 1 MiB frame limit. Snapshot frames on busy feeds can exceed it, and the resulting close looks like a random disconnect.
• The subscribe message is sent inside the loop, so it is re-sent after every reconnect. Forgetting that is the classic bug: the consumer reconnects happily and then receives nothing.

Separate Receiving From Writing#

The receive loop must never block. If you write to a database inside it, a slow write stalls reading, the server's send buffer fills, and many servers disconnect slow consumers. Put an asyncio.Queue between the socket and the writer:

from collections import deque

async def writer(queue: asyncio.Queue, path: str = "trades.jsonl") -> None:
    recent_ids = deque(maxlen=50_000)
    recent_set = set()
    count = 0
    with open(path, "a", encoding="utf-8") as f:
        while True:
            frame = await queue.get()
            msg = json.loads(frame)
            if msg.get("type") in ("ping", "heartbeat"):
                continue
            msg_id = msg.get("id")
            if msg_id is not None:
                if msg_id in recent_set:
                    continue                  # redelivered after a reconnect
                if len(recent_ids) == recent_ids.maxlen:
                    recent_set.discard(recent_ids[0])
                recent_ids.append(msg_id)
                recent_set.add(msg_id)
            f.write(json.dumps(msg) + chr(10))
            count += 1
            if count % 200 == 0:
                f.flush()

async def main():
    queue = asyncio.Queue(maxsize=100_000)
    await asyncio.gather(receiver(queue), writer(queue))

asyncio.run(main())

An append-only JSON Lines journal (chr(10) is the newline character) is the safest first landing place: appends are cheap, a crash loses at most the unflushed tail, and you can batch-load the file into your database later, as described in storing data. The bounded deque plus set keeps deduplication memory flat over days of running.

Gaps: The Silent Data Loss#

Reconnecting is easy. Knowing what you missed while disconnected is the hard part. Well-designed feeds give you two tools:

• Sequence numbers. Each message carries seq. If a message arrives with seq other than the previous one plus one, you have a gap.
• Snapshots. A REST endpoint or a snapshot frame returns the full current state (an order book, a scoreboard).

The standard pattern used by trading systems: on connect, subscribe first and buffer deltas, then fetch a snapshot, discard buffered deltas older than the snapshot's sequence, and apply the rest. On any gap, throw away local state and repeat. Without this, a consumer that reconnects cleanly can still be wrong for the rest of the day.

If the feed has no sequence numbers, record connection start and end times in your journal. At minimum, downstream users can see where the holes are, and an incremental scraping job over the regular pages can fill them.

Server-Sent Events and Long Polling#

Not every live page uses WebSockets. Check the Network tab for these too:

SSE is plain HTTP, one direction only, and simple to read:

import httpx

timeout = httpx.Timeout(10.0, read=60.0)
with httpx.stream("GET", "https://example.com/events",
                  headers={"Accept": "text/event-stream"}, timeout=timeout) as resp:
    for line in resp.iter_lines():
        if line.startswith("data:"):
            print(line[5:].strip())
        elif line.startswith("id:"):
            last_event_id = line[3:].strip()

Keep the last id: value. On reconnect, send it back in the Last-Event-ID header and servers that support it will replay what you missed, which solves the gap problem for free.

Capturing Frames From a Real Browser#

When the stream is hard to reproduce (tokens minted by obfuscated JavaScript, a binary protocol you have not decoded yet), let a real browser run the app and record its frames. Playwright exposes WebSocket traffic directly, so there is no need to patch WebSocket in page scripts:

from playwright.sync_api import sync_playwright

frames = []

def on_ws(ws):
    print("socket opened:", ws.url)
    ws.on("framereceived", lambda payload: frames.append(payload))

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.on("websocket", on_ws)
    page.goto("https://www.example.com/live")
    page.wait_for_timeout(30_000)
    browser.close()

print(len(frames), "frames captured")

This is also the quickest way to collect sample frames for building a decoder. Once you understand the protocol, move to a direct client: one browser per stream costs far more memory than one socket. See browser automation for running this reliably.

Be a Good Subscriber#

One connection with the channels you need is the polite shape. Opening dozens of parallel sockets to the same host, or subscribing to every symbol when you need three, is the streaming equivalent of hammering a site, and feeds often cap connections per IP or account. Check the provider's terms; many real-time feeds are licensed data with an official paid API, and that is the right route when it exists.

Practice#

Connect to a public echo or test WebSocket server, send a subscribe-style JSON message, and log replies. Then kill your network for 30 seconds and confirm the consumer reconnects, re-subscribes, and records the disconnect window in its journal.