Form Submission and Multi-Step Flows

The Web Is a Series of Forms#

Search filters, login, "load more" buttons on older sites, public records portals, and every multi-page wizard are form submissions. A large class of scraping targets is not crawled by following links at all; the data only appears after you submit something. Two skills cover most of it: serializing a form exactly as a browser would, and walking a multi-step flow with one persistent session.

Stay on the read-only side. Submitting search and filter forms to retrieve data is normal scraping; submitting forms that create accounts, post content, place orders, or send messages has real-world side effects and usually breaks the site's terms. Test anything that writes data only against your own systems.

How a Browser Serializes a Form#

When a user clicks submit, the browser builds the request from the form's markup using fixed rules. Reproducing those rules is more reliable than guessing field values:

The request itself is controlled by three attributes:

• action: the URL to submit to, relative to the page. An empty or missing action means the current page URL.
• method: GET puts the fields in the query string; POST puts them in the body.
• enctype: for POST, application/x-www-form-urlencoded by default, or multipart/form-data for file uploads.

Let lxml Do the Serialization#

lxml's HTML forms support implements these rules, so you do not have to:

import httpx
from lxml import html as lh

def as_form(pairs):
    # httpx wants a dict for form bodies; lists as values become repeated keys
    out = {}
    for name, value in pairs:
        out.setdefault(name, []).append(value)
    return out

client = httpx.Client(base_url="https://records.example.gov", timeout=20.0,
                      follow_redirects=True)

page = client.get("/search")
tree = lh.fromstring(page.content, base_url=str(page.url))
form = tree.get_element_by_id("search-form")

form.fields["q"] = "building permits"      # set only the fields a user would touch
form.fields["year"] = "2025"               # must be one of the <option> values

values = form.form_values()                # [(name, value), ...] as a browser sends
values.append(("action", "search"))        # the clicked submit button, if it has a name
print(form.method, form.action)
print(values)

if form.method == "POST":
    resp = client.post(form.action, data=as_form(values))
else:
    resp = client.get(form.action, params=values)

POST https://records.example.gov/search/results
[('csrf_token', '9f2c...'), ('q', 'building permits'), ('year', '2025'), ('status', 'any'), ('action', 'search')]

What this handles for you:

• Hidden fields such as CSRF tokens, form ids, and view state are copied from the page. Never invent or reuse values from an earlier session.
• base_url makes form.action an absolute URL, resolving relative and empty actions correctly.
• form_values() returns a list of pairs, not a dict, so repeated names (multi-selects, checkbox groups like tags=a&tags=b) survive. A plain dict(values) silently keeps only the last value. requests accepts the list directly for data=; httpx accepts it for params= but treats a list passed as data= as a raw byte stream, hence the small as_form helper, which builds a dict of lists that httpx encodes as repeated keys.
• Setting a <select> to a value that is not one of its options raises an error. That is a useful check: the server would probably reject it too.

Submit buttons are the one thing form_values() leaves out, because it cannot know which button was clicked. Many forms have several ("Search" and "Export CSV", or "Next" and "Back") and the server branches on which name/value pair arrives. Look at the button in the HTML and append its pair yourself.

Verify Against the Browser#

When a submission does not return what the browser gets, compare the two requests directly. In DevTools, submit the form, click the request in the Network tab, and open the Payload tab to see the parsed form data (or "view source" for the exact encoded body). Then print what your code sends:

req = client.build_request("POST", form.action, data=as_form(values))
print(req.headers["content-type"])
print(req.content.decode())

Differences usually fall into a few categories: a field set by JavaScript at submit time (a timestamp, a hashed value, coordinates of a click), a missing submit button pair, a checkbox that should be unchecked, or a different encoding. If JavaScript intercepts the submit and sends JSON with fetch instead, the form markup is irrelevant; replicate the JSON request as described in JSON payloads.

Multi-Step Wizards Are a State Machine#

A wizard spreads one logical submission across several pages. Each step's response contains the form for the next step, and each form carries state minted by the server: a step token, a CSRF value, sometimes a wizard id in a hidden field or in the session cookie. Two rules make wizards reliable:

1. One client per wizard run. The session cookie often holds the wizard's server-side state. A fresh client starts a new wizard, and step 3 sent from it answers for a flow it never entered.
2. Always parse the next form from the current response. Tokens usually change at every step, and reusing a token from step 1 in step 2 is the most common failure.

def submit(client, response, form_xpath, changes=None, button=None):
    tree = lh.fromstring(response.content, base_url=str(response.url))
    forms = tree.xpath(form_xpath)
    if not forms:
        raise RuntimeError(f"form {form_xpath!r} not found at {response.url}")
    form = forms[0]
    for name, value in (changes or {}).items():
        form.fields[name] = value
    values = form.form_values()
    if button:
        values.append(button)
    if form.method == "POST":
        return client.post(form.action, data=as_form(values))
    return client.get(form.action, params=values)

with httpx.Client(base_url="https://records.example.gov", follow_redirects=True,
                  timeout=20.0) as c:
    r = c.get("/permits/lookup")
    r = submit(c, r, '//form[@id="step-1"]', {"county": "north"}, ("next", "Next"))
    r = submit(c, r, '//form[@id="step-2"]', {"type": "residential"}, ("next", "Next"))
    r = submit(c, r, '//form[@id="step-3"]', {"from": "2025-01-01", "to": "2025-06-30"},
               ("search", "Search"))
    results = lh.fromstring(r.content).xpath('//table[@id="results"]//tr[td]')
    print(len(results), "rows")

The RuntimeError when a form is missing is deliberate. If step 2 returns an error page or bounces you back to step 1, you want the run to stop with the URL in the message, not to continue submitting into the wrong form.

ASP.NET WebForms and Other Postback Frameworks#

Older enterprise and government sites built with ASP.NET WebForms deserve a special mention because they are common scraping targets. Every page is one big form. Hidden fields named __VIEWSTATE, __VIEWSTATEGENERATOR, and __EVENTVALIDATION carry page state, sometimes tens of kilobytes of it. Links and pagers are JavaScript calls like __doPostBack('ctl00$grid','Page$2'), which fill the hidden fields __EVENTTARGET and __EVENTARGUMENT and submit the form.

To follow such a link without a browser, submit the page's form with those two fields set:

tree = lh.fromstring(r.content, base_url=str(r.url))
form = tree.forms[0]
form.fields["__EVENTTARGET"] = "ctl00$grid"
form.fields["__EVENTARGUMENT"] = "Page$2"
r = c.post(form.action, data=as_form(form.form_values()))

The view state changes on every response, so always take it from the latest page. Pagination on these sites is therefore strictly sequential: page 3 can only be requested with the state from page 2. This is also why pagination on postback sites cannot be parallelised within one session.

File Uploads#

Forms with enctype="multipart/form-data" need a multipart body. Pass the regular fields in data and the files in files, keyed by the input's name:

with open("query.csv", "rb") as fh:
    files = {"upload": ("query.csv", fh, "text/csv")}
    r = c.post(form.action, data=as_form(form.form_values()), files=files,
               timeout=120.0)

The library generates the boundary and the Content-Type header; setting Content-Type yourself breaks the boundary. Opening the file in a with block closes the handle even if the request fails, and a longer timeout accounts for the upload time.

Read the Response, Not the Status Code#

Forms rarely fail with an error status. A validation failure usually returns 200 with the same form re-rendered and an error message, and an expired session often redirects to the start of the wizard. After every step, check for evidence of success: the expected next form or results table is present, the URL is the expected one, and no known error markers (.field-error, "session expired", "please correct") appear. Log the step name, URL, and a snippet of any error text. Treat a result page with zero rows as suspicious until you have confirmed the same query in a browser also returns nothing.

Submitting forms repeatedly can be expensive for the server, since each search may run a database query. Apply the same pacing as for any other request, cache result pages, and do not submit the same query twice (see rate limiting and politeness).