Hashing vs Encoding

Three Tools That Look Alike#

One of the most common mistakes in code review is confusing encoding, encryption and hashing. All three turn data into something that looks like gibberish, but they answer different questions, and using the wrong one is how passwords end up "protected" by Base64.

Encoding: Translation, Not Protection#

Encoding changes the representation of data so a system can carry it. Base64 turns arbitrary bytes into 64 safe text characters so an image can travel inside JSON or an email. There is no key, so anyone can reverse it:

import base64

encoded = base64.b64encode(b"admin:SuperSecret1")
print(encoded)                      # b'YWRtaW46U3VwZXJTZWNyZXQx'
print(base64.b64decode(encoded))    # b'admin:SuperSecret1'

That exact string is what an HTTP Basic Auth header carries, which is why Basic Auth without TLS is effectively plaintext. The same trap appears with JWTs: the header and payload are only base64url-encoded, so anyone holding the token can read the claims. The signature stops tampering, not reading. Rule: never use encoding for security, and never put secrets in a JWT payload.

Encryption: Reversible Secrecy#

Encryption scrambles data with a key so that only key holders can reverse it. Use it when you need the original data back later: a stored API token you must present to a third party, a document, a backup. The encryption lesson covers AES-GCM and key management.

Hashing: One-Way Fingerprints#

A cryptographic hash function takes input of any size and produces a fixed-size digest. Good hash functions have these properties:

• Deterministic: the same input always gives the same digest.
• Preimage resistant: given a digest, you cannot find an input that produces it (other than by guessing).
• Collision resistant: you cannot find two different inputs with the same digest.
• Avalanche effect: changing one bit of input changes about half the output bits.

You can see the avalanche effect directly:

import hashlib

a = hashlib.sha256(b"hello").hexdigest()
b = hashlib.sha256(b"hellp").hexdigest()
print(a)   # 2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824
print(b)

diff = bin(int(a, 16) ^ int(b, 16)).count("1")
print(f"{diff} of 256 bits differ")    # typically around 128

"You cannot reverse a hash" needs one caveat: you cannot compute the input, but you can guess it. If the input space is small, the hash protects nothing. Hashing phone numbers, email addresses or national IDs to "anonymise" them fails because an attacker can hash every possible phone number in minutes and build a lookup table. Low-entropy input needs a keyed hash (HMAC with a secret key) or real pseudonymisation, not a bare hash.

Integrity Checks: Files and Downloads#

The everyday use of SHA-256 is proving a file has not changed. Hash large files in chunks so you never load them fully into memory:

import hashlib

def sha256_file(path: str) -> str:
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(1024 * 1024), b""):
            h.update(chunk)
    return h.hexdigest()

# Python 3.11+ has a shortcut: hashlib.file_digest(f, "sha256").hexdigest()

From a shell, sha256sum installer.tar.gz on Linux, shasum -a 256 on macOS, or Get-FileHash installer.tar.gz in PowerShell produce the same digest to compare against the publisher's value. A checksum alone only proves the file matches the checksum you were given. If an attacker controls the download page, they replace both, which is why package ecosystems sign their checksums (digital signatures) and lock files pin hashes of dependencies (supply chain security).

Which hash? SHA-256, SHA-512, SHA-3 and BLAKE2 are fine for integrity. MD5 and SHA-1 are broken for collision resistance: practical collisions have been demonstrated for both, so never use them for signatures, certificates or anything where an attacker picks the input. They still appear as non-security checksums (Git historically used SHA-1 for object IDs, with collision detection added), and you will meet them in legacy systems.

Keyed Hashes: HMAC#

A plain hash proves data matches a digest, but anyone can compute a new digest for modified data. To prove a message came from someone holding a shared secret, use HMAC. This is how services like Stripe or GitHub sign webhooks:

import hmac, hashlib

SECRET = b"whsec_example_shared_secret"

def sign(body: bytes) -> str:
    return hmac.new(SECRET, body, hashlib.sha256).hexdigest()

def verify(body: bytes, received_sig: str) -> bool:
    expected = sign(body)
    return hmac.compare_digest(expected, received_sig)   # constant-time compare

body = b'{"event":"payment.succeeded","amount":4200}'
sig = sign(body)
print(verify(body, sig))                                   # True
print(verify(body.replace(b"4200", b"1"), sig))           # False

Two details matter. First, do not build your own MAC as sha256(secret + message): SHA-256 and SHA-512 are vulnerable to length-extension attacks, where an attacker who knows one valid digest can compute a valid digest for the message with extra data appended, without knowing the secret. HMAC's construction prevents this. Second, compare signatures with hmac.compare_digest, not ==, which can return early on the first mismatched character and leak timing information.

Real-World Usage: Storing Passwords#

You should never store user passwords in plaintext or with reversible encryption. If the database leaks, so do the passwords (and, because people reuse them, their other accounts). Instead you store a hash. At login you hash what the user typed and compare. You can verify that someone knows the password without ever storing it.

A plain SHA-256 hash is not enough, for two reasons:

1. Precomputed tables. Attackers keep huge maps of common passwords to their SHA-256 digests. A stolen unsalted digest is looked up, not cracked.
2. Speed. A single modern GPU computes billions of SHA-256 hashes per second, so even an unlisted password falls quickly to a dictionary with mutation rules.

The fixes are a salt (a random per-user value stored alongside the hash, so identical passwords produce different digests and every account must be attacked separately) and a deliberately slow, memory-hard algorithm. The standard library gives you PBKDF2:

import hashlib, hmac, os

def hash_password(password: str) -> str:
    salt = os.urandom(16)
    dk = hashlib.pbkdf2_hmac("sha256", password.encode(), salt, 600_000)
    return f"pbkdf2_sha256$600000${salt.hex()}${dk.hex()}"

def check_password(password: str, stored: str) -> bool:
    _, iters, salt_hex, dk_hex = stored.split("$")
    dk = hashlib.pbkdf2_hmac("sha256", password.encode(),
                             bytes.fromhex(salt_hex), int(iters))
    return hmac.compare_digest(dk.hex(), dk_hex)

stored = hash_password("hunter2")
print(check_password("hunter2", stored), check_password("hunter3", stored))  # True False

Storing the algorithm and iteration count inside the string lets you raise the cost later without breaking old hashes. Django stores passwords in almost exactly this format.

For new systems, prefer Argon2id (memory-hard, tunable time, memory and parallelism) via the argon2-cffi package, or bcrypt:

from argon2 import PasswordHasher
from argon2.exceptions import VerifyMismatchError

ph = PasswordHasher()                         # sensible defaults
stored = ph.hash("correct horse battery staple")
print(stored[:30])                            # $argon2id$v=19$m=...,t=...,p=...

try:
    ph.verify(stored, "correct horse battery staple")
    if ph.check_needs_rehash(stored):         # parameters were raised since
        stored = ph.hash("correct horse battery staple")
except VerifyMismatchError:
    print("wrong password")

import bcrypt

hashed = bcrypt.hashpw(b"super_secret_password", bcrypt.gensalt(rounds=12))
print(hashed[:7])                              # b'$2b$12$'
print(bcrypt.checkpw(b"super_secret_password", hashed))   # True

bcrypt only uses the first 72 bytes of the password; recent versions of the Python bcrypt package raise an error for longer inputs rather than silently truncating, so decide how you handle long passphrases. Tune any password hash so one verification takes tens to a few hundred milliseconds on your hardware. The password security lesson covers the login-flow side.

Recognising Hashes in the Wild#

When you find a hash in a database dump, config file or pentest, the format tells you a lot:

Tools like hashid or name-that-hash automate this. If you find 32 or 64 hex characters in a users table with no salt column, that is a finding worth reporting on its own.

Choosing the Right Tool#

• Need the data back later? Encrypt it.
• Need to send binary through a text channel? Encode it (and add encryption if it is sensitive).
• Need to detect changes to a file or record? SHA-256 or BLAKE2.
• Need to prove a message came from a shared-secret holder? HMAC-SHA256.
• Need to prove who signed it, publicly verifiable? A digital signature.
• Storing a password? Argon2id, scrypt, bcrypt, or high-iteration PBKDF2. Never MD5, SHA-1 or unsalted SHA-2.