Python Roadmap

Advanced Python programming topics.

29 stages · 116 topics 0% done

Python

✓ What Metaclasses Are

✓ __init__ vs __new__ in Metaclasses

✓ Foundations

✓ Metaclass Method Injection

✓ Metaclass Attribute Validation

✓ Core Concepts

✓ ORM-Style Metaclass Usage

✓ Singleton Metaclass

✓ Abstract Base Class Metaclass

✓ Metaclass Conflict Resolution

✓ Intermediate Skills

✓ Metaclass __prepare__

✓ Real-World: Django Models

✓ Advanced Topics

✓ Descriptor Protocol Basics

✓ __set_name__ Protocol

✓ Property Descriptor Implementation

✓ Data vs Non-Data Descriptors

✓ Expert Level

✓ Slot Descriptors

✓ Validation with Descriptors

✓ Mastery

✓ Lazy Property Descriptor

✓ Classmethod & Staticmethod Descriptors

✓ Async/Await Under the Hood

✓ Event Loop Implementations

✓ Ecosystem

✓ Task Groups & Structured Concurrency

✓ Async Context Managers

✓ Architecture

✓ Async Generators

✓ Cancellation & Shield

✓ Queue Patterns & Backpressure

✓ Writing Custom Awaitables

✓ Optimization

✓ Threading vs Multiprocessing

✓ The GIL in Depth

✓ Deployment

✓ Thread Pools & Executors

✓ Multiprocessing Shared Memory

✓ Process Pools & Map-Reduce

✓ Concurrent Futures Patterns

✓ Phase 11

✓ Synchronization Primitives

✓ Thread-Safe Queues

✓ Phase 12

✓ Python Object Model Internals

✓ Reference Counting & Cycles

✓ GC Generational Collection

✓ Memory Profiling with tracemalloc

✓ Phase 13

✓ Small Object Optimization (obmalloc)

✓ Memory Views & Buffer Protocol

✓ Phase 14

✓ Weak References & WeakValueDictionary

✓ allocators & PyMem API

✓ PyPy vs CPython Memory Differences

✓ Python C API Basics

✓ Phase 15

✓ Creating Extension Modules

✓ PyObject Reference Management

✓ Phase 16

✓ Exception Handling in C

✓ cffi Library

✓ ctypes Deep Dive

✓ Cython Overview

✓ Phase 17

✓ PyBind11 for C++ Extensions

✓ Type Hints & PEP 484

✓ Phase 18

✓ Generic Types & TypeVar

✓ Protocol Structural Subtyping

✓ TypedDict for Structured Dicts

✓ Literal & Final Types

✓ Phase 19

✓ Type Guards & Type Narrowing

✓ Overload Decorator Patterns

✓ Phase 20

✓ mypy Configuration & Strict Mode

✓ Domain-Specific Languages in Python

✓ AST Manipulation

✓ Custom Parsing with Lark

✓ Phase 21

✓ Code Generation with Jinja2

✓ Operator Overloading for DSLs

✓ Phase 22

✓ Context Manager DSL Pattern

✓ Fluent Interface Design

✓ Profiling with cProfile & py-spy

✓ JIT Compilation with Numba

✓ Phase 23

✓ Cython for Performance

✓ __slots__ for Memory Optimization

✓ Phase 24

✓ Generator vs List Performance

✓ String Concatenation Patterns

✓ Built-in Function Optimization

✓ Socket Programming & Non-blocking I/O

✓ Phase 25

✓ asyncio Streams Protocol

✓ HTTP/2 & HTTP/3 with aiohttp

✓ Phase 26

✓ WebSocket Server Implementation

✓ gRPC Client/Server in Python

✓ DNS Resolution & Custom Resolvers

✓ ZeroMQ Patterns

✓ Phase 27

✓ pytest Fixtures & Conftest

✓ Mocking with unittest.mock

✓ Phase 28

✓ Property-Based Testing with Hypothesis

✓ Fakes vs Mocks vs Stubs

✓ Test Coverage & Mutation Testing

✓ Pytest Plugins Architecture

✓ Phase 29

✓ Integration Testing Patterns

Keep learning

Algorithms JavaScript C++ Rust Go Java All roadmaps ->

Text version of this roadmap

1. Foundations

What Metaclasses Are: A metaclass is the class of a class. When you define a class, Python uses `type` as the default metaclass to create it. By subclassing `type`, you can intercept class creation to register subclasses, enforce interfaces, or auto-generate methods. The metaclass's `__new__` receives the metaclass itself, the class name, bases tuple, and namespace dict.
__init__ vs __new__ in Metaclasses: In metaclasses, `__new__` creates the class object and `__init__` initializes it. `__new__` receives `(mcs, name, bases, ns)` and must return a class. `__init__` receives the created class and can perform post-creation setup. Use `__new__` for attribute injection and `__init__` for registration.
Metaclass Method Injection: Metaclasses can inject methods into the class namespace before the class is created. This is useful for adding utility methods, implementing mixins, or generating proxy methods. Methods can be injected as regular functions, static methods, or class methods.

2. Core Concepts

Metaclass Attribute Validation: Metaclasses can enforce constraints on class definitions at creation time. You can require certain attributes, validate method signatures, or ensure consistent naming conventions. This catches errors early and enforces architectural rules across a codebase.
ORM-Style Metaclass Usage: ORMs like SQLAlchemy and Django use metaclasses to map class attributes to database columns. The metaclass intercepts attribute definitions, wraps them in descriptor objects, and builds internal metadata about the model schema. This is a practical, real-world metaclass application.
Singleton Metaclass: The Singleton pattern via metaclass controls `__call__` to return the same instance. The metaclass's `__call__` method intercepts `ClassName()`, checks an internal cache, creates the instance on first call, and returns the cached instance thereafter.

3. Intermediate Skills

Abstract Base Class Metaclass: Python's `ABCMeta` (from `abc` module) is a metaclass that supports abstract methods. It prevents instantiation of classes that don't implement all abstract methods. Understanding how `ABCMeta` works internally shows how metaclasses integrate with Python's object system.
Metaclass Conflict Resolution: When a class has multiple metaclass parents, Python must resolve conflicts. The metaclass must be a subtype of all parent metaclasses. A compatible metaclass inheriting from all conflicting metaclasses resolves the issue.
Metaclass __prepare__: `__prepare__` returns the namespace object for class body execution. It can return a custom dict-like object that tracks attribute definition order, rejects duplicates, or transforms names. This is how Python 3 preserves class attribute definition order.

4. Advanced Topics

Real-World: Django Models: Django's model system uses `ModelBase` metaclass to convert class attributes into database field descriptors. It creates `Meta` options objects, sets up `_meta` API, registers signals, and builds `__init__`. This is one of the most widespread metaclass uses.
Descriptor Protocol Basics: A descriptor is an object whose class implements `__get__`, `__set__`, or `__delete__`. When a class attribute is a descriptor, Python intercepts dotted access and delegates to the descriptor methods. Data descriptors (with `__set__`) take priority over instance dict.
__set_name__ Protocol: Python 3.6 introduced `__set_name__` on descriptors. When a class is created, Python calls `__set_name__` on all descriptors in the class body, passing the owning class and the attribute name. This eliminates the need for `__init_subclass__` hacks.

5. Expert Level

Property Descriptor Implementation: The `@property` decorator creates a descriptor. The `property` type has `__get__`, `__set__`, and `__delete__` methods that call the user-supplied fget, fset, fdel functions. Understanding this reveals how computed attributes work.
Data vs Non-Data Descriptors: Data descriptors define both `__get__` and `__set__`; they override instance dictionaries. Non-data descriptors only define `__get__`. Instance attributes shadow non-data descriptors but not data descriptors. This affects how `property`, `staticmethod`, `classmethod` behave.
Slot Descriptors: `__slots__` creates descriptors for each slot name. These are data descriptors stored in per-instance fixed arrays, reducing memory overhead. Slotted attributes are faster than dict lookup but break `__dict__` by default.

6. Mastery

Validation with Descriptors: Descriptors excel at input validation. By centralizing validation logic in a descriptor, you avoid repetition across properties. The descriptor stores validated values in the instance dict and can check types, ranges, or arbitrary constraints on assignment.
Lazy Property Descriptor: Lazy evaluation via descriptors defers computation until first access and caches the result. The descriptor replaces itself with the computed value in the instance dict. This pattern is common for expensive properties like configuration loading or database lookups.
Classmethod & Staticmethod Descriptors: `@classmethod` and `@staticmethod` are built-in descriptors. `classmethod.__get__` binds the method to the class, passing `cls` as the first argument. `staticmethod.__get__` returns the underlying function unchanged. Both use the descriptor protocol.

7. Ecosystem

Async/Await Under the Hood: The `async` and `await` keywords compile to coroutine objects that yield control at suspension points. A coroutine is a generator that yields `Future` or `Task` objects. The event loop drives execution by calling `coro.send(None)` and catching `StopIteration` for results.
Event Loop Implementations: Python's `asyncio` event loop uses I/O multiplexing via the `selectors` module. It maintains a queue of callbacks and scheduled tasks. The loop runs callbacks, polls file descriptors, and schedules awaited coroutines. `asyncio.run()` creates a new loop, runs the coroutine, and closes it.
Task Groups & Structured Concurrency: Python 3.11 introduced `TaskGroup` for structured concurrency. Tasks spawned in a group are cancelled when the group exits, ensuring no orphaned tasks. If any task raises an exception, all sibling tasks are cancelled and the exception is propagated at the `async with` boundary.

8. Architecture

Async Context Managers: Async context managers use `__aenter__` and `__aexit__`. They're essential for managing async resources like database connections, file handles, and network streams. The `async with` statement awaits entry and exit, ensuring proper cleanup even on exceptions.
Async Generators: Async generators produce async iterables. They implement `__aiter__` and `__anext__` and can be consumed with `async for`. They support `asend()`, `athrow()`, and `aclose()` for advanced coroutine communication.
Cancellation & Shield: Cancellation in asyncio is cooperative. `asyncio.CancelledError` is raised at await points. `asyncio.shield()` protects an awaitable from cancellation, useful for critical cleanup. Proper handling requires try/finally or try/except CancelledError.

9. Optimization

Queue Patterns & Backpressure: `asyncio.Queue` coordinates producers and consumers. Bounded queues provide backpressure: `put()` blocks when full. Common patterns include fan-out (many consumers), batching, and priority queues. Queues are the backbone of most async data pipelines.
Writing Custom Awaitables: Any object implementing `__await__` is awaitable. `__await__` returns an iterator that yields values until the result is ready. This enables custom awaitable types beyond built-in coroutines, like delayed values or retry wrappers.
Threading vs Multiprocessing: Threading uses OS threads sharing memory; multiprocessing uses separate processes with independent memory. Threads are limited by the GIL in CPython for CPU-bound work. Multiprocessing bypasses the GIL but has higher overhead and IPC complexity.

10. Deployment

The GIL in Depth: The Global Interpreter Lock prevents multiple threads from executing Python bytecode simultaneously. The GIL is released during I/O operations and in C extension code. PEP 703 (nogil) proposes making the GIL optional for CPU-bound parallelism.
Thread Pools & Executors: `concurrent.futures.ThreadPoolExecutor` manages a pool of worker threads. Tasks are submitted as callables and return `Future` objects. The executor reuses threads to avoid creation overhead. Use `max_workers` to control concurrency level.
Multiprocessing Shared Memory: The `multiprocessing` module supports shared memory via `Value`, `Array`, and `shared_memory` (Python 3.8+). Synchronization uses locks, semaphores, or conditions. `SharedMemory` provides direct memory mapping for high-performance data sharing.

11. Phase 11

Process Pools & Map-Reduce: `multiprocessing.Pool` provides `map`, `starmap`, `apply`, and `imap` for parallel work distribution. Chunking controls how many items each worker processes at once. Ordered results are guaranteed with `map`; unordered reduces latency.
Concurrent Futures Patterns: `concurrent.futures` provides a high-level interface for threads and processes. Key patterns: `wait()` with FIRST_COMPLETED for early-exit, `map()` for bulk tasks, and `as_completed()` for streaming results. ProcessPoolExecutor requires picklable arguments.
Synchronization Primitives: Python provides `Lock`, `RLock`, `Semaphore`, `Condition`, `Event`, and `Barrier` for threading synchronization. `RLock` allows re-entrant acquires by the same thread. `Condition` combines a lock with wait/notify. `Barrier` synchronizes N threads at a rendezvous point.

12. Phase 12

Thread-Safe Queues: `queue.Queue` provides thread-safe FIFO, LIFO, and priority queues. Internally it uses `threading.Lock` and `threading.Condition` for blocking `put`/`get` with timeout. `task_done()` and `join()` coordinate worker completion.
Python Object Model Internals: In Python_All, python object model internals is a key concept within memory. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.
Reference Counting & Cycles: In Python_All, reference counting & cycles is a key concept within memory. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.

13. Phase 13

GC Generational Collection: In Python_All, gc generational collection is a key concept within memory. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.
Memory Profiling with tracemalloc: In Python_All, memory profiling with tracemalloc is a key concept within memory. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.
Small Object Optimization (obmalloc): In Python_All, small object optimization (obmalloc) is a key concept within memory. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.

14. Phase 14

Memory Views & Buffer Protocol: In Python_All, memory views & buffer protocol is a key concept within memory. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.
Weak References & WeakValueDictionary: In Python_All, weak references & weakvaluedictionary is a key concept within memory. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.
allocators & PyMem API: In Python_All, allocators & pymem api is a key concept within memory. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.

15. Phase 15

PyPy vs CPython Memory Differences: In Python_All, pypy vs cpython memory differences is a key concept within memory. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.
Python C API Basics: In Python_All, python c api basics is a key concept within c-extensions. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.
Creating Extension Modules: In Python_All, creating extension modules is a key concept within c-extensions. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.

16. Phase 16

PyObject Reference Management: In Python_All, pyobject reference management is a key concept within c-extensions. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.
Exception Handling in C: In Python_All, exception handling in c is a key concept within c-extensions. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.
cffi Library: In Python_All, cffi library is a key concept within c-extensions. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.

17. Phase 17

ctypes Deep Dive: In Python_All, ctypes deep dive is a key concept within c-extensions. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.
Cython Overview: In Python_All, cython overview is a key concept within c-extensions. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.
PyBind11 for C++ Extensions: In Python_All, pybind11 for c++ extensions is a key concept within c-extensions. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.

18. Phase 18

Type Hints & PEP 484: In Python_All, type hints & pep 484 is a key concept within type-system. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.
Generic Types & TypeVar: In Python_All, generic types & typevar is a key concept within type-system. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.
Protocol Structural Subtyping: In Python_All, protocol structural subtyping is a key concept within type-system. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.

19. Phase 19

TypedDict for Structured Dicts: In Python_All, typeddict for structured dicts is a key concept within type-system. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.
Literal & Final Types: In Python_All, literal & final types is a key concept within type-system. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.
Type Guards & Type Narrowing: In Python_All, type guards & type narrowing is a key concept within type-system. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.

20. Phase 20

Overload Decorator Patterns: In Python_All, overload decorator patterns is a key concept within type-system. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.
mypy Configuration & Strict Mode: In Python_All, mypy configuration & strict mode is a key concept within type-system. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.
Domain-Specific Languages in Python: In Python_All, domain-specific languages in python is a key concept within dsl. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.

21. Phase 21

AST Manipulation: In Python_All, ast manipulation is a key concept within dsl. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.
Custom Parsing with Lark: In Python_All, custom parsing with lark is a key concept within dsl. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.
Code Generation with Jinja2: In Python_All, code generation with jinja2 is a key concept within dsl. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.

22. Phase 22

Operator Overloading for DSLs: In Python_All, operator overloading for dsls is a key concept within dsl. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.
Context Manager DSL Pattern: In Python_All, context manager dsl pattern is a key concept within dsl. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.
Fluent Interface Design: In Python_All, fluent interface design is a key concept within dsl. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.

23. Phase 23

Profiling with cProfile & py-spy: In Python_All, profiling with cprofile & py-spy is a key concept within optimization. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.
JIT Compilation with Numba: In Python_All, jit compilation with numba is a key concept within optimization. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.
Cython for Performance: In Python_All, cython for performance is a key concept within optimization. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.

24. Phase 24

__slots__ for Memory Optimization: In Python_All, __slots__ for memory optimization is a key concept within optimization. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.
Generator vs List Performance: In Python_All, generator vs list performance is a key concept within optimization. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.
String Concatenation Patterns: In Python_All, string concatenation patterns is a key concept within optimization. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.

25. Phase 25

Built-in Function Optimization: In Python_All, built-in function optimization is a key concept within optimization. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.
Socket Programming & Non-blocking I/O: In Python_All, socket programming & non-blocking i/o is a key concept within networking. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.
asyncio Streams Protocol: In Python_All, asyncio streams protocol is a key concept within networking. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.

26. Phase 26

HTTP/2 & HTTP/3 with aiohttp: In Python_All, http/2 & http/3 with aiohttp is a key concept within networking. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.
WebSocket Server Implementation: In Python_All, websocket server implementation is a key concept within networking. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.
gRPC Client/Server in Python: In Python_All, grpc client/server in python is a key concept within networking. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.

27. Phase 27

DNS Resolution & Custom Resolvers: In Python_All, dns resolution & custom resolvers is a key concept within networking. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.
ZeroMQ Patterns: In Python_All, zeromq patterns is a key concept within networking. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.
pytest Fixtures & Conftest: In Python_All, pytest fixtures & conftest is a key concept within testing. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.

28. Phase 28

Mocking with unittest.mock: In Python_All, mocking with unittest.mock is a key concept within testing. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.
Property-Based Testing with Hypothesis: In Python_All, property-based testing with hypothesis is a key concept within testing. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.
Fakes vs Mocks vs Stubs: In Python_All, fakes vs mocks vs stubs is a key concept within testing. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.

29. Phase 29

Test Coverage & Mutation Testing: In Python_All, test coverage & mutation testing is a key concept within testing. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.
Pytest Plugins Architecture: In Python_All, pytest plugins architecture is a key concept within testing. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.
Integration Testing Patterns: In Python_All, integration testing patterns is a key concept within testing. Understanding this enables developers to write more expressive and efficient code. Mastery of this topic distinguishes advanced Python_All programmers from intermediate ones.