Data drift in production: knowing when a model has aged
Models don't fail with errors: they lose accuracy silently as data changes. How to monitor input and prediction drift with PSI, and decide when to retrain.
Technical explanations of the problems we meet while building complex systems. No shortcuts, with code where it helps.
Models don't fail with errors: they lose accuracy silently as data changes. How to monitor input and prediction drift with PSI, and decide when to retrain.
Data leakage makes a model shine in validation and fail in production. The most common kinds, how to detect them, and how to validate on time-ordered data.
Binary floating point can't represent 0.10 exactly. How to represent money, choose rounding rules and split amounts without losing or creating cents.
Most RAG errors start in retrieval, not in the model. How to build an evaluation set and measure recall, MRR and faithfulness in CI.
Indexers that treat recent blocks as final accumulate orphaned data. How to detect reorganizations, roll back safely and expose confirmation levels to consumers.
Retries are unavoidable on real networks. How to implement idempotency keys in an API: storage, concurrency, payload validation and expiry.
How to design risk controls that run before every order: which checks to include, in what order, how to keep state without locks, and why to fail closed.
An AI agent is only as safe as the tools it can call. How to scope tools, propagate user permissions and require approval before real-world effects.
Upgradeable contracts solve one problem and create others. How proxies work, the most common storage mistakes and the governance an upgrade requires.
Microservices solve scaling and organizational problems, and create network and consistency ones. How to build a modular monolith that can be split later.
Writing to a database and publishing to a broker fail independently. How the outbox pattern fixes this, with a relay, ordering and idempotent consumers.
A seemingly simple migration can lock an entire table in production. How to avoid long locks, use expand/contract, and add indexes and constraints safely.