Posts on Company Engineering

Observability Patterns for Distributed Systems

Wed, 10 Jun 2026 00:00:00 +0000

When a request crosses a dozen services, “check the logs” stops being useful advice. Observability is what lets us ask new questions about a running system without shipping new code to answer them. These are the patterns we rely on.

The three signals, and what each is for

We treat metrics, logs, and traces as complementary rather than interchangeable:

Metrics answer how much and how often — and they’re cheap enough to keep for everything. They’re how we notice a problem.
Traces answer where — they show the path of a single request across services and where the time went. They’re how we localize a problem.
Logs answer why — the detailed, contextual record of what a specific component did. They’re how we explain a problem.

You reach for them in that order: a metric alerts, a trace narrows it to a service, logs explain the failure.

What We Learned From Running Background Workers in Production

Fri, 05 Jun 2026 00:00:00 +0000

Background workers are where a lot of our most important work happens — sending notifications, generating exports, syncing data, billing. They’re also where the most surprising production incidents start. Here’s what running them at scale has taught us.

Jobs are not functions

A function call either returns or throws. A background job can also be retried, duplicated, delayed for hours, killed mid-execution, or run on code that has since been deployed over. Designing jobs means designing for all of those states, not just success and failure.

Designing Reliable Data Synchronization at Scale

Mon, 01 Jun 2026 00:00:00 +0000

Keeping data consistent across systems we don’t control is one of the hardest problems we work on. Every external integration is a small distributed system: networks fail, third-party APIs rate-limit us, and the “source of truth” is often whichever side wrote last. This post covers the patterns we lean on to make synchronization predictable rather than heroic.

The shape of the problem

A sync flow looks deceptively simple: read from a source, transform, write to a destination. In practice each step can fail independently, and the same change can arrive more than once. We design every flow around three assumptions: