Company Engineering

Observability Patterns for Distributed Systems

Wed, 10 Jun 2026 00:00:00 +0000

When a request crosses a dozen services, “check the logs” stops being useful advice. Observability is what lets us ask new questions about a running system without shipping new code to answer them. These are the patterns we rely on.

The three signals, and what each is for

We treat metrics, logs, and traces as complementary rather than interchangeable:

Metrics answer how much and how often — and they’re cheap enough to keep for everything. They’re how we notice a problem.
Traces answer where — they show the path of a single request across services and where the time went. They’re how we localize a problem.
Logs answer why — the detailed, contextual record of what a specific component did. They’re how we explain a problem.

You reach for them in that order: a metric alerts, a trace narrows it to a service, logs explain the failure.

What We Learned From Running Background Workers in Production

Fri, 05 Jun 2026 00:00:00 +0000

Background workers are where a lot of our most important work happens — sending notifications, generating exports, syncing data, billing. They’re also where the most surprising production incidents start. Here’s what running them at scale has taught us.

Jobs are not functions

A function call either returns or throws. A background job can also be retried, duplicated, delayed for hours, killed mid-execution, or run on code that has since been deployed over. Designing jobs means designing for all of those states, not just success and failure.

Designing Reliable Data Synchronization at Scale

Mon, 01 Jun 2026 00:00:00 +0000

Keeping data consistent across systems we don’t control is one of the hardest problems we work on. Every external integration is a small distributed system: networks fail, third-party APIs rate-limit us, and the “source of truth” is often whichever side wrote last. This post covers the patterns we lean on to make synchronization predictable rather than heroic.

The shape of the problem

A sync flow looks deceptively simple: read from a source, transform, write to a destination. In practice each step can fail independently, and the same change can arrive more than once. We design every flow around three assumptions:

About this blog

Mon, 01 Jan 0001 00:00:00 +0000

Company Engineering is a space for the people who build and operate our software to share what they learn. It is written by engineers, for engineers.

We use this blog to talk about the work behind the product — the parts that rarely make it into a changelog but shape how reliable, fast, and pleasant the system is to build on.

What you’ll find here

Technical lessons. Honest write-ups of what worked, what didn’t, and what we’d do differently next time.
Architecture decisions. The reasoning behind how we structure services, data flows, and integrations — including the trade-offs we accepted.
Reliability practices. How we think about failure, retries, idempotency, rollouts, and operating systems under real-world load.
Tooling. The internal tools and developer-experience investments that make our teams faster and safer.
Product engineering stories. How engineering decisions connect back to customer impact.

Why we write

Writing forces clarity. Explaining a system to someone outside the team exposes the assumptions we didn’t know we were making, and turns hard-won context into something the whole organization can build on. It also helps us hire and work with people who care about the same things we do.

Authors

Mon, 01 Jan 0001 00:00:00 +0000

The teams who write here build and operate our software every day. Each post is attributed to the team closest to the work.