What 130K Installs Taught Me About Production Reality

There’s a big difference between writing software and running software that people depend on.

You only start to understand that difference when your code leaves the comfort of development environments and enters production, where real users, real businesses, and real expectations live.

During my time contributing to backend systems powering Wix Proposals, I had the opportunity to work on infrastructure supporting a marketplace application with over 130,000 installs.

That number alone changes how you think about engineering.

Because at that point, you're not just shipping features anymore, you're maintaining trust.

And production reality has a way of teaching lessons you won’t find in tutorials.

Software Becomes Real When Someone Depends on It

In development, failure is cheap.

A bug means restarting a server, fixing a function, maybe rolling back a commit.

In production, failure has a different texture.

A failed request might mean:

• A business owner can’t send a proposal to a client
• A workflow someone depends on suddenly stops working
• A customer support ticket appears somewhere in the world

Suddenly your code isn’t just code anymore.

It’s part of someone’s livelihood.

That realization changes how carefully you think about every decision.

Production Systems Are Mostly About Handling the Unexpected

When engineers imagine production problems, they often think about obvious bugs.

In reality, most issues come from the unexpected edges of systems interacting with other systems.

Things like:

• An API responding slower than usual
• A dependency temporarily failing
• A retry mechanism accidentally multiplying traffic
• A timeout occurring at just the wrong moment

Working with the Wix ecosystem meant interacting with APIs, services, and workflows that had their own behaviors and constraints.

It quickly becomes clear that robust systems aren’t the ones that avoid failure entirely.

They’re the ones designed with the expectation that failure will happen.

The System Is Bigger Than the Codebase

One of the most humbling realizations in production engineering is that the codebase is only a small part of the system.

The real system includes:

• Infrastructure
• Databases
• Background workers
• Queues
• Logs
• Deployment pipelines
• Monitoring tools
• External services

When something goes wrong, the answer is rarely found in a single file.

Instead, you’re tracing the path of a request through a network of moving parts.

Production debugging often feels less like coding and more like investigative work.

You gather clues.

You follow signals.

And eventually, the story reveals itself.

Logs Become Your Best Friend

Before working on production systems at scale, logging can feel like an afterthought.

Something you sprinkle in occasionally.

Once thousands of users rely on your system, logs become essential.

They tell you:

• What happened
• When it happened
• Who it affected
• How the system responded

Without logs, production incidents are mysteries.

With them, they become solvable problems.

A good logging strategy often determines whether a production issue takes minutes or hours to resolve.

Scale Reveals Assumptions

Systems don’t usually break immediately.

They break when scale quietly exposes assumptions you didn’t know you made.

A query that felt fast during development suddenly slows down with real data.

A retry mechanism that seemed harmless starts amplifying traffic during an outage.

A background job queue grows faster than expected.

Scale has a way of turning small decisions into big consequences.

And those moments teach you to think ahead, to ask what happens when something runs not just once, but thousands of times.

Reliability Is an Act of Responsibility

One of the most rewarding aspects of working on systems used by thousands of businesses is the sense of responsibility it creates.

You start thinking less about clever solutions and more about dependable ones.

Reliability becomes the goal.

That means designing systems that are:

• Observable
• Predictable
• Resilient to failure
• Easy to diagnose when things go wrong

Because behind every request is someone expecting the system to work.

The Real Lesson

Working on a platform used by 130,000+ businessess didn’t just teach me technical lessons.

It reshaped how I think about software engineering.

Production systems aren’t defined by the elegance of the code.

They’re defined by how well they hold up under the messy, unpredictable conditions of the real world.

And perhaps the most important shift is this:

You stop asking “Does it work?”

You start asking:

“What happens when it doesn’t?”

That question is where production engineering truly begins.