Summarizer

> At scale, even your bugs have users. First place I worked right out of college had a big training seminar for new hires. One day we were told the story of how they’d improved load times from around 5min to 30seconds, this improvement was in the mid 90s. The negative responses from clients were instant. The load time improvements had destroyed their company culture. Instead of everyone coming into the office, turning on their computers, and spending the next 10min chatting and drinking coffee the software was ready before they’d even stood up from their desk! The moral of the story, and the quote, isn’t that you shouldn’t improve things. Instead it’s a reminder that the software you’re building doesn’t exist in a PRD or a test suite. It’s a system that people will interact with out there in the world. Habits with form, workarounds will be developed, bugs will be leaned for actual use cases. This makes it critically important that you, the software engineer, understand the purpose and real world usage of your software. Your job isn’t to complete tickets that fulfill a list of asks from your product manager. Your job is to build software that solves users problems.

This was well talked about in Hyrums Law, which came from a Googler as well. https://www.hyrumslaw.com/ > With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody.

For close to a decade my business revolved around a common bug in both Microsoft and Netscape, the dominant browsers of the day. With every release we were thinking 'this time they'll fix it' and that would have caused us some serious headaches. But they never did!

I was curious what the commenter's business was, and found this post about HTTP protocol latency: https://jacquesmattheij.com/the-several-million-dollar-bug/

> The negative responses from clients were instant. Back when I was designing TTL circuits, the TTL specifications gave a min and max time for the delay between the inputs and the outputs. I was instructed to never rely on the min delay, as the chips kept getting faster and the older, slower replacement parts will not be available anymore. The IBM PC was frustrating to many hardware engineers, as too much software relied on timing loops and delays in the original design, which made it difficult to make the hardware go faster.

On older cars, like my '72 Dodge, the system voltage varied between 12 and 18 volts. But the dash instruments needed 5 volts. This was achieved with a clever "buzzer" circuit using an electromagnet and contacts. The circuit would open when it was above 5 volts and close when it was below. This created 5V, but was a noisy 5V. Many people decided to improve this with a semiconductor voltage regulator, which would nail the output at 5V. But the instruments wouldn't work! The problem turned out to be the instruments relied on the noisy 5V to "unstick" the needles on the instruments. So the electronics guys had to add a "noise" circuit to the voltage regulator circuit. P.S. Watch an old aviation movie, where the pilot getting ready to fly would tap the instruments to unstick them.

Ah, the Turbo Button! I think by the time I got my first IBM PC the button no longer did anything, but it was still there on the case for some reason. I remember pushing it repeatedly, puzzled that nothing went faster.

Absolutely - one of my favorite bug with users was an application we had made in which the loading of a filtered view was so slow, that results would come in one-at-a-time, such that clicking 'select all' would only select those ones. When this was removed, users complained until we added shift-clicking to select groups of items

So what is the correct solution to that specific problem then, adjust loading time per customer?

Probably just let them vent until they adjust their habits and just chat with their co-workers, without the need to use this as an excuse. Then, they can enjoy the fast loading times :)

Craziest I got was users complaining their laptops were getting too hot / too noisey because I correctly parallelized a task and it became too efficient . They liked the speed but hated the fans going on at full speed and the CPU (and hence the whole laptop) getting really warn (talking circa 2010). So I had to artificially slow down processing a bit as to not make the fans go brrrrr and CPU go too hot.

I optimised out some redundant processes on a unix system and sped up boot time. But I had to release dummy processes which just printed out the same logs, as management didn't want to retrain operators or reprint the documentation. Mid 90s. All training and operations manuals were hard copy.

https://xkcd.com/1172/

Hyrum's Law: https://www.hyrumslaw.com/

That is https://xkcd.com/1172/

> At scale, even your bugs have users Also known as ossification. It is a term most often heard in the context of network protocols, but it applies more generally to every system where users depend on unspecified behaviors and even bugs. Reading about HTTP/3 and QUIC is interesting in that aspect. I first didn't understand the insistance on encryption. Turns out it is not just security and privacy, but by encrypting everything that is not strictly necessary for proper transport, you make it impossible for any "middlebox" to make assumptions they shouldn't make. I think similar approaches can be used by APIs. Never expose more than what is specified, treat the ability to access internal state as a bug, not because it is secret, but because if users start relying on it, internal changes that shouldn't break anything will.

Similar to Hyrum 's Law for APIs

>What I learned was: >• Almost nobody else in engineering did this. >• I was considered weird for doing it. >• It was viewed negatively by managers and promo committees. >• An engineer talking directly to users was considered especially weird and problematic. >• The products did always have serious bugs that had escaped QA and monitoring Sincerely, thank you for confirming my anecdotal but long-standing observations. My go-to joke about this is that Google employees are officially banned from even visiting user forums. Because otherwise, there is no other logical explanation why there are 10+ year old threads where users are reporting the same issue over and over again, etc. Good engineering in big tech companies (I work for one, too) has evaporated and turned into Promotion Driven Development. In my case: write shitty code, cut corners, accumulate tech debt, ship fast, get promo, move on.

I have an issue with the first point as well, but differently. Having worked on a user-facing product with millions of users, the challenge was not finding user problems, but finding frequent user problems. In a sufficiently complex product there are thousands of different issues that users encounter. But it's non-trivial to know what to prioritize.

> wonder why Google UX always sucked so much and in particularly in the recent years seem to be getting even worse UX? Google doesn't even bother helping folks locked out of their Gmail accounts. For people who use Android (some 3bn), that's like a digital death sentence, with real-world consequences. It is almost comical that anyone would think Google is customer-focused, but might if they were being paid handsomely to think otherwise, all the while drinking a lot of kool-aid. https://news.ycombinator.com/item?id=36024754 The top comment there is from a Xoogler which sums it up nicely: The thing is that at scale your edge cases are still millions of people. Companies love the benefits that come from scale, like having a billion people use their service, but they never seem to be capable of handling the other parts that come with it :( Google rakes in $100bn a quarter; that's $1bn every day.

And how are they supposed to do it if users did not add proper 2FA (and backup those recovery keys)? Even banks are struggling to authenticate folks. For a longtime in EU people with 3rd world passports cannot create accounts easily. Google cannot connect identity of a person to email address easily. Or they need to create CS - that will authenticate passports? And hundreds of countries, stolen IDs? Nay. > The thing is that at scale your edge cases are still millions of people > never seem to be capable of handling the other parts that come with it Same thing with govts. If you go to driver license. passport or any govt office then there will one person with some strange issue.

GCP has had some multi-region outages

> Bias towards action. Ship. …The quest for perfection is paralyzing. Unfortunately for users this is more often used as an excuse to ship buggy / badly done software.

I like this one > At scale, even your bugs have users. Something I discovered the hard way over many years of maintaining rclone. Fixing a bug has consequences and there are sometimes users depending on that bug! xkcd: https://xkcd.com/1172/

I know this as Hyrum's Law (which also comes from a Googler): "With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody." https://www.hyrumslaw.com/