[WDT] NATIONAL BASTILLE DAY/FIFA CUP FINALS [NSFW]

I'm here. Having a shit ass fucking Monday. I got here this morning and my idiot part timer TURNED THE AC OFF when she left on Friday. It was 100 degrees over the weekend. When I got here it was fucking 85. It will take a full day running to catch up. How fucking dumb can a human be?

Up next was opening the fridge for something cold to drink since it's the Mojave Desert in here. Nope, it was defrosted on Thursday (guess who did that?) and never turned back on. SERIOUSLY. I don't see her until next week but I did fire off a text.

Right now Shawonda or what the fuck ever that little trolls name was would make a better employee.

I want to punch someone in the face.

ROM aren't you glad you asked?

 

Ahh... I misread your note... "I'm out until the 9th" apparently doesn't mean "I'm off on vacation until the 19th".

Go figure.

 

I'm working on eating some of the junk el hubs left behind including these beef taquito things. I did not think I would like them as much as I do.

This is one junk food purchase I will never question again.

 

Sure. Blame it on the dogs.

 

Yeah, shit done broke on them. As I always say, scalable architecture is more than just endless cloud services... you need to have smart software that runs on top of that cloud.

At scale, sometimes the smallest most inconsequential thing can rear up and bite you in the ass and break shit on a big scale.

Funnily enough I used this exact problem with Amazon Prime day as a real-world example to validate my assertion that they need to have proper stress and load testing on shit they want to build and deploy at scale... because until you watch it run successfully at scale, it doesn't.

 

Don't believe the AWS (Amazon Web Services) marketing hype. It's only resources, not the application that was built to use those resources. It's got everything to do with software architecture, not compute (cloud) resources.

Amazon is actually thousands of little software pieces parts that each do their own thing to get the final result to the end user. It's like guts of a very, very complicated watch.

And like a watch, not all of those pieces-parts move at the same speed, or the same distance. The second hands moves way, way faster and covers more distance than the hour hand, for instance.

Unless you intentionally stress and load test that stuff to the same levels before hand, you don't know for sure if all those pieces-parts will work at that load. All it takes is one little part that's not used to working that hard to fail and you get dogs... dogs everywhere.

When the site comes under load, as in peak load when Prime Day happens, everything is running way, way faster than it normally does. And everything kind of ramps up at different rates as that load comes up. It is usually a small part that hits some hitherto unknown max performance level and then breaks, well before everything else breaks, that causes the system to fail.

For instance... it might be that the little customized search cache that can only handle a max rate of 1,000 transactions per second, that normally sees only 200 on a busy day, that fails due to the specific load that Prime Days puts on it. It's being asked to do 1,500, but it can only handle 1,000, so it fails... or shows shitty information... or falls back onto some default or last-known info as some form of self-preservation. Who knows.

It's kind of like an F1 car... "but the engine is made out of titanium, so why would it blow up?" Because some part was faulty, or stressed beyond what it was meant to do, and it had a cascading effect on the rest of the engine.

Same goes for software architectures, especially at scale.

When I architected EA Sports' first online web service, we spent 2 years building out stress and load infrastructure and software to test 15 million CONCURRENT users... as in, 15 million people playing Madden at the same time.

When we first did stress and load, it failed at about 1,000 concurrent users. We found what broke, and fixed it, and tested it again. Now we failed at 15,000. Then 50k, then 120k, then we plateaued at 240k and realized we needed to rearchitect some parts of the system because we found a natural limit of the system as it was designed.

It's complicated shit that most developers I work with have next to no idea how to tackle.

It's fucking HARD to do this shit at scale, and not many people know how to pull it off well. The default state is failure... the fact that it works at all is a miracle.

 

Do not underestimate how hard it is to simulate/emulate load into a site.

How do you simulate 50 million people hitting that one piece of software? How do you know that software "x, y, z" running together, in this configuration, at this particular load, results in a failure mode of "a", which could then cause "b and c" to fail.

It's REALLY FUCKING COMPLICATED... so to "guarantee" that shit works is time consuming and really expensive.

Here's some interesting discussion around it: https://news.ycombinator.com/item?id=17544161

Counterintuitively enough, as you scale up, it increases complexity and latency (especially when dealing with a full mesh network topology)... which can cause it's own failure scenarios.

I don't get why people think it's simple, and are surprised when it fails... I tend to think of it the other way around... it's fucking complicated, and I'm surprised when it works.

Maybe it's because this is what I do all day, every day...