I think it's hard to argue that software under the hood has gotten simpler.
Some people have also pointed out (e.g. Casey Muratori [1][2]) that the average software experience has also lagged behind in performance as compared to hardware improvements, but that's another, albeit closely related, story...
Maybe the average experience of software for a common consumer has gotten less complicated, i.e. less clicks to achieve something. And from time to time even had faster turn around speeds for the overall experience of using the software.
And yet these dark patterns in unsubscribing flows and a sinister 'remove friction to improve sales' undertone to modern software is still very much in your face.
Now here's a great line:
"Complexity has to live somewhere."
I use that quote a lot as it has lodged itself into my brain like a first-time climber holding on for dear life.
So much so that I didn't even know where I got it from until I was writing this little piece. Imagine, so true has been my experience to the above quote while writing software, that I just thought it came out right out of my brain naturally.
That said, let's talk about depending on things built by others.
Edit: Future me, guess what, there's a sequel to our NPM worm attack. The spice must flow!
The recent "Shai-Hulud" attack causes security risks for anyone having 1000s of unchecked NPM dependencies.
"Perhaps."
An AWS outage causes issues for beds.
"Understandable."
DHH says he sees benefits to his business after moving off cloud. [1][2 @3:23:00].
"Outrage!"
But seriously, why the F*CK does my bed have to depend on AWS servers to
function?
Silly question. Clearly it's because the company making the beds is faced with the reality of needing to run their own infrastructure. And so, they depend on AWS solving their issues for them.
I think Casey nicely summarised how this mentality is the default in seemingly most businesses now on this episode of The Standup. While DHH actually described a model of operation for cloud infrastructure that seems so obvious but I've never considered in his conversation with Lex. He described effectively just the run of the mill datacenters who supply the network, power and rack space and you supply the hardware to put in there.
A model which is abundantly obvious for anyone who didn't develop their professional career in the cloud. Unfortunately, most of my world at work revolves around virtual machines, not hardware specs.
And perhaps if we leave the cloud for a second, we can descent onto something as commonplace now as anything else - ML, we can maybe see that it's a massive nest of abstractions, frameworks and endless staring at torch compilation logs and CUDA traces. Not only that, but if you ever try to build something in python that uses torch compile, threading, async, triton kernels, and CUDA, you might be in for a fun ride.
All that is to say that there are various consequences in terms of performance, security and maintainability of anything you build on top of someone else's shoulders. That said, the more you depend on others, the faster you can move. And while I can't disagree with that too much, mindless dependence on others doesn't make all the complexity disappear completely. But what it definitely does do, is make you understand less.
For me, the hidden visceral risk associated with increased reliance on "someone else will solve it" mentality is that it makes me dumber. Relying on others too often has the cost of you losing the skills you have in a thing and all the new skills you would have gained not relying on AWS.
A trade-off like that shouldn't be made thoughtlessly and motivations for relying on something should be clear and not solely grounded in difficulty of learning.
A great example of how I've been caught out in my over-reliance on things made by others in a similar way is the following little anecdote.
I was building a project in my spare time using C, which is quickly becoming my
favourite language of the year. And while having a conversation with a technical
friend of mine about it, I complained about how seemingly the hash-map in
stb_ds.h under multi-threaded conditions was
causing weird bugs to occur during concurrent reads which I would have assumed
would not mutate the map itself ¯\_(ツ)\_/¯, anyway - classic.
As I finished my semi-rant, I unironically said something incredibly dumb:
"Man, maybe I should rewrite it in Rust because its easier to find a hash-map library in Rust"
Let's park the fact that you don't need a separate hash-map library in Rust as the standard library already comes with a pretty good API for maps.
And let's consider that a hash-map is incredibly easy to write if you know what a hash-map is (which I was afraid to learn about), particularly if you don't need top of the line performance. What my friend said to me in response is something that will stick with me forever:
"That is such a classic Rust developer thing to say"
That woke *clap* me *clap* up *clap*.
Clearly he wasn't talking about the language, he was talking about the fact that instead of tackling problems myself which actually aren't that hard for what I was trying to achieve, I was trying to find someone else's work to make my life easier, pushing complexity away instead of facing it as a potential learning opportunity and a way to increase independence.
It is effectively the JavaScript way which I already felt had a it's own
unpleasant odor. But, then again, how different is just slapping in a cargo add? I guess at least there's no DOM accounting...
I think this only hit as hard as it did because of a few reasons.
If you understand and have built your stack from inside out, it's an incredible superpower.
Not only has the process of getting there taught you tons of new concepts and skills, but the opposite - building blindly on abstractions, dependencies and frameworks - will likely not be ever as performant.
If you don't understand exactly each layer and how it fits the exact shape of
your problem and what trade-offs are made at each layer and it's too slow, you
will be forced to do it whether you like it or not.
And let's not forget that a line of hand crafted assembly might add two numbers together 180x faster than a python integer sum, but if you've just build on top of abstractions to get to the python sum without understanding what's under the hood, you will never what good looks like in the first place.
If you build something yourself, you unarguably will build some form of higher understanding of the solution and it's fit to the problem. I really like this notion of programming as building up this theory of the code proposed by Naur in 'Programming as theory building'.
Mike Acton and Casey more or less opened my eyes to the idea that if you built it well yourself, you have effectively solved the problem posed to you in a way that perfectly aligns with your preferred and expected trade-offs and requirements.
All of this really is to say that if you want to grow in skill, you have to go and learn the stack. You can't duck and dodge hard problems if you want to be good and if you ultimately want to produce things you understand, you are proud of and that don't just make your bad product someone else's fault/problem.
Yes you are trading away 'productivity', but you are definitely going to win on quality and maybe even performance if you're serious about going all the way down. The former is arguably what matters to a craftsman over the number of lines of code you produce.
And YES, of course you shouldn't write encryption code yourself as it's one of those areas where taking on some dependence increases many positive aspects of your code such as security. And remember this is about conscious trade-offs and validating that you're not bringing too much on for very little fit, not about never relying on something.
And so, clearly, you will not write your own OpenGL, you are unlikely to rewrite a browser from scratch and we can't get rid of your OS kernel as it stands (but if you do try I will be there watching your content).
But, striving to reduce the amount of things that you don't understand and that you depend on 'just because' seems to be the right direction of travel in my life, hobbies and career.
So what does all of this have to do with this mub thing? What is it even?
Well, mub is actually not at all an exciting piece of software! It's a static site generator that makes the thing you see in front of you. It's a simple piece of software that in 2018 I would have never considered hand-rolling. After all there's Hugo and co. But it's deceptively simple in what it does and, really, sometimes writing your own tooling is one of the most rewarding things and processes because they fit like a glove and it's just plain fun. I would encourage everyone to find a thing they need and build it themselves.
Mub was one of the first things that I wrote with this simplicity mindset, even before I consumed a lot of the aforementioned ideas and that little anecdote. While writing it, I realised that I don't want to depend on things if I don't absolutely have to. I weighed carefully dependencies against the time I had to try and implement them myself.
Yes, it's still has quite a few dependencies (I actually list them in the
README), but it has way less than it would have had otherwise. And this
translates right into the rest of the website. At the time of writing, if you
have a WebGPU capable browser, you will see a WGSL shader I wrote as my logo
(thank you to Inigo Quilez for the continuous
inspiration to fiddle with shaders), running inside the browser live. I didn't
use some JavaScript library to help me setup the canvas, I did it myself and I
feel like I learned and gained a measure of confidence. Same with the RSS
feed. All of which actually were built on a very very basic set of
functionality in mub - you effectively just need templating functionality and
you're done.
Edit: I also have used HTMX [1][2] to cut out as much javascript as possible while also making the website feel nice and smooth.
Fun fact, the way the logo looks seems to vary between person to person and device to device. Not sure why, maybe some GPU random state? But it's fun to have this behaviour as a little Easter Egg. If you know why this is happening please tell me.
Well, here we are, I can't image many old-school C devs or anyone that had to live through the Wipeout/Carmack/peak demo-scene era of software engineering is surprised about anything that was said here. All this complexity and slew of pre-built solutions simply did not exist back then.
But, here's the thing, I started my software engineering journey in 2015 in python, in data analysis no less (albeit I did write some Visual Basic as a teen but that's another story). The route I have taken has been long and self taught. I didn't go to university to study CS, I learned a lot of these concepts myself (thank you teach yourself cs), I went deep in the stack. And only now, a decade later am I reaching these conclusions.
Wisdom truly comes with age and I'm excited about all the learning to come.
Edit: oh and this
- Art