Working in the Software industry has been rewarding to the majority of people in it. There is no doubt that the knowledge workers of today are much more valued and have many comforts that generations before did not enjoy. Unlike other knowledge fields such as Medicine, Law, Finance, or Science, the barrier to entry is extremely low in technology. Anyone can theoretically decide that they would like to become a software engineer and there is nothing stopping them.
While a university degree is often a pre-requisite for job, even that requirement is slowly becoming “optional” for the right talent. The sheer number of learning resources that are available for free to anyone willing to learn dwarf anything else in comparison.
In all of these aspects, the tech industry is unique, and there has been no historical precedent.
So while one might readily think that becoming a software engineer must be easy and that after doing a few (or many) online courses, one can be ready to join Big Tech, they might be disappointed. Technology’s reality is not nearly as pretty as its utopian vision.
The expectations from a Software Engineer of today is vastly different from what is traditionally taught in University and even bootcamps. I’ll list down a few primary points.
In today’s industry, good engineers are expected to have:
- Algorithms/Data Structures and Problem Solving.
- Good grasp in at least one programming language.
- Reasonably good with Unix/Linux.
- Servers and Databases (backend) / DOM & browser-specific behavior (frontend).
- Some level of expertise in a Cloud (AWS, Google Cloud, or Azure).
- Familiarity with containerization technologies (Docker, Kubernetes et all).
It’s a mix of things that are taught in University (DS/Algo), things that do not have a clear definition, or a definition that changes according to company (“problem solving”) and vendor-specific technologies (cloud).
Data Structures/Algorithms are a very contentious topic in software these days. While many argue that it is absolutely essential for a programmer to have a good grasp of them, many think that because of software abstraction, a programmer does not have to know how an algorithm or data structure works in order to build a solution - kind of like building lego.
Problem Solving is sometimes confused with the knowledge of existing well-known algorithms, but this is something which is different for every company. Microsoft, Google and other big tech firms infamously gave many puzzles, math-heavy problems or just plain nonsensical questions in the name of testing candidates’ analytical abilities. Essentially you can create anything under the guise of problem solving and set your benchmark, and there is nothing stopping you.
Programming languages An Engineer is as opinionated about her language of choice as a Photographer is about her Camera gear. Some people want an engineer who has deep expertise in one language (a “C++ wizard”), or sometimes you encounter job postings that expect a good programmer to be a polyglot - someone who can easily switch between a variety of languages. The fact that languages tend to have a wide variety of designs - dynamic vs static typing, compiled vs interpreted, functional vs object-oriented (or a mix of them), threads vs async, and LOTS of other factors make the decision to pick a programming language and stick to it pretty hard as well.
Unix/Linux: I think this is the one requirement that almost nobody would disagree with. Knowing your way around a unix system is essential since it is (either directly or via its descendents) the OS that the Internet essentially runs on. There are multiple flavors of unix, and within the OS, just navigating isn’t enough. One must be familiar with the file system, the OS-level concepts, a few basic syscalls, some tools like grep, sed, awk, netstat, netcat, lsof, top, the list goes on.
Servers Backend or Frontend, you need to know how a web server works. There isn’t really any getting around this fact. The request-response model over HTTP is so ubiqutous that if you do not understand it, you’ll be forced to, sooner or later, or you won’t last.
There is the fact that just knowing how a typical server works isn’t enough, since we have created various communication models (I wouldn’t call them protocols since that term has a different meaning) b/w how servers communicate with applications or with each other. You’ll soon come across terminologies like APIs, REST, RPC, SOAP (if you’re unlucky) and you’ll have to understand what those are.
Databases Backend engineers will eventually need to store data. Whether that’s transaction data, user information, a video game’s state, or a company’s analytics warehouse. Eventually you will have to deal with a data storage system. Things don’t get simple here, but even more complex.
For 30+ years, relational data model reigned supreme and Oracle, SQL Server (and recently MySQL/Postgres) were kings. One learned SQL and that was that. Then came the NoSQL revolution in the 2000s and the world started doing things in various other models - document model, graph model, a “almost-SQL” model. Each system came with its own quirky language that programmers had to learn. And now, we’re seeing a revival of SQL in which scalable systems are being created on top of the tried and tested SQL system.
In short, I think just like servers, you won’t get very far with backend development without learning SQL if you want to create interesting systems. You can make a compromise and learn relatively easy systems (think MongoDB) and that will do the job for you, but the relational model is something that is crucial to our understanding of the data systems and you’ll realize that sooner or later.
Cloud: Over the past decade, AWS’s dominance in the cloud space led to a revolution - now developers and ops engineers were being hired not only for their expertise in general software practices, but also how well versed they were with Amazon’s ecosystem and their managed service offerings. This isn’t entirely unfair, because the world moved on from the on-premise/roll-your-own server model to a virtual, cloud-based model. AWS came up with many innovative technologies, separating compute and storage, their managed database offerings, their networking technology, spot-instances, and a lot of other interesting things. And since companies started to migrate to AWS to handle all of their infrastructure needs, it made sense for them to hire AWS experts.
Now, Google is catching up, and coming out with its own set of services - some similar and some completely new and unique. Microsoft has also decided to focus its energy on the cloud instead of its enterprise software business, and is pouring its energies into Azure. We are entering a multi-cloud world and engineers will need to have cloud expertise in order to survive and thrive.
Containers: The Cloud revolution lead to the latest must-know technologies for engineers is containerization. The rise of Docker and then Kubernetes has ensured that the way engineers think about deploying and managing servers has changed. Servers are now cattle instead of pets.
And of course, it isn’t enough. You should know programming design patterns, MVC, API development, REST/RPC, concurrency, some basic knowledge about message queues and a couple lessons on distributed systems won’t hurt either.
And then there are libraries and frameworks and build tools. Flask/Django/Rails. Angular/React/Vue. Maven/Gradle. Grunt/Gulp/Webpack.
I’m not even gonna mention AI/Machine Learning. The theory vs practical divide is huge there as well.
There is too much to learn. Right from the basics in university, to the more practical foundations, and then the innovation coming in from both industry and open source. Trends change every few years. People aren’t gonna stop creating new and better software. The industry isn’t going to stop and say X is good enough, lets not improve upon it.
This increasing mountain of technical knowledge raises some questions. Where should we stop? What level of abstraction is sufficient? Should engineers always keep up with the latest and greatest? How do you make the decision on what to learn vs what to discard/ignore. How do you distinguish between hype and reality?
Do you grapple with these questions? What answers have you come up with?