Becoming a better programmer by CJ Q. '23

Unknown unknowns

As I’ve talked about several times, this summer, I’m doing an internship as a frontend developer. So far I’ve blogged about the adulting-oriented questions, because that’s what I expected to learn, coming into the internship. But one thing I didn’t expect to learn much about was programming.

I guess that sounds odd; after all, isn’t the point of a software engineering internship to learn software engineering? Well, yes. But to be honest, coming into the internship, I wasn’t sure what there was left to learn. I didn’t have a good idea of what separates a junior from a senior developer.

Part of that was pride, and the other part was ignorance. Sure, I was familiar with the different programming languages used, popular libraries and frameworks, revision control, design principles, testing and debugging, developer tools. And I understood the importance of well-organized code that is easy to understand, ready for change, and safe from bugs.⁠01 Shoutout to 6.1020 Elements of Software Construction (old number 6.031).

I knew I lacked experience in the “soft skills” department, but this post isn’t about that. It’s about the hard skills I wasn’t good at because I didn’t realize they existed in the first place. And now that I think about all the things I don’t know, and the possibility of more unknown unknowns, and I can understand why software engineering is a career.

What is abstraction?

Let’s say we’re having a conversation. What happens is that:

• I have this big picture in my head, like “what I did this weekend”, that I break down into smaller ideas, like “go on a walk”.
• I break down “go on a walk” even more. Maybe I’ll tell you where I went, who I took a walk with, or what we talked about.
• I take one of these small parts, and convert it into words. I need to think about what each word means, and how to arrange them in a way that’s understandable.
• Each word is made of sounds. I need to remember how each word is announced. I also want to vary my tone of voice, and add pauses at the right times.
• Then I make these sounds. I move my diaphragm, lungs, tongue, lips, throat, to make the air vibrate.
• That vibration goes through the air, reaching your ears. You put together the sounds to make words, and the words to make sentences, and the sentences to make these parts, that build up into ideas.

When we have a conversation, we don’t think about all those details. We don’t think about moving our tongue, we don’t recall the definition of each word. A conversation isn’t a series of sounds, words, or sentences: it’s ideas going back and forth. And when we work on the level of ideas, we can abstract the lower details away.

Frontend development is also one abstraction on top of another. You need to know the programming language you’re using, then whatever libraries and frameworks you’re using, and even those depend on each other. For example, in our codebase we use Redux Toolkit. To understand the point of Redux Toolkit, you need at least passing familiarity with Redux. To understand the point of Redux, you need to understand React state, and have seen React applications with large, complicated state.⁠02 The <em>state</em> of a website is all the data it has. The state for Twitter's website includes the tweets you're viewing, their likes and retweets, their author and the link to their profile picture, the date the tweet was posted, and lots of other stuff. Web apps like Twitter tend to have a lot of state.

Handling bigger abstractions

Abstractions help, but you still need to load them in your head, and that’s hard. One thing I’m learning is how to increase the level of abstraction I’m comfortable with.

Let’s go back to the conversation example. Consider three English speakers learning Spanish, of different familiarities:

• Alice is a new speaker. When listening, she has to follow word-for-word, thinking about the definition of each one, and then translating them to English. When speaking, she thinks about each word she wants to say, and focuses on pronouncing the words correctly, and in the right order. She thinks in the level of words.
• Bob has more experience. He can listen to short sentences and translate their meanings to English, without having to translate word-for-word. When speaking, he can put together phrases and sentences with ease. Sometimes he has to think about words, especially ones he’s less familiar with. But most of the time, he thinks in the level of sentences.
• Carol is a fluent speaker. She can hold extended conversations in Spanish, without having to slow down to think about what each sentence means before hearing the next one. When speaking, she can think mostly in Spanish, and explain her ideas without having to think about English. She thinks in the level of ideas.

The three speakers—Alice, Bob, and Carol—all think in different levels or units of Spanish. Going down levels is necessary sometimes, like how Bob has to think about hard words, or how Carol might take a moment to understand complex sentences. And it’s not as if Alice can’t say sentences or convey ideas, but it’s going to take her more time than it would Carol. Alice is only fast enough to keep up with words, while Carol can keep with ideas. Each person has some highest level that they can work with quickly and comfortably. Let’s call that their abstraction level.

Part of becoming a better programmer is raising my abstraction level. When I started out my internship, I was unfamiliar with Redux Toolkit. To do anything with the state, I had to think about useDispatch, and useSelector, and createAction and createReducer. I could still use Redux Toolkit to change the state, but it took time to think about which function I needed to call where, and what each of them did. Now that I’m more familiar with Redux Toolkit, I can now think in the level of “do some action” or “read some state”, and the way I became familiar was by using it over and over again.

As I talked about in The joys of web development, I enjoyed the feeling of “learning how to do things”:

I feel so powerful learning how to do more things. It feels like power to SSH somewhere and not feel completely lost. It feels like power when it took weeks to understand the first programming language, but hours to understand the tenth.

I think this feeling of “learning how to do more things” isn’t just about learning more things, but about raising my abstraction level. If all I learned was how to use frontend library number twelve, without feeling like I’m doing anything more abstract, then it doesn’t feel like learning power, but like memorizing facts. A good tool is one that gives good abstractions, one that hides details, not adds more.

Planning and scoping

Another reason abstraction levels matter is scope. The higher your abstraction level is, the larger your work can be. Five-paragraph essays are at the paragraph level, where each paragraph has a specified role, like an introduction or conclusion. Research articles are at the section level, with sections like related work or methodology. A book is at the chapter level, each chapter having sections of its own. To a person who’s only comfortable with paragraphs, writing a book seems daunting. But once you’re comfortable with chapters, you can put several chapters together to make a book.

My internship was structured as a series of projects, and I was given lots of latitude to plan each one. I learned how to take a big feature, like, say, adding operations to a group of files rather than a single file. What features do we need to add, what features are nice to have? Why do users want these features? I think this is called scoping, the phase of defining the requirements and judging how important they are.

Then I planned the technical changes. I needed a new backend endpoint, I needed to add checkboxes and their state, then I needed to add buttons that call the new endpoint. Then I planned the implementation of the new endpoint: what’s the logic? How is it different from the existing one? What are the options for implementation, what are the trade-offs with each option? By breaking it down into parts and planning each one, I’d turned a big feature into a series of small tasks.

How small do the tasks need to be? I only needed to break down the task to the level of my highest abstraction level, I guess. For example, I didn’t have to break down the state management into how I’d call the Redux Toolkit functions, because I was comfortable with that level. But I had to break down the endpoint implementation into smaller parts, like how I’d schedule tasks and run them in parallel, because I wasn’t as familiar with the backend. I could imagine that, if my abstraction level for working with the backend was higher, I’d have needed less planning and less time to write that part.

This kind of thinking—I’ve heard it called computational thinking—works on smaller scales too. If a task I’m working on is too large, I find that I can’t get in the zone. It helps to mentally start with something quick and wrong, then gradually turn it into something that’s good and works; else I spend too much time thinking and not enough time writing. And it applies to non-programming things, like how I write blog posts by making more and more detailed outlines. Why else would eating an elephant one bite at a time be a famous proverb?

Investigating code

I like to think of planning a project as going up the ladder of abstraction, in a sense. You have the highest abstraction level you can comfortably work on, but you have to work at a higher level. The opposite skill, going down the ladder, is also important. When something doesn’t work the way you expect it to, you have to go down the ladder and see what’s wrong.

For example,⁠03 This is a pretty technical example; feel free to skip to the bullet points at the end of this section. I was implementing deletion in a sidebar. The sidebar has some documents, and documents can be expanded to show a list of records in each one. It seemed that, after deleting a document, one of the documents, would become collapsed. These are separate functionalities, and so they shouldn’t be talking to each other. What gives?

I used Redux DevTools to go down and look at the state, which was correct. It wasn’t collapsed, it only seemed collapsed. Using the browser devtools, I saw it only looked like it was collapsed because it was behind the file in front of it.

After reading the code, I settled on investigating react-virtualized’s List. The List should place the files so that they don’t overlap. But it doesn’t, which means what I knew was wrong. In other words, I had the wrong abstraction for what List did. I then had to read the source code for List to figure out what was going wrong. After ruling out several possibilities, I concluded that it wasn’t recomputing the layout when it needed to.

Why wasn’t it redoing the layout? Again, I made several guesses, none of them correct. After reading more source code, I realized where my abstraction was wrong. The default renderer uses indexes as a key. That means the layout of a file depends on its index. That’s why the new file was in the wrong place: it was in the place of the old file at that index!

Investigations like these made me realize a few things:

• All abstractions fail, and sometimes you need to figure out where. That means knowing the lower levels of abstraction too. If you know them well enough, you can single-shot debug.
• When investigating, start from the highest abstraction you think could go wrong, then go down. It’s easier to trace what an abstraction depends on, rather than starting with an abstraction and seeing what depends on it.
• Things like Redux DevTools, React DevTools, Chrome DevTools, and Firefox DevTools are useful because they’re abstractions over the lower levels. Rather than having to touch the lower levels directly, you can look at them with abstractions you’re more confident with. It pays to know how to use your devtools well.
• GitHub Copilot is cool, but writing new code is only one part of the job. My hardest programming problems involve reading and editing existing code. Because so much time is spent reading code, it’s more important for a programming language to be easy to read than it is to be easy to write.

Applying patterns and principles

There’s this field of knowledge that I don’t know a good name for, but the closest thing is “programming principles”⁠04 This is taking a page from <a href="https://senseis.xmp.net/?GoProverbs">go proverbs</a>, which is kind of what I’m going for. or “design patterns”. The haphazard field is built up largely through folklore, essays, and talks. It includes things like:

• The Zen of Python, which has things like “There should be one—and preferably only one—obvious way to do it.”
• The saying “prefer duplication over the wrong abstraction,” which originates from a Ruby on Rails conference talk.
• John Ousterhout’s book A Philosophy of Software Design, which has sayings like “Define errors out of existence.”
• Humorous sayings like the ninety-ninety rule or two hard things.

Knowing theory and applying it are different skills. In a conversation I had with a coworker, we talked about online competitive Tetris, and “openers” and “garbage” and “the meta”. Later he looked me up on some Tetris websites, and discovered that I only have a few hours of experience playing Tetris. He then told me, and this is a direct quote, “lol yes you know way more theory than your stats show”.

This isn’t only a Tetris thing. I can recite lots of go proverbs, but I’m only 13 kyu on OGS.⁠05 Wikipedia categorizes me as a <a href="https://en.wikipedia.org/wiki/Go_ranks_and_ratings">casual player</a>. I one day dream to become single-digit kyu. And I’ve read a lot about software design patterns, but I’ve only written so little code. It turns out theory is useless if you can’t recognize when to apply it.

When I do recognize theory, though, it feels so satisfying. For example, when I was implementing the deletion mentioned earlier, there was some existing code that did something similar. There was the code that added files, and the code that saved files. I thought about whether I could take out the common logic between adding, saving, and deleting. I decided not to, following the principle of “prefer duplication over the wrong abstraction”. It turned out to be the right call, because the deleting logic soon became different.

As another example, I recently helped migrate our TypeScript codebase to use strict null checks. Some migration involved changing what existing functions did when the inputs were null or undefined. Rather than throwing errors, I edited them so they did nothing gracefully, like the idea of “Define errors out of existence.”

I wanna get better

The people I’ve met at the company are a lot like the people I’ve met in MIT: they’re brilliant, they’re grounded, and they’re relatable. If I ask my coworkers, “What’s your favorite part of working here?”, I know what answer I’m going to get: “the people”.

Being around such cool people makes me want to get better. I want to be a better person, someone who’s humble, and kind, and knows how to listen. I want to be a better coworker, someone who’s a joy to work with and be around. And I want to become a better programmer:

• I want to reach the point where I can write larger and larger features, by building larger and larger abstractions, without having to go all the way down each time.
• I want to learn the right questions to ask to learn what users want, identify priorities, and estimate how much time it takes to do things.
• I want to know the frontend stack better. I want to understand how React does so many rerenders, and yet is reasonably fast. I want to understand performance so well that I can find the smallest changes that will have the largest effects.
• I want to have better judgments about when to factor out code, where to place a piece of logic, and what names to give variables or types.
• And I want to share what I’ve learned, once I get there. I want to help discover the best way to teach not only programming, but software engineering, because I think this is the most important unsolved problem in the field.

I want to close with a song from Bleachers, I Wanna Get Better. I think this captures the feeling I’m going for so well. It’s like I’ve looked up and seen such great heights to climb, and I want to get there. I have so much to learn, and it’s not intimidating, but exciting. I have years ahead of me that I can dedicate to becoming better, and I know that still, it would not be enough.

I wanna get better. That’s it. That’s what I want in my life.