Denouement
Curse you, concurrency, my old nemesis! You have bested me once again!
Pycon 2024 in Pittsburgh was quite good. A few decades ago I lived outside Philadelphia for a few years and the experience made me write off the whole state. I had heard good things about Pittsburgh but was unprepared to be so impressed. The landscape reminded me of Portland, OR, but it has way more bridges: 446 according to the bus tour I took, more than any city in the world. It also had a lot of things that reminded me of Seattle when I lived there. The conference will be in Pittsburgh again next year and I recommend the experience.
My presentation was called Functional Error Handling (at this writing it is not yet up on YouTube but should be soon). I had proposed talks on concurrency but these were not accepted.
Last Fall I began diving into the book project, creating the necessary tooling to test and update the examples in the book. Despite the number of books I’ve written, I seem to reinvent these tools. I get new and better ideas from one book to the next.
I was greatly enjoying the process of building these command-line tools, using modern libraries like typer and rich. You can see it here. I also tried to finally understand project organization. This is a remarkably slippery topic and I mostly rely on what rye produces (I am aware that rye will eventually be replaced by uv but I’m comfortable with it for the time being).
I just drifted to a stop. For months, I was puzzled and frustrated by this. I find being in the middle of a book project to be one of the most gratifying experiences, and in the past I tried to somehow constantly be working on a book (unsuccessfully, and I’ve slowly come to understand that this may be impossible for me).
Fortunately I had the Pycon deadline to restart my interest. I worked on the presentation for months. In the meantime I slowly understood why my brain decided I should stop writing a concurrency book. Here are contributing factors:
I found an article arguing that we are in a “Concurrency Cambrian Explosion.” This agreed with my experience that, right now, most of the skill required is in knowing lots of strategies and deciding which one fits your situation.
There’s an uphill battle with the Dunning-Kruger Effect. I don’t know how many times I’ve talked about concurrency and had people declare that it just isn’t as hard as I claim. At the conference, I had a discussion with a language expert who thinks the free-threading experiment will surprise people who assume that innocent-looking code is going to work fine, and don’t understand what it means to make libraries thread-safe, or conceive of the effort it will take to create an application using free threading.
I had other reinforcing discussions at the conference. Some were promising: subinterpreters will be a more restrictive approach than free threading, but should have a vastly easier mental model. I had lunch with one of the maintainers of the Trio library, who made me realize that I had quite misunderstood the situation. I had thought that Python’s built-in asyncio library had acquired all of Trio’s capabilities and so we could just use asyncio. That turns out to be wrong, and Trio has significant advantages over asyncio. I made that mistake and I’m sure I’ve made others. I may never be the person to write an authoritative book on concurrency.
Concurrency is currently a niche, because adding it is so impactful to your project. The complications cost a lot, and not just when creating the code: maintainability is an expense nightmare. There’s also the understandability factor. I’ve heard more than one story where maintainers come across performance code, don’t understand it and just take it out. They assume the original programmer was either misinformed or on drugs.
I’ve discovered I’m more attracted to writing for a general audience than a niche audience. Perhaps it is simply impact: I think there are techniques that could make all programs better and all programmers more productive. Teaching these techniques will benefit the world, and that inspires me.
For the last 3+ years I have been helping out with a book called Effect Oriented Programming. This has been quite a struggle and has produced, slowly, a significant rewiring of my programmer-brain. I mention this because effect management systems have ways to almost transparently implement concurrency. When all this settles down I suspect we might see something akin to this, where the programmer doesn’t need to be an expert in concurrency in order to use it. But in the meantime it is a domain for specialists.
Months of confusion and dismay led me to see that I needed to quit. I notice the relief I feet. Concurrency as it is now is too hard for me to understand, much less explain. I have learned bits of concurrency for decades, but not enough to tackle it as it is now.
I wrote a diatribe about how Substack doesn’t fit my needs for a programming blog, but decided not to publish it—”programmers aren’t their target market” says enough. Right now I’m finding Obsidian a much better solution. For plain writing, though, Substack is appealing. If I feel the desire to write about something else (as I did with Reinventing Business) I will reconsider Substack.
Concurrency is very challenging. And the Dunning–Kruger effect quite common. Most programmers seem to expect some quick trick or tool will magically solve all their problems, like a silver bullet. Even pointing out their errors to them, face-to-face often has little effect, as their faith is so strong.
We've had to implement scanners for mutable global (static) variables and Collections, as quite a few programmers seem to have any idea that this is a bad thing -- on a multi-user web server.
Many expect that adding a "synchronized" wrapper to a Collection makes all uses thread safe -- without regard to how the code using the collection uses its methods.
Loved your talk at PyCon! Tried to implement some of it right away even though it's a little fancy for my level. Impossible to compare Pittsburgh to Philly.