January 202513 entries

I love Randy Au's newsletter in general, but this one makes a particularly useful point: Not thinking about why you're doing something ahead of time is expensive and wasteful.

Randy's discussion of how this issue manifests in software teams in particular is completely on point, in my experience. But it applies to all areas of life. Software engineers get asked all the time to just "log everything", with the assumption being that since we have "big data" tools now, we'll be able to make use of it all later. The reality is that extracting value from heaps of unorganized, unstructured raw data is actually very difficult and potentially expensive, and the time to think about what questions you want to answer is before you start collecting data in the first place. It's also mostly true that once something is being collected or logged, it will go on forever. Another consequence of not thinking through your reporting and monitoring infrastructure ahead of time is that nobody really knows who is depending on what, and so things get left in place "just in case" it breaks something downstream. Of course this is madness, but it's how many organizations operate in reality.

It's true that storage is cheap now, but that doesn't mean accessing said storage is cheap, let alone doing something useful with the mess that's in there. Keeping terabytes of logs in S3 is like having a free warehouse in Labrador for all the tchotchkes you know you don't need but can't bring yourself to throw out.

A lot of this data hoarding stems from organizations wanting to be "data-driven", without ever explaining what that means for their context or why it matters. Collecting data is easy and cheap but does not inherently add value, and the difficulty of extracting value from raw data is a function of its volume (among other things). It's worth spending at least a little bit of time up front thinking about what

Rich Hickey just pushed a new namespace to the core.async library that looks really interesting. The accompanying rationale explains what it's all about in simpler terms than the code, but it feels like a big step forward in making core.async more intuitive to use. Channels are a super useful abstraction, but as the rationale explains, they are still a relatively low-level construct that require expertise and good architectural decisions to use well. This new namespace provides some more abstractions on top of channels that handle, among other things, "all channel I/O, thread lifecycle and coordination with the flow graph". This seems very powerful.

These are very high-level abstractions over an entire system, but I often wonder if this is just the inevitable direction in which software is headed. As we come to understand software systems better and better, and as the tools for generating implementations of solutions to known problems become better and better, it seems like this shift toward ever higher-level abstractions is the natural outcome.

It reminds me a bit of what Hyperfiddle is doing with electric Clojure, abstracting over the entire network, not just the "front" or "back" end of a web app. It's a totally different domain, but similar in that it feels like an impossibly high level to develop at, yet it appears to work very well. I'm excited to out both of these.

This is a really cool tool. I use LLMs extensively in my side projects with great success (my workplace is still investigating whether there are ways we can use them that will align with their privacy needs) and often simulate what this tool is purpose-built to do. I think it's going to save me even more time.

When I first started playing with AI-assistant editors (like Cursor, Windsurf, Co-pilot, etc.) I assumed they would be way better than chatting back-and-forth with an LLM, but it turns out, counterintuitively (to me anyway), that I much prefer chatting over the in-editor suggestions. I find these mostly intrusive and wrong, and they just create more work for me. In a chat interface, though, I can guide the conversation in constructive directions.

It might have something to do with the types of questions I ask them, or maybe just my personality. In any case, I frequently copy-paste dozens of files into a project, describe in great details a project's structure and data model, and then ask questions about it to an LLM. It sounds like this tool can automate and improve that process, which is great news for people like me who frequently engage in this kind of workflow.

I found this article interesting. It's about how AI-generated web UIs might help add more value to the work data scientists do. It was also a little depressing to read as a software engineer, but a good reminder of what our role is morphing into within organizations.

The main takeaway for me was that most code is so shoddy these days that we may as well just get LLMs to spit out UIs that are good enough to serve our immediate needs and then move on. She argues that between the break-neck hype cycle JS frameworks go through, high industry turnover, piles of existing tech debt, and constantly changing requirements, code written by LLMs isn't really that much worse than code written by developers anyway and given the astronomical cost savings, maybe we shouldn't be so skeptical about using generated, disposable UIs.

Of course my first reaction is cringe and outrage, but then I have to immediately admit that she's has a point. I've seen a lot of codebases, and I'm ashamed to say that as far I can tell the average quality of software in the wild is poor to terrible. The sad state of web development these days combined with the increasingly impressive quality of code that LLMs (especially Claude) can write, means I have to agree that it actually makes a lot of sense to use them to write software. They're at least very useful for prototyping and getting started, as she points out.

Ultimately this is just another reminder that software engineering is table stakes now. Anybody doing anything tech-adjacent in many organizations is more or less expected to know how to turn their output into a functioning web app a normal person can see, and despite our best efforts this is still an exceedingly complicated task. Nobody can be an expert in so many things at once, and as long as that remains the industry expectation, I predict that we'll continue to see more and more of the gaps filled by LLMs.

I came across this paper through another Coursera course I'm exploring about reproducible research in preparation for a talk I'm giving in March and found it both enlightening and a little depressing. Enlightening because it offers an interesting perspective on some of the problems with the current state of data work that contribute to the replication crisis. As a software engineer, it's wild to me that it was not just acceptable but totally normal and inoffensive to publish exclusively the results of a computation, without providing any data or code that could allow someone else to reproduce the results. This is improving since the time this paper was published, but it is still very standard to find no indication of how a given analysis was run or result was achieved in a given paper.

The author describes one system their team at the journal Biostatistics implemented to tag papers with confirmed reproducible results which, at the time of this paper's publication, had been done successfully on 5 out of 125 papers. That was a little depressing to read. The author later points out that, like so many things in life, the standard we get is really just whatever the community is willing to expect and enforce.

I had a super interesting conversation with Ananth Packkildurai of Data Engineering Weekly today and a couple of interesting tools came up. One was this: dltHub. There are a lot of people trying to solve some of the main pain points in data engineering, and I'm convinced (like Ananth) that functional languages and approaches to software engineering have a lot to offer in solving them. This tool in particular appears to attempt to automate away some of the painful parts of the process, and I'm super curious to explore their approach. It seems like a really hard thing to do well. The problem of extracting data from traditional sources (DBs, files, APIs) and moving it elsewhere is well solved, but it's the increasingly complex transformations, large volumes, and disparate end use cases that are causing the strain with the current standard data stack.

One of my favourite blogs on the internet is Simon Willison's Weblog. I recently read this post of his about link blogging and love the idea. I occasionally post longer-form articles or essays on my blog, but usually only one or two per month, and I have a totally chaotic and unorganized web of ways to keep track of other interesting things I'm learning about or doing, but a lot of them are things I would love to share or connect with other people about.

I think his style of blogging is a great fit for the ways I use the internet and keep track of notes. I'm already working on refreshing my website (again), and I think this way of thinking about what to publish and how will really work for me. I'm still enjoying the experience of writing in Obsidian, and I find the built-in daily notes functionality makes it really easy to keep things organized. Now I just have to turn all the content into a website :) I'd like to also eventually migrate all of my various prior attempts at blogging, at least on technical topics, to this system so it's all in one place.

I know reddit is a real mixed bag, but I often find good information there. It seems like it's one of the last remaining places on the internet that's not dominated by SEO-optimized or AI-generated hooey. I started the Coursera course associated with the AWS Certified Data Engineer Associate exam and was looking for recommendations for more learning resources when I came across this. I haven't started it yet so I'm not sure how good it is, but in the past I've had a great time learning a lot from these types of community-made resources. I'm looking forward to checking it out.

It's also the first time I've ever heard of the DataTalksClub. I don't know anything about it but it's github profile makes it seem like type of online community I absolute love. I am super passionate about creating free learning resources for technologies I'm interested in and really admire people who have had success doing that in their own niches. I'm excited to join this community and see what happens!

This is a great book that appears to mirror much of the content from the Fundamentals of Data Engineering book by Joe Reis and Matt Housley. I'm finding it really satisfying this week digging more into the distinctions between data engineering, data science, data analysis, software engineering, and everything in between. I started out in very standard software development roles, building rails apps for various companies. My roles over the last few years have been decidedly not typical web development ones, but I'm never quite sure how exactly to explain what I do best. I feel like I'm finally on to something with all of this data engineering stuff.