#data-engineering9 entries

This was a great read, an interesting takeaway is that we should be thinking about interacting with APIs differently when writing "agents" (I like Simon Willison's definition of "agent" as an LLM that runs "tools in a loop to achieve a goal").

Normally a human user of an API knows exactly what they want and makes a specific request for it. LLMs have no idea what they're looking for (they should have some means of validating or benchmarking the responses they get, but that's another too-often-ignored problem for another day), so they need to be able to iterate faster. Instead of sending a single (AI-generated) request, getting some (AI-generated) response, checking it, then repeating, we should be sending multiple requests and evaluating multiple responses in parallel before converging on the "best" answer. This means adding much faster and more robust support for branching queries and rollbacks, since in the world of agents these are extremely common compared to the rare cases they're needed during human interaction.

I work with a lot of different kinds of data, and I'm very interested in the processes around how we transform the piles and piles of messy information that are so ubiquitous these days into useful data. I'm learning about data observability on Coursera and just came across this article that I think articulates many of the biggest problems in data engineering and data science right now really well. In particular this point:

Data downtime — periods of time when data is partial, erroneous, missing, or otherwise inaccurate — only multiplies as data systems become increasingly complex, supporting an endless ecosystem of sources and consumers.

This hits home. It's so easy to just pull a dataset out of anywhere now, but we rarely give any thought to whether the data in it make sense. Virtually every dataset I come across has duplicate and missing values, obviously incorrect values, and doesn't line up with its metadata. It's a huge problem, and beginning to untangle it is a very complicated problem but one I'm super passionate about.

I love Randy Au's newsletter in general, but this one makes a particularly useful point: Not thinking about why you're doing something ahead of time is expensive and wasteful.

Randy's discussion of how this issue manifests in software teams in particular is completely on point, in my experience. But it applies to all areas of life. Software engineers get asked all the time to just "log everything", with the assumption being that since we have "big data" tools now, we'll be able to make use of it all later. The reality is that extracting value from heaps of unorganized, unstructured raw data is actually very difficult and potentially expensive, and the time to think about what questions you want to answer is before you start collecting data in the first place. It's also mostly true that once something is being collected or logged, it will go on forever. Another consequence of not thinking through your reporting and monitoring infrastructure ahead of time is that nobody really knows who is depending on what, and so things get left in place "just in case" it breaks something downstream. Of course this is madness, but it's how many organizations operate in reality.

It's true that storage is cheap now, but that doesn't mean accessing said storage is cheap, let alone doing something useful with the mess that's in there. Keeping terabytes of logs in S3 is like having a free warehouse in Labrador for all the tchotchkes you know you don't need but can't bring yourself to throw out.

A lot of this data hoarding stems from organizations wanting to be "data-driven", without ever explaining what that means for their context or why it matters. Collecting data is easy and cheap but does not inherently add value, and the difficulty of extracting value from raw data is a function of its volume (among other things). It's worth spending at least a little bit of time up front thinking about what

I had a super interesting conversation with Ananth Packkildurai of Data Engineering Weekly today and a couple of interesting tools came up. One was this: dltHub. There are a lot of people trying to solve some of the main pain points in data engineering, and I'm convinced (like Ananth) that functional languages and approaches to software engineering have a lot to offer in solving them. This tool in particular appears to attempt to automate away some of the painful parts of the process, and I'm super curious to explore their approach. It seems like a really hard thing to do well. The problem of extracting data from traditional sources (DBs, files, APIs) and moving it elsewhere is well solved, but it's the increasingly complex transformations, large volumes, and disparate end use cases that are causing the strain with the current standard data stack.

I know reddit is a real mixed bag, but I often find good information there. It seems like it's one of the last remaining places on the internet that's not dominated by SEO-optimized or AI-generated hooey. I started the Coursera course associated with the AWS Certified Data Engineer Associate exam and was looking for recommendations for more learning resources when I came across this. I haven't started it yet so I'm not sure how good it is, but in the past I've had a great time learning a lot from these types of community-made resources. I'm looking forward to checking it out.

It's also the first time I've ever heard of the DataTalksClub. I don't know anything about it but it's github profile makes it seem like type of online community I absolute love. I am super passionate about creating free learning resources for technologies I'm interested in and really admire people who have had success doing that in their own niches. I'm excited to join this community and see what happens!

This is a great book that appears to mirror much of the content from the Fundamentals of Data Engineering book by Joe Reis and Matt Housley. I'm finding it really satisfying this week digging more into the distinctions between data engineering, data science, data analysis, software engineering, and everything in between. I started out in very standard software development roles, building rails apps for various companies. My roles over the last few years have been decidedly not typical web development ones, but I'm never quite sure how exactly to explain what I do best. I feel like I'm finally on to something with all of this data engineering stuff.

I always wonder how useful these types of exams are in practice. It often seems like they're mostly not very applicable to "real" work, but this one (and the associated Coursera courses) seem well constructed and practical.

I started a new job a couple of months ago that's much more like what this course describes as "data engineering" than the type of software engineering I'm used to and I have become painfully aware of how little I know about the "cloud" and all the various hosted services for working with data. This course feels different than other ones I've come across so far in that it appears to focus on very hands-on and realistic types of work, and notably also tasks that are very similar to what I'm actually doing day to day now. I think it could actually be very useful to pursue.