We call an application data intensive if data management is one of the primary challenges in developing the application. While in compute-intensive systems the challenge is parallelizing a very large computation, in data-intesnive applications we usually worry more about things like storing and processing large data volumes

The difference in use between backend engineers (who modify data & generally look at one user at a time) and business analysts/data scientists has led to a split between operational systems and analytical systems that are often kept separate.

point query: a query that looks up a small number of records based on a key
OLTP: Online Transaction Processing - a system that inserts, updates or deletes records generally based on a key
OLAP: Online Analytical Processing - a system that generally scans a huge number of records to calculate aggregate statistics rather than returning individual records to the user

They differentiate ClickHouse etc (Pinot, Druid? never heard of them) as product analytics or real-time analytics which are designed for analytical workloads, but serve user-facing products

Data from OLTP systems is often spread across the enterprise, and BAs do not want to have to query across potentially dozens of systems. A data warehouse contains data extracted from many OLTP systems in a company. (They go on to say that it can come from many other sources as well, which is more accurate imo)

A data warehouse often uses relational tables, which is well suited to what BAs want, but less well suited to data scientists' desires, training ML models and using NLP or computer vision.

(They say that feature engineering is particularly difficult to express using SQL? I'd like to understand what they mean by that more here. [citation needed])

a data lake is a centralized data repository that holds a copy of any data that might be useful for analysis. The difference from a data warehouse is that a data lake simply contains files, without imposing any particular file format. (So, s3 is a data lake? Seems like a kinda useless definition imo)

the term cloud native is used to describe an architecture that is designed to take advantage of cloud services

This is an extremely fuzzy definition to me. They say that Postgres and ClickHouse are self-hosted systems that are not cloud native, but that Aurora and Snowflake are cloud-native, and I think you'd be hard-pressed to really apply the definition above to make that distinction clear.

The things that distinguish them to me are:

• usually specialized to the operating environment of a particular cloud service provider
• usually not available as open source
• usually not available to run on a general-purpose computer, or available but in severely degraded form

To me, if we want to split the mysqls and postgreses of the world from the aurorae and BigQueries, it's about how specialized they are to cloud service providers' computers as opposed to general-purpose computation

The key idea of cloud native services is not only to use the computing resources managed by your operating system, but also to build upon the lower-level cloud services to create higher-level services

yeah that's closer for me

Let’s not mince words: everything is kind of awful right now.

Today, I wanted to offer some inspiration—a blueprint for action—for building a better world. Let’s dig in!

Build community

The single best way I’ve found to stave off that looming feeling of existential dread about all the things is to build community.

Not a Discord or Signal group. Real, in-your-town community.

Pick something you care about: a problem in your town that needs fixing or something that would make life better for everyone. Then, find some friends, post about it everywhere locals congregate (digital and/or physical), and get to work.

Connecting with and seeing a direct impact in the place you live is a different kind of rush. It’s tangible.

It’s also a form of prefiguration, demonstrating the kind of world you’d want to live in to folks who can’t imagine the value of it until they can see and touch it.

Music

I grew up in the punk and hardcore scene, where anti-fascism, mutual aid, and community care are core values.

Last year, I started listening to punk music a lot more again, and it’s done a lot to rewire my brain in very positive ways. Here’s a quick starter list…

Revolution by Pennywise

One million strong.
We cannot fall.
It’s all for one and one for all!

Architects by Rise Against

Do you still believe in all the things that you stood by before?
Are you out there on the front lines or at home keeping score?
Do you care to be the layer of the bricks that seal your fate?
Or would you rather be the architect of what we might create?

Rise by the Flobots

So much pain, we
Don’t know how to be but angry
We rise together

Hinds Hall 2 by Macklemore

Rap, but still punk AF.

Long the live the resistance if there’s something to resist.
Had enough of you mother fuckers murdering little kids.

It’s bigger than hip-hop by Dead Prez

Again, rap but punk AF.

One thing about music: when its real, they get scared.
Got us slaving for the welfare.
Ain’t no food, clothes, or healthcare.

Dead Men Don’t Rape by Delilah Bon

All the content warnings on this one, and it’s honestly a hard listen but an incredible piece of art.

It’s never been pro-life, only been pro-white
Black mothers fighting for justice
And what about babies? You wanna save them?
You only care when they’re unborn

I’m from the bay by LaRussell

Revolution isn’t just about anger and fighting injustice.

It’s about community and care. And I haven’t seen anyone do that better than LaRussell.

Everything he makes is focused on his community, the people there, spreading joy, and taking care of people. And fuck, do we need more of that in the world!

Books

There’s probably a lot of great nonfiction out there on how to lead and create change. But for me, reading fiction that paints a picture of what could be inspires me to get off my fucking ass and do something more than any biography ever could.

• The Monk & Robot series by Becky Chambers
• The Wayfarer series by Becky Chambers
• To Be Taught, If Fortunate by Becky Chambers
• The Murderbot Diaries by Martha Wells
• The Dungeon Crawler Carl series by Matt Diniman (more of a how-to for right now)

Shows & Videos

I’m reading a lot more than I’m watching these days, but I will pop on a lot of YouTube videos in the background while I fold laundry, do the dishes, and garden.

• Andrewism, on solarpunk and anarchism
• The Leftist Cooks, on all sorts of left-leaning human interest topics
• F.D. Signifier, a nice mix of media analysis, politics, and culture
• Our Changing Climate, on solarpunk and socialism
• Thought Slime, on all sorts of leftist shit
• Kipo and the Age of the Wonderbeasts, an amazing show about building community in dark times

Ms. Rachel

I don’t have a lot to say here other than “be more like Ms. Rachel.”

If you’re not familiar, she’s the host of a children’s TV show in the spirit of Mr. Rogers and Reading Rainbow. And she’s an very outspoken supporter of LGBTQ+ kids and Palestinian liberation.

As you might imagine, this has pissed off a lot of people and jeopardized her career.

Unlike most media folks, though, she didn’t just stand firm. She doubled down, and has publicly stated she’s willing to destroy her entire media empire to stand up for what’s right.

She has more courage and moral backbone than any US politician. She’s badass AF.

Be more like Ms. Rachel!

Mutual Aid

Care for others. Share what you have. Ask for what you need.

Join your local Buy Nothing Group. Start a community fridge. Shovel driveways for the elderly and disabled in your town. Raise funds for people who’ve lost their jobs or had family members kidnapped by ICE.

Care for your neighbors. Build community.

Anti-Capitalism

This goes hand-in-hand with mutual aid, but build systems outside of capitalism.

Share. Repair. Reuse.

Buy less and save more. Support local and family owned businesses when you can.

Be weird

There is almost nothing fascism hates more than people who won’t toe the line and conform.

Be weird. It’s a good form of virtue signaling.

Dye your hair. Paint your nails. Wear what you want. Ignore artificial norms around gender expression and what “people your age” should wear and what “looks attractive.”

It’s all bullshit. It’s all made up. None of it is real. None of it matters. It’s all about control.

So fuck that. Do what you want. Be weird!

Like this? A Lean Web Club membership is the best way to support my work and help me create more free content.

What if you read things around the web the way you watch movies or listen to music?

A couple of days ago I made a post on Mastodon introducing lettrss.com, a project that takes a book in the public domain and sends one chapter a day to your RSS reader.

Xinit replied with a great point about RSS feed management:

This is fascinating, but I know how it would go based on the thousands of unread RSS feeds I’ve had, and the thousands of unheard podcasts I subscribed to. I’d end up with an RSS of unread chapters, representing a whole book in short order.

Regardless of my inability to deal, it remains a great idea, and I will absolutely recommend while hiding my shame of a non-zero inbox.

When I first started using RSS, I thought I’d found this great tool for keeping tabs on news, current events, and stuff I should and do care about.

After adding newspapers, blogs, magazines, publications, YouTube channels and release notes from software I use, I felt a false sense of accomplishment, like I’d finally been able to wrangle the craziness of the internet into a single app, like I had rebelled against the algorithm™️.

But it didn’t take long to accumulate hundreds of posts, most of which I had no true desire to read, and soon after I abandoned my RSS reader. I came back to check on it from time to time, but its dreadful little indicator of unread posts felt like a personal failure, so eventually I deleted it entirely.

Will Hopkins wrote a great post on this exact feeling.

I don’t actually like to read later:

I used Instapaper back in the day, quite heavily. I built up a massive backlog of items that I’d read occasionally on my OG iPod Touch. At some point, I fell off the wagon, and Instapaper fell by the wayside.

[…] The same thing has happened with todo apps over the years, and feed readers. They become graveyards of good intentions and self-imposed obligations. Each item is a snapshot in time of my aspirations for myself, but they don’t comport to the reality of who I am.

I couldn’t have said it better myself. This only happens with long-form writing, whenever I come across an essay or blog post that I know will either require my full attention or a bit more time than I’m willing to give it in the moment.

I’ve never had that issue with music. Music is more discrete. It’s got a timestamp. I listen to music through moods and seasons, so much so that I make a playlist for every month of the year like a musical scrapbook.

What if we took this approach to RSS feeds?

Here’s what I replied to Xinit:

This is something I find myself struggling with too.

I think I’m okay knowing some RSS feeds are seasonal, same as music genres throughout the year. Some days I want rock, others I want jazz.

Similarly with RSS feeds, I’ve become comfortable archiving and resurfacing feeds.

For reference, I follow around 10 feeds at any given time, and the feeds I follow on my phone are different from the ones on my desktop.

You shouldn’t feel guilty about removing feeds from your RSS readers. It’s not a personal failure, it’s an allocation of resources like time and attention.

Further Reading

Marco Arment, Sane RSS Usage:

RSS is a great tool that’s very easy to misuse. And if you’re subscribing to any feeds that post more than about 10 items per day, you’re probably misusing it. I don’t mean that you’re using it in a way it wasn’t intended — rather, you’re using it in a way that’s not good for you.

This article was produced with support from WIRED.

A Meta executive in charge of building the company’s metaverse products told employees that they should be using AI to “go 5x faster” according to an internal message obtained by 404 Media . 

“Metaverse AI4P: Think 5X, not 5%,” the message, posted by Vishal Shah, Meta’s VP of Metaverse, said (AI4P is AI for Productivity). The idea is that programmers should be using AI to work five times more efficiently than they are currently working—not just using it to go 5 percent more efficiently.

“Our goal is simple yet audacious: make Al a habit, not a novelty. This means prioritizing training and adoption for everyone, so that using Al becomes second nature—just like any other tool we rely on,” the message read. “It also means integrating Al into every major codebase and workflow.” Shah added that this doesn’t just apply to engineers. “I want to see PMs, designers, and [cross functional] partners rolling up their sleeves and building prototypes, fixing bugs, and pushing the boundaries of what's possible,” he wrote. “I want to see us go 5X faster by eliminating the frictions that slow us down. And 5X faster to get to how our products feel much more quickly. Imagine a world where anyone can rapidly prototype an idea, and feedback loops are measured in hours—not weeks. That's the future we're building.”

Meta’s metaverse products, which CEO Mark Zuckerberg renamed the company to highlight, have been a colossal timesink and money pit, with the company spending tens of billions of dollars developing a product that relatively few people use. 

Zuckerberg has spoken extensively about how he expects AI agents to write most of Meta’s code within the next 12 to 18 months. The company also recently decided that job candidates would be allowed to use AI as part of their coding tests during job interviews. But Shah’s message highlights a fear that workers have had for quite some time: That bosses are not just expecting to replace workers with AI, they are expecting those who remain to use AI to become far more efficient. The implicit assumption is that the work that skilled humans do without AI simply isn’t good enough. At this point, most tech giants are pushing AI on their workforces. Amazon CEO Andy Jassy told employees in July that he expects AI to completely transform how the company works—and lead to job loss. "In the next few years, we expect that this will reduce our total corporate workforce as we get efficiency gains from using AI extensively across the company," he said. 

Many experienced software engineers feel like AI coding agents are creating a new crisis, where codebases contain bugs and errors that are difficult to fix since humans don’t necessarily know how specific code was written or what it does. This means a lot of engineers have become babysitters who have to fix vibe coded messes  written by AI coding agents. 

In the last few weeks, a handful of blogs written by coders have gone viral, including ones with titles such as: “Vibe coding is creating braindead coders,” “Vibe coding: Because who doesn’t love surprise technical debt!?,” “Vibe/No code Tech Debt,” and “Comprehension Debt: The Ticking Time Bomb of LLM-Generated Code.” 

In his message, Shah said that “we expect 80 percent of Metaverse employees to have integrated AI into their daily work routines by the end of this year, with rapid growth in engineering usage and a relentless focus on learning from the time and output we gain.” He went on to reference a series of upcoming trainings and internal documents about AI coding, including two “Metaverse day of AI learning” events.

“Dedicate the time. Take the training seriously. Share what you learn, and don’t be afraid to experiment,” he added. “The more we push ourselves, the more we’ll unlock. A 5X leap in productivity isn’t about small incremental improvements, it’s about fundamentally rethinking how we work, build, and innovate.” He ended the post with a graphic featuring a futuristic building with the words “Metaverse AI4P Think 5X, not 5%” superimposed on top. 

A Meta spokesperson told 404 Media “it's well-known that this is a priority and we're focused on using AI to help employees with their day-to-day work."

💬 Join the discussion on kottke.org →

Armin Ronacher, who is quite a well-known open source developer (especially in the Python community) and who would probably consider himself a very rational, pragmatic person when it comes to “AI” usage posted an interesting post on Mastodon today:

Post by @mitsuhiko@hachyderm.io

View on Mastodon

So his thought is that giving especially younger, less experienced developers “AI” assistants makes sense to get their feet wet, get into contributing to open source but that the quality of the “AI” systems is not there yet to be used by experienced developers who also might have to shoulder a lot more responsibility for the overall quality of a product.

I do absolutely disagree but one has to admit that this sounds very reasonable at first glance. I have literally heard the same argument made by a computer science professor, just not about computer science. And that conversation made it 100% clear that the argument does not hold up.

At the end of last year I was part (don’t ask me who or how, sometimes things just happen) of a sorta academic debate/workshop thingy on “AI” and its influence on learning and culture and creativity etc.

I was in a session with a professor for computer science who also is a musician (I forgot the instrument he plays): His argument was that giving people access to service like Suno (an “AI” company that lets you generate a song from a prompt) is democratizing because learning to play an instrument or even compose a song is really hard and takes a long time. (This is coincidentally exactly the argument the Suno CEO made at the beginning of this year: “It’s not really enjoyable to make music now […] It takes a lot of time, it takes a lot of practice, you need to get really good at an instrument or really good at a piece of production software. I think the majority of people don’t enjoy the majority of the time they spend making music.“)

So this professor’s hope was that if Suno let’s people generate music quicker they will get into it and then actually learn how to do it properly – either for the love of creative expression or just because at some point generating boring Suno stuff is no longer enough. There is one problem though. He had argued against that just minutes before.

He and another professor were minutes before talking about how teaching students to program no longer works the way it used to. The mode that I also had to go through when I was still a student is that you have a course that gives you the theory and concepts and an accompanying seminar in which you get increasingly difficult programming tasks to do either by yourself or in a group to actually put what you learned in the lecture in practice: The thought is that you cannot learn programming just by reading about it. It is a practice. Which means you need to practice. And it sucks to admit that: That’s the reason I failed my first programming course. I didn’t do the homework because I thought it was dumb to do these weird programming tasks. So I failed the test.

Students these days don’t get into the practice of programming because there’s always a “Copilot” or “Assistant” basically doing the work for them. And it absolutely works because the tasks you get while learning to program are dumb and keep repeating. Because you need to learn a handful of things and there’s just not too much use in trying to find a new way of expressing them so the plagiarism engine doesn’t generate the solution.

You cannot give people tasks to practice because they’ll hand in generated answers. That’s not the students’ fault: They are young and inexperienced, they have been told for years at that point that they need to get their degree as quickly as possible to get a job and their reason to study CS is to get a good job. It’s not a profession it’s just a random certificate that could mean more money doing some boring job (ideally as consultant so you never have to actually get involved or bear the consequences of any decision).

But what both professors who teach at different German universities confirmed is that the students who get through their courses using “assistance” will in fact not learn to actually program. Because why would they? They are optimizing towards getting their certificate and spending time to learn to program when some generator can give you code that works well enough for a passing grade without having to spend nights figuring shit out is a waste. It does not align with their goals. (And again: That’s not the fault of the students. That’s on us for shaping our education system into something not about intellectual curiosity and development of skills but into a form of certification generator that might get you a well-paid job.)

People who you teach that something comes from nothing will (mostly) not start doing “the same” thing in a more complex way, a way the demands more off of them. Because it’s hard to develop an appreciation of the things doing something can give you of the different quality of the product if you never had to touch the product in the first place.

LLMs teach you that you do not need to engage with any substance. That if you can think it, it will appear. Now people with more experience will instantly see that the quality often is garbage: Essays written with LLM help are often bland and boring, code contains structural problems, Suno music is soul-crushingly boring.

The best things in life are often acquired tastes. Sure (almost) everyone loves sweet stuff but having developed more of a complex appreciation of the nuances of what is possible allows you to experience a world of wonders that “just put more sugar in” doesn’t allow. But it’s a bit of an uphill battle. Understanding and appreciation takes time and engagement. You need to be present and actually interact with the world.

And this is where we come back to code. Learning to code is fucking annoying. It’s hard and you make a lot of mistakes. And when looking at experienced people’s code you often feel like a dumbass. It’s frustrating. And I am all for making the frustration less: Give people mentorship based on positive feedback and appreciation of the learner’s work and effort.

But that’s not what LLMs provide. Or promise. LLMs promise that you can be a part of a community and all the status and clout that comes with without putting in the work. And I do not see that there is a path to mend that foundational issue.

So in summary I don’t agree with Armin that his position is the rational or mature position. I think it’s still based on a significant amount of hopium, on a narrative about the world that does not align with what we see in reality.