llm/302a36fb-79e1-4f4b-b047-e145d20e4497/batch-0-f7b4bfbb-a88b-4378-af1a-593a80adf159-input.json
The following is content for you to classify. Do not respond to the comments—classify them.
<topics>
1. CMU Database Group Teaching
Related: Praise for CMU's eccentric teaching style including gangsta intros, DJ sets before lectures, and unique course materials on YouTube covering database internals for building systems
2. SQLite Production Usage
Related: Discussion of SQLite's viability in production, WAL mode for concurrent writes, single-file simplicity, Litestream backups, limitations for multi-user systems, and comparisons to traditional databases
3. DuckDB Use Cases
Related: Enthusiasm for DuckDB's columnar storage, JSON handling, WASM support, S3 integration, and use as analytical complement to SQLite for OLAP workloads
4. SQLite-DuckDB Integration
Related: Interest in combining SQLite for writes/OLTP with DuckDB for reads/analytics, discussing watermarks, sync strategies, and latency tradeoffs between row and columnar storage
5. MCP Security Concerns
Related: Skepticism about MCP database access opposing least privilege principles, risks of unfettered LLM access, hallucination-driven SQL injection, and need for guardrails and monitoring
6. Immutable Bi-temporal Databases
Related: Advocacy for XTDB and Datomic for fintech compliance, discussion of audit requirements, time-travel queries, and lack of production-ready options in this category
7. PostgreSQL vs MySQL Popularity
Related: Debate over metrics measuring database popularity, distinguishing installed base from new project adoption, noting momentum shift toward PostgreSQL despite MySQL's larger deployment footprint
8. Embedded Database Benefits
Related: Discussion of local databases without network overhead, caching implications, RAM management differences from server databases, and when to migrate to PostgreSQL
9. MySQL Project Concerns
Related: Commentary on Oracle firing MySQL open-source team, project becoming rudderless, MariaDB financial problems, and potential impact on ecosystem
10. Database Consolidation Trends
Related: Concern about software development gravitating toward same tools like PostgreSQL and React, loss of diversity and nuance in technical decisions
11. JSON in Databases
Related: Appreciation for JSON field support in modern databases, arrow functions in SQLite, and DuckDB's superior JSON handling with columnar extraction
12. EdgeDB/Gel Acquisition Impact
Related: Disappointment about Gel sunsetting after Vercel acquisition, appreciation for EdgeQL language design, and discussion of community fork efforts
13. Time Series Databases
Related: Questions about time series database developments, mentions of QuestDB, ClickHouse's experimental time series engine, and need for InfluxDB alternatives
14. Enterprise Database Omissions
Related: Noting absence of Oracle, MS SQL Server, DB2 from article despite being top-ranked databases, discussion of boring enterprise tech that powers critical systems
15. Database Caching Strategies
Related: Discussion of PostgreSQL's built-in caching benefits versus SQLite requiring custom read caching, Redis/memcached integration, and CDN layer caching
16. Write Scalability Patterns
Related: Analysis of SQLite's write throughput capabilities, serial write handling, edge sharding with Cloudflare D1, and when single-node architecture suffices
17. Vector Database Developments
Related: Brief mentions of Milvus features for RAG, vector indexing in DuckDB, and general traction of vector databases in AI ecosystem
18. Nested Transactions for Agents
Related: Technical discussion of MVCC databases providing isolated snapshots for agent playgrounds, nested transaction support, and preventing accidental commits
19. File Format Competition
Related: Interest in new formats challenging Parquet including Vortex, F3, AnyBlox, discussion of format interoperability problems and WASM decoder approaches
20. TiDB Momentum
Related: Question about TiDB adoption in Silicon Valley as OLTP/OLAP hybrid, seeking commentary on its position in database landscape
0. Does not fit well in any category
</topics>
<comments_to_classify>
[
{
"id": "46496573",
"text": "Maybe off-topic but,\n\nIf you're not familiar with the CMU DB Group you might want to check out their eccentric teaching style [1].\n\nI absolutely love their gangsta intros like [2] and pre-lecture dj sets like [3].\n\nI also remember a video where he was lecturing with someone sleeping on the floor in the background for some reason. I can't find that video right now.\n\nNot too sure about the context or Andy's biography, I'll research that later, I'm even more curious now.\n\n[1] https://youtube.com/results?search_query=cmu+database\n\n[2] https://youtu.be/dSxV5Sob5V8\n\n[3] https://youtu.be/7NPIENPr-zk?t=85"
}
,
{
"id": "46497668",
"text": "Indeed, I was delighted when I read the part about wutang's time capsule and obviously OP is a wu-tang and general hip hop fan. The intro you shared is dope!"
}
,
{
"id": "46500710",
"text": "I can't understand if their \"intro to database systems\" is an introductory (undergrad) level course or some advanced course (as in, introduction to database (internals)).\n\nAnyone willing to clarify this? I'm quite weak at database stuff, i'd love to find some undergrad-level proper course to learn and catch up."
}
,
{
"id": "46503630",
"text": "It is an undergrad course, though it is cross-listed for masters students as well. At CMU, the prerequisites chain looks like this: 15-122 (intro imperative programming, zero background assumed, taken by first semester CS undergrads) -> 15-213 (intro systems programming, typically taken by the end of the second year) -> 15-445 (intro to database systems, typically taken in the third or fourth year). So in theory, it's about one year of material away from zero experience."
}
,
{
"id": "46502173",
"text": "It's the internals.\n\nHe is training up people to work on new features for existing databases, or build new ones.\n\nNot application developers on how to use a database.\n\nKnowing some of the internals can help application developers make better decisions when it comes to using databases though."
}
,
{
"id": "46506542",
"text": "Here is the playlist: https://www.youtube.com/playlist?list=PLSE8ODhjZXjYMAgsGH-Gt...\n\nYou can tell from the topics, it's related to building databases, not using them."
}
,
{
"id": "46504230",
"text": "(I consed \"https://\" onto your links so they'd become clickable. Hope that's ok!)"
}
,
{
"id": "46497305",
"text": "While the author mentions that he just doesn't have the time to look at all the databases, none of the reviews of the last few years mention immutable and/or bi-temporal databases.\n\nWhich looks more like a blind spot to me honestly. This category of databases is just fantastic for industries like fintech.\n\nTwo candidates are sticking out.\nhttps://xtdb.com/blog/launching-xtdb-v2 (2025)\nhttps://blog.datomic.com/2023/04/datomic-is-free.html (2023)"
}
,
{
"id": "46499766",
"text": "> none of the reviews of the last few years mention immutable and/or bi-temporal databases.\n\nWe hosted XTDB to give a tech talk five weeks ago:\n\nhttps://db.cs.cmu.edu/events/futuredata-reconstructing-histo...\n\n> Which looks more like a blind spot to me honestly.\n\nWhat do you want me to say about them? Just that they exist?"
}
,
{
"id": "46502137",
"text": "Nice work Andy. I'd love to hear about semantic layer developments in this space (e.g. Malloy etc.). Something to consider for the future. Thanks."
}
,
{
"id": "46502699",
"text": "> I'd love to hear about semantic layer developments in this space (e.g. Malloy etc.)\n\nWe also hosted Llyod to give a talk about Malloy in March 2025:\n\nhttps://db.cs.cmu.edu/events/sql-death-malloy-a-modern-open-..."
}
,
{
"id": "46501172",
"text": "You can get pretty far with just PG using tstzrange and friends: https://www.postgresql.org/docs/current/rangetypes.html\n\nOtherwise there are full bitemporal extensions for PG, like this one: https://github.com/hettie-d/pg_bitemporal\n\nWhat we do is range types for when a row applies or not, so we get history, and then for 'immutability' we have 2 audit systems, one in-database as row triggers that keeps an on-line copy of what's changed and by who. This also gives us built-in undo for everything. Some mistake happens, we can just undo the change easy peasy. The audit log captures the undo as well of course, so we keep that history as well.\n\nThen we also do an \"off-line\" copy, via PG logs, that get shipped off the main database into archival storage.\n\nWorks really well for us."
}
,
{
"id": "46498591",
"text": "People are slow to realize the benefit of immutable databases, but it is happening. It's not just auditability; immutable databases can also allow concurrent reads while writes are happening, fast cloning of data structures, and fast undo of transactions.\n\nThe ones you mentioned are large backend databases, but I'm working on an \"immutable SQLite\"...a single file immutable database that is embedded and works as a library: https://github.com/radarroark/xitdb-java"
}
,
{
"id": "46497644",
"text": "I see people bolting temporality and immutability onto triple stores, because xtdb and datomic can't keep up with their SPARQL graph traversal. I'm hoping for a triple store with native support for time travel."
}
,
{
"id": "46498247",
"text": "Lance graph?"
}
,
{
"id": "46503220",
"text": "FYI I made a comment very similar to yours, before reading yours. I'll put it here for reference. https://news.ycombinator.com/item?id=46503181"
}
,
{
"id": "46497406",
"text": "Why fintech specifically?"
}
,
{
"id": "46497836",
"text": "Destructive operations are both tempting to some devs and immensely problematic in that industry for regulatory purposes, so picking a tech that is inherently incapable of destructive operations is alluring, I suppose."
}
,
{
"id": "46497683",
"text": "I would assume that it's because in fintech it's more common than in other domains to want to revert a particular thread of transactions without touching others from the same time."
}
,
{
"id": "46498079",
"text": "Not only transactions - but state of the world."
}
,
{
"id": "46497784",
"text": "compliance requirements mostly (same for health tech)"
}
,
{
"id": "46497524",
"text": "Because, money."
}
,
{
"id": "46502948",
"text": "XTDB addresses a real use-case. I wish we invested more in time series databases actually: there's a ton of potential in a GIS-style database, but 1D and oriented around regions on the timeline, not shapes in space.\n\nThat said, it's kind of frustrating that XTDB has to be its own top-level database instead of a storage engine or plugin for another. XTDB's core competence is its approach to temporal row tagging and querying. What part of this core competence requires a new SQL parser?\n\nI get that the XTDB people don't want to expose their feature set as a bunch of awkward table-valued functions or whatever. Ideally, DB plugins for Postgres, SQLite, DuckDB, whatever would be able to extend the SQL grammar itself (which isn't that hard if you structure a PEG parser right) and expose new capabilities in an ergonomic way so we don't end up with a world of custom database-verticals each built around one neat idea and duplicating the rest.\n\nI'd love to see databases built out of reusable lego blocks to a greater extent than today. Why doesn't Calcite get more love? Is it the Java smell?"
}
,
{
"id": "46504599",
"text": "> it's kind of frustrating that XTDB has to be its own top-level database instead of a storage engine or plugin for another. XTDB's core competence is its approach to temporal row tagging and querying. What part of this core competence requires a new SQL parser?\n\nMany implementation options were considered before we embarked on v2, including building on Calcite. We opted to maximise flexibility over the long term (we have bigger ambitions beyond the bitemporal angle) and to keep non-Clojure/Kotlin dependencies to a minimum."
}
,
{
"id": "46497115",
"text": "From my perspective on databases, two trends continued in 2025:\n\n1: Moving everything to SQLite\n\n2: Using mostly JSON fields\n\nBoth started already a few years back and accelerated in 2025.\n\nSQLite is just so nice and easy to deal with, with its no-daemon, one-file-per-db and one-type-per value approach.\n\nAnd the JSON arrow functions make it a pleasure to work with flexible JSON data."
}
,
{
"id": "46497145",
"text": "From my perspective, everything's DuckDB.\n\nSingle file per database, Multiple ingestion formats, full text search, S3 support, Parquet file support, columnar storage. fully typed.\n\nWASM version for full SQL in JavaScript."
}
,
{
"id": "46498780",
"text": "This is a funny thread to me because my frustration is at the intersection of your comments: I keep wanting sqlite for writes (and lookups) and duckdb for reads. Are you aware of anything that works like this?"
}
,
{
"id": "46499015",
"text": "DuckDB can read/write SQLite files via extension. So you can do that now with DuckDB as is.\n\nhttps://duckdb.org/docs/stable/core_extensions/sqlite"
}
,
{
"id": "46499246",
"text": "My understanding is that this is still too slow for quick inserts, because duckdb (like all columnar stores) is designed for batches."
}
,
{
"id": "46499452",
"text": "The way I understood it, you can do your inserts with SQLite \"proper\", and simultaneously use DuckDB for analytics (aka read-only)."
}
,
{
"id": "46499705",
"text": "Aha! That makes so much sense. Thank you for this.\n\nEdit: Ah, right, the downside is that this is not going to have good olap query performance when interacting directly with the sqlite tables. So still necessary to copy out to duckdb tables (probably in batches) if this matters. Still seems very useful to me though."
}
,
{
"id": "46500307",
"text": "Analytics is done in \"batches\" (daily, weekly) anyways, right?\n\nWe know you can't get both, row and column orders at the same time, and that continuously maintaining both means duplication and ensuring you get the worst case from both worlds.\n\nLocal, row-wise writing is the way to go for write performance. Column-oriented reads are the way to do analytics at scale. It seems alright to have a sync process that does the order re-arrangement (maybe with extra precomputed statistics, and sharding to allow many workers if necessary) to let queries of now historical data run fast."
}
,
{
"id": "46500937",
"text": "Not all olap-like queries are for daily reporting.\n\nI agree that the basic architecture should be row order -> delay -> column order, but the question (in my mind) is balancing the length of that delay with the usefulness of column order queries for a given workload. I seem to keep running into workloads that do inserts very quickly and then batch reads on a slower cadence (either in lockstep with the writes, or concurrently) but not on the extremely slow cadence seen in the typical olap reporting type flow. Essentially, building up state and then querying the results.\n\nI'm not so sure about \"continuously maintaining both means duplication and ensuring you get the worst case from both worlds\". Maybe you're right, I'm just not so sure. I agree that it's duplicating storage requirements, but is that such a big deal? And I think if fast writes and lookups and fast batch reads are both possible at the cost of storage duplication, that would actually be the best case from both worlds?\n\nI mean, this isn't that different conceptually from the architecture of log-structured merge trees, which have this same kind of \"duplication\" but for good purpose. (Indeed, rocksdb has been the closest thing to what I want for this workload that I've found; I just think it would be neat if I could use sqlite+duckdb instead, accepting some tradeoffs.)"
}
,
{
"id": "46502002",
"text": "> the question (in my mind) is balancing the length of that delay with the usefulness of column order queries for a given workload. I seem to keep running into workloads that do inserts very quickly and then batch reads on a slower cadence (either in lockstep with the writes, or concurrently) but not on the extremely slow cadence seen in the typical olap reporting type flow. Essentially, building up state and then querying the results.\n\nI see. Can you come up with row/table watermarks? Say your column store is up-to-date with certain watermark, so any query that requires freshness beyond that will need to snoop into the rows that haven't made it into the columnar store to check for data up to the required query timestamp.\n\nIn the past I've dealt with a system that had read-optimised columnar data that was overlaid with fresh write-optimised data and used timestamps to agree on the data that should be visible to the queries. It continuously consolidated data into the read-optimised store instead of having the silly daily job that you might have in the extremely slow cadence reporting job you mention.\n\nYou can write such a system, but in reality I've found it hard to justify building a system for continuous updates when a 15min delay isn't the end of the world, but it's doable if you want it.\n\n> I'm not so sure about \"continuously maintaining both means duplication and ensuring you get the worst case from both worlds\". Maybe you're right, I'm just not so sure. I agree that it's duplicating storage requirements, but is that such a big deal? And I think if fast writes and lookups and fast batch reads are both possible at the cost of storage duplication, that would actually be the best case from both worlds?\n\nI mean that if you want both views in a consistent world, then writes will bring things to a crawl as both, row and column ordered data needs to be updated before the writing lock is released."
}
,
{
"id": "46502786",
"text": "Yes! We're definitely talking about the same thing here! Definitely not thinking of consistent writes to both views.\n\nNow that you said this about watermarks, I realize that this is definitely the same idea as streaming systems like flink (which is where I'm familiar with watermarks from), but my use cases are smaller data and I'm looking for lower latency than distributed systems like that. I'm interested in delays that are on the order of double to triple digit milliseconds, rather than 15 minutes. (But also not microseconds.)\n\nI definitely agree that it's difficult to justify building this, which is why I keep looking for a system that already exists :)"
}
,
{
"id": "46498843",
"text": "I think you could build an ETL-ish workflow where you use SQLite for OLTP and DuckDB for OLAP, but I suppose it's very workload dependent, there are several tradeoffs here."
}
,
{
"id": "46499263",
"text": "Right. This is what I want, but transparently to the client. It seems fairly straightforward, but I keep looking for an existing implementation of it and haven't found one yet."
}
,
{
"id": "46498931",
"text": "very interesting. whats the vector indexing story like in duckdb these days?\n\nalso are there sqlite-duckdb sync engines or is that an oxymoron"
}
,
{
"id": "46500539",
"text": "https://duckdb.org/docs/stable/core_extensions/vss\n\nIt's not bad if you need something quick. I haven't had a large need of ANN in duckdb since it's doing more analytical/exploratory needs, but it's definitely there if you need it."
}
,
{
"id": "46498835",
"text": "From my perspective - do you even need a database?\n\nSQLite is kind-of the middle ground between a full fat database, and 'writing your own object storage'. To put it another way, it provides 'regularised' object access API, rather than, say, a variant of types in a vector that you use filter or map over."
}
,
{
"id": "46499245",
"text": "If I would write my own data storage I would re-implement SQLite. Why would I want to do that?"
}
,
{
"id": "46502937",
"text": "Not sure if this is quite what you are getting at, but the SQLite folks even mention this as a great use-case: https://www.sqlite.org/appfileformat.html"
}
,
{
"id": "46497179",
"text": "As a backend database that's not multi user, how many web connections that do writes can it realistically handle? Assuming writes are small say 100+ rows each?\n\nAny mitigation strategy for larger use cases?\n\nThanks in advance!"
}
,
{
"id": "46497235",
"text": "Couple thousand simultaneous should be fine, depending on total system load, whether you're running on spinning disks or on SSDs, p50/99 latency demands and of course you'd need to enable the WAL pragma to allow simultaneous writes in the first place. Run an experiment to be sure about your specific situation."
}
,
{
"id": "46506018",
"text": "You also need BEGIN CONCURRENT to allow simultaneous write transactions.\n\nhttps://www.sqlite.org/src/doc/begin-concurrent/doc/begin_co..."
}
,
{
"id": "46497386",
"text": "After 2 years in production with a small (but write heavy) web service... it's a mixed bag. It definitely does the job, but not having a DB server does have not only benefits, but also drawbacks. The biggest being (lack of) caching the file/DB in RAM. As a result I have to do my own read caching, which is fine in Rust using the mokka caching library, but it's still something you have to do yourself, which would otherwise come for free with Postgres.\nThis of course also makes it impossible to share the cache between instances, doing so would require employing redis/memcached at which point it would be better to use Postgres.\n\nIt has been OK so far, but definitely I will have to migrate to Postgres at one point, rather sooner than later."
}
,
{
"id": "46497413",
"text": "How would caching on the db layer help with your web service?\n\nIn my experience, caching makes most sense on the CDN layer. Which not only caches the DB requests but the result of the rendering and everything else. So most requests do not even hit your server. And those that do need fresh data anyhow."
}
,
{
"id": "46497486",
"text": "As I said, my app is write heavy. So there are several separate processes that constantly write to the database, but of course, often, before writing, they need to read in order to decide what/where to write. Currently they need to have their own read cache in order to not clog the database.\n\nThe \"web service\" is only the user facing part which bears the least load. Read caching is useful there too as users look at statistics, so calculating them once every 5-10 minutes and caching them is needed, as that requires scanning the whole database.\n\nA CDN is something I don't even have. It's not needed for the amount of users I have.\n\nIf I was using Postgres, these writer processes + the web service would share the same read cache for free (coming from Posgres itself). The difference wouldn't be huge if I would migrate right now, but now I already have the custom caching."
}
,
{
"id": "46498083",
"text": "I am no expert, but SQLite does have in memory store? At least for tables that need it..ofc sync of the writes to this store may need more work."
}
,
{
"id": "46497208",
"text": "Why have multiple connections in the first place?\n\nIf your writes are fast, doing them serially does not cause anyone to wait.\n\nHow often does the typical user write to the DB? Often it is like once per day or so (for example on hacker news). Say the write takes 1/1000s. Then you can serve\n\n1000 * 60 * 60 * 24 = 86 million users\n\nAnd nobody has to wait longer than a second when they hit the \"reply\" button, as I do now ..."
}
]
</comments_to_classify>
Based on the comments above, assign each to up to 3 relevant topics.
Return ONLY a JSON array with this exact structure (no other text):
[
{
"id": "comment_id_1",
"topics": [
1,
3,
5
]
}
,
{
"id": "comment_id_2",
"topics": [
2
]
}
,
{
"id": "comment_id_3",
"topics": [
0
]
}
,
...
]
Rules:
- Each comment can have 0 to 3 topics
- Use 1-based topic indices for matches
- Use index 0 if the comment does not fit well in any category
- Only assign topics that are genuinely relevant to the comment
Remember: Output ONLY the JSON array, no other text.
50