Executable Is a SQLite Database
fzakaria.com321 points by setheron 12 hours ago
321 points by setheron 12 hours ago
As far as agent workflows are concerned whats needed is the other direction. Store the files wherever but let an agent interact with it as if its on local FS via bash or other languages using their file io.
Whats worked for others?
(author) I'm enjoying the comments. When I published a short-paper with this idea in academic circles, the feedback wasn't so kind
This whole article is fantastic, but already at the start, the SQLite virtual tables thing is blowing my mind.
https://www.sqlite.org/vtablist.html
You can "mount" your filesystem (or anything else) as a SQL database, wtf. That's amazing.
This sounds like it could be extremely useful.
> The format itself is incredibly terse, designed for a world where disk space and network bandwidth was at an extreme premium. Modifying the format is hard, you often have to zero out sections and add new ones since it is packed so tightly. There is also no self-describing schema. ELF itself is a very generic format that supports sections of data that by convention are interpreted in specific ways but the format does not enforce it.
Sounds like a great use case for:
1. ELF file to SELF file
2. modify SELF file
3. SELF file to ELF file> I realized something that bugged me. ELF is already a database.
Even more broadly applied: Every base of data is already a database (that's what the compound word means). Programs like sqlite and postgres used to be called by the more precise term "Relational Data Base Management Software" or "RDBMS" until their use became so widespread that they were colloquially called databases instead.
The very first data base structures were more like, each geometric sector of a hard drive is a record, and each head is a column. This was a straightforward translation of punch-card workflow onto a magnetic disk.
Yes !
This has been on the back of my mind for a while. And, as other commenters have noted, it would be great for the file to contain the (self-modifiable) Lisp image, a builtin virtual file system, and whatever the application want to use as (runtime modifiable) extra tables.
I find SQLite dynamic linking being basically compatible with ELF dynamic linking to be very impressive, I can imagine that if well done, it cloud replace most uses of AppImages with a much more efficient format, like the author suggests.
How about an option for compressing section contents within the SQLite blobs, since the author mentioned you can't mmap directly the text pages and have to copy anyway ?
There are two extensions that I was thinking would make a SQLite executable truly unique.
First, a linking extension that would allow patching in functions and hooks more directly to allow for a very powerful plug-in system. Imagine the plug-in SQLite defining a BEFORE/AFTER/REPLACE hook for some symbol the host SQLite defines as extensible.
Second, re-linking at runtime. This would require application author cooperation because you won't be able to do that from anywhere, but imagine changing a dependency or loading a plugin at runtime through editing the db, and the interpreter just maps that on demand/automatically in the background, now next time your web server accept(), it calls the new version of the handling function.
You can do any of this in-memory wife ELF of course, it's just that nobody does and nobody's written the tools for it (the most obscure tool I know of is `ld -r`, which links two or more .o files and produces a new .o file instead of an executable)
For sure, the main advantage is that since the SQLite file format is easy to use compared to ELF, it's more likely that such experiments would be even tried at all.
I got another one: bundling multiple (completely separate) executables with shared dependencies in the same SQLite, selected by argv[0] when called, like what busybox does
The copied vs. mapped memory situation is the only deal breaker in this experiment. Otherwise, file format unification would be a big step forward. The PE/COFF executable file format used by Windows (and some older Unix systems) is a relational database as well. The same goes to .NET assembly format - it's a relational database too. The wheel gets reinvented over and over again.
I was thinking that one could just write a loader that goes from SELF->ELF on load. You then pay the cost on startup. Probably not worth it for one-shot programs like ls/cat/etc, but for long running programs and daemons it might be workable.
I can buy that an object file can be viewed as a relational db. Why SQLite though? Why not SQL query engine over the object file using a virtual table abstraction? I’m not seeing how most the SQLite features, with the exception of a subset of the query engine would translate over.
If the author wants to make a case for including schema metadata in an object file, again why SQLite? This strikes me a lot as someone who is trying to find uses for their favorite hammer (a very useful hammer I might say) rather than a serious exploration of what a new, improved object file format would look like.
(And that’s totally ok)
I wrote a tool[0] for manipulating elf files to support inline assembly for a compiler that doesn’t support it (based on a similar python tool). That required knowing any of the ELF sections that were being changed and how those changes would impact other sections.
sqlite would have made this pretty trivial. Replace the .text fields of a few rows. Insert a few rows for symbols, update a few from the old object.
The best part - there’s tons of library support for sqlite. If it was a new object format, there would be no support and I’d just have to write different parsers and generators.
The schema might the the biggest issue for efficiency. As others have noted, ELF->SELF->ELF might be the best use case for compatibility. That said, a big part of [0] was performance, and I don’t know how sqlite would have done compared to my naive elf parsing and manipulation.
> I wrote a tool[0] for manipulating elf files to support inline assembly for a compiler that doesn’t support it (based on a similar python tool). That required knowing any of the ELF sections that were being changed and how those changes would impact other sections.
Would this normally be something the compiler would handle, or would it be the linker? I guess I’m curious how this would impact any optimizations the compiler implements.
Also I’m guessing for your use case (elf edits) you would be looking at both SELECT and INSERT type operations?
There was a recent post that probably explains the details better than I could[0].
This would most like just be the concern of the compiler and assembler. In GCCs case, my understanding is it just inlines the assembly into the output assembly stream and the assembler is none the wiser. The issue is that MetroWerks doesn’t support inline assembly at the top level (for defining new function symbols), but only inline within the context of a function. That means some additional setup and tear down that shouldn’t be present may be injected instead of the raw assembly.
There’s possibly some upseet potential and the idea that it could be done in place could avoid loading the entire thing k to memory (though the files are small for modern systems).
(author) In the post I say I did write sqlelf which is just that, sqlite + virtual tables prior to this.
sqlite is a great replacement for fopen - although the C interface is much more verbose.
Oh do you mean as a way to store app state? Yeah 100%. It also helps that there are native reimplementations in many major languages, which means you don’t even need to link a libsqlite3 in contexts where it can sometimes be impractical to use the official C library (looking at you, CGO_ENABLED=0). And let’s not forget about sqlite3 running in wasm :)
I don't think this goes far enough! Make the actually app store be the same file itself. so it's a living application and the file is constantly updated to how you use it. Copy it around, and you carry your data wiht you as well.
Let's go deeper. it's a webserver app + the server code + application code + db, so pocketbase++ where it's also the deployment target.
Then combine it with APE liek system, and the same file loads and stores things on every platform. evil laugh
Very cool hacking! My hats off to the author.
The application writing its state to the executable is basically Smalltalk (or Lisp). It's an extremely powerful way of writing software, IMO.
Or Forth. But all of these are extreme spaghetti-bait too. Being able to see all the code that goes into an executable, and generate a clean one using only that code, is important for good development hygiene, which of course they didn't know yet in the 1970s. Imagine trying to use version control on a REPL - it doesn't work - you have to version control the commands that go into the REPL and running them on a fresh REPL every time.
I didn't have this idea exactly but I did play around with sqlite as an embedded database for packing ruby apps into a single file a while ago. It was less direct than this: you basically compiled all the dependencies (including .so's) into a fake filesystem that an overridden `require` loaded from, which was unpacked to a `:memory:` database at runtime from a `.data` section that got bolted onto the interpreter. It died when 1.8->1 9 changed how the build system works and I never got round to updating.
A version based on this which carries around an overlay filesystem would be comparatively straightforward, the hard work is already done.
This has been my hope.
BIOS -> Database that has everything including OS.
This way we get WORM and AI integration.
This was super fun. Dude’s been on an absolute tear recently.
NB: ELF executables and libraries don't need to have sections, or a section header table. The program headers are what matters. You can use sstrip to remove the SHT.
I appreciate the inventiveness of this idea. But I don’t find myself thinking I must have this.
Yeah... Original and clever, and a great read! But it seems mostly useful to the handful of people dealing with ELF internals, than to the vast majority of people executing ELF blissfully ignorant of its internals... Maybe if the latency and size trade-offs were the other way around it would be more appealing to the masses.
Reminds me of the DBOS, everything's a database, ethos. https://www.dbos.dev/
Isn't "everything in a database" what AS/400 and PICK operating systems do?
Then there was Microsoft WinFS, which promised much but got killed in ~2003. A shame.
I've wanted to do something like this for quite a while! Of course I wanted to make my own OS, language and DB from scratch in order to do it, so naturally I've never gotten around to it. It goes to show that there is nothing new under the sun.
an executable that is also a database means my bugs get relational now. instead of a segfault i get a foreign key constraint violation.
A lot of kernel filesystem features should be inside a database as well.
Yes. In this interview Michael Stonebreaker, one of the original developers of PostgreSQL, discusses (among other topics) replacing the file system with a database :
I want to see a kernel managed by NETCONF.
Not the XML protocol, that's annoying (although usable) but the structured data model.
Congratulations, you’ve rediscovered mainframes.
I wonder how flexible is the SQLite binary format to allow to design a tool that would take a SELF binary/db and rewrite it to make BLOB values more mmap-able (OS page aligned) in order to allow shortcuts in the self-exec loader.
You'd have to at least increase the page size to allow 4kB of consecutive data without interfering with the page header.
OR you restructure the code and data to work around the page headers.
Or you could write a new SQLite VFS that somehow separates the page headers from the page data.
This is really creative and clever, nicely done!
The “Inception” graphic was hilarious and made me wonder if perhaps the site itself is served by some SELF binary (is it?? that would be insane)
Great read. I always find these turn a feature of a computer into a database an interesting read/idea. It certainly has upside, and downsides. For example having an entire filesystem as a sqlite database etc, or in this case an executable. It would be nice to have a structured way of talking to our utilities, rather than interpreting various commands as json through a few layers of tools to then get the output in a common format. That said I'd probably rather have a split up tool-set, than everything having to be in a single database for it to interoperate.
Two tools come to mind that you might be interested in
- osquery: virtual table abstraction over proc fs. https://www.osquery.io/
- nushell: shell with structured I/O. https://www.nushell.sh/
We do have a single database, it's the filesystem. But quite importantly it's a very composable kind of database, because you can "section off" a part of the database, calling it a folder, and the folder looks like a whole filesystem of its own. Working in the folder is like working in a separate database that's visible from the outside but the outside isn't visible from the inside unless you deliberately choose to go out through the escape hatch. SQL isn't like that - it's a flat namespace and that is like the original filesystems that didn't have folders. Those were usable when a computer was only big enough to hold a few activities at a time and when you had a stack of disks next to your computer, each with a different program's filesystem.
A flat namespace might actually work with an app-based OS model (like on android) where everything that happens is associated with an app ID and apps mostly can't talk to each other. But there's a reason nerds don't do nerd stuff on smartphones.
I was thinking that someone managed to put an ELF header in an SQLite file while keeping it compatible with SQLite... but no, "just" a new binary format. Very interesting nevertheless!
Unless I missed something. The author queries the bin in one of the examples using sqlite3.
The author instructs the OS to execute it natively my adding it as a binary format.
So where does this lead to? Is it leading us towards SQLite as some kind of alternative container image format? Because the benefit seems to only occur when it incorporates dependencies
The word "Your" was dropped from the start of the title for some reason.
Well, for the usual reason :-)
I’m wondering if the anti-clickbait filter could benefit from some lightweight LLM integration. E.g. when the submission hits the front page, if the title was “de-clickbaited” and wasn’t changed manually yet, run it through a cheap model and see if that was the right call.
Maybe "your" at the beginning of a title should be replaced by "my" instead of getting stripped.
That would preserve the non-clickbait meaning of all such titles I can spontaneously think of.
“SQLite database as an executable format”? A bit farther away from the original but captures the meaning and sounds more neutral.
The handling of dynamic libraries and closing over them into single dbs made this a fun read.
What if, we put the sqlite driver in kernel space? So this is your only filesystem like a unikernel
It's a lot of machinery to put into a kernel.
(Even with ELF, a kernel can have a relatively simple parser and loader and leave more challenging work to user space... like relocations, dynamic libraries, etc.)
tbh execve should be userspace operation instead of syscall, idk why kernel needs to care about the executable format at all
not sure if the people exploring binaries prefer sql over command line tools
SQL is so much more consistent and introspective though. No need to remember which flags are for readelf and which are for objdump.
Yeah, I have been amazed for an entire life how many tools the IT industry invented during last 50 years to just disguise a database.
OK, in '70s, '80s or '90s when compute and storage resources were limited and every bit counted, specialized formats did make sense.
But nowadays we'd save enormous efforts by just packaging stuff in SQLite databases. Microsoft's proprietary file formats (Office, Power BI etc.), OpenOffice/LibreOffice OpenDocument format, or almost everything else would fit perfectly.
Documents, files, are all data (called "data files", aren't they?), including executables, which as can be seen in this article are also databases.
Some of it is convergent evolution at work. 60s/70s/80s specialized formats as much predated modern database designs and data structures as coexisted with them. SQLite is the culmination of decades of database design. Some of that was influenced by those specialized formats of previous decades that SQLite could now in theory replace.
(Which is to say a lot of the specialized formats didn't exist just because of compute and storage resource limitations, but also because training/research/standards work came later. Plus complications like software patents. ZIP files have been around since the 1990s but ZIP only became a common general use container format for "everything" almost exactly 14 years later when patents finally expired.)
At first I thought it was still for a word document to be saved in SQL but then I thought about embedded images and it makes complete sense. Makes more sense for a Publisher/Impress app though as they have more self-contained objects. In Word/Writer the main text is one huge run of characters, how would you sqlize that effectively?
[dead]
[flagged]