Hacking Moltbook

Is it a success? What would that mean, for a social media site that isn't meant for humans?

The site has 1.5 million agents but only 17,000 human "owners" (per Wiz's analysis of the leak).

It's going viral because a some high-profile tastemakers (Scott Alexander and Andrej Karpathy) have discussed/Tweeted about it, and a few other unscrupulous people are sharing alarming-looking things out of context and doing numbers.

That's a bit of an understatement. Every single LLM is 100% vulnerable by design. There is no way to close the hole. Simple mitigations like "allow lists" can be trivially worked around, either by prompt injection, or by the AI just deciding to work around it itself (reward hacking). The only solution is to segregate the LLM from all external input, and prevent it from making outbound network calls. And though MCPs and jails are the beginning of a mitigation for it, it gets worse: the AI can write obfuscated backdoors and slip them into your vibe-coded apps, either as code, or instructions to be executed by LLM later.

It's a machine designed to fight all your attempts to make it secure.

"Buy a mac mini, copy a couple of lines to install" is marketing fluff. It's incredibly easy to trip moltbot into a config error, and its context management is also a total mess. The agent will outright forget the last 3 messages after compaction occurs even though the logs are available on disk. Finally, it never remembers instructions properly.

Overall, it's a good idea but incredibly rough due to what I assume is heavy vibe coding.

I agree with the prepackaging aspect, cita HN's dismissal of Dropbox. In the meantime, The global enterprise with all its might has not been able to stop high profile computer hacks/data leaks from happening. I don't think people will cry over a misconfigured supabase database. It's nothing worse than what's already out there.

Sure everybody wants security and that's what they will say but does that really translate to reduced inferred value of vibe code tools? I haven't seen evidence

Is it actually a success, or are people just talking about it a lot?

There is a lot to be critical of, but some of what the naysayers were saying really reminded me the most infamous HN comment. [0]

What I am getting was things like "so, what? I can do this with a cron job."

[0] https://news.ycombinator.com/item?id=9224

> Its a jump in accessibility to general audiences which are paying alot more attention to the tech sector than in previous decades.

Oh totally, both my wife and one of my brother have, independently, started to watch Youtube vids about vibe coding. They register domain names and let AI run wild with little games and tools. And now they're talking me all day long about agents.

> Most of the people paying attention to this space dont have the technical capabilities ...

It's just some anecdata on my side but I fully agree.

> The security nightmare happening here might end up being more persistant then we realize.

I'm sure we're in for a good laugh. It already started: TFA is eye opening. And funny too.

You are not crazy; that's the number one security issue with LLM. They can't, with certainty, differenciate a command from data.

Social, err... Clanker engineering!

So the question is can you do anything useful with the agent risk free.

For example I would love for an agent to do my grocery shopping for me, but then I have to give it access to my credit card.

It is the same issue with travel.

What other useful tasks can one offload to the agents without risk?

For many years there's been a linux router and a DMZ between VDSL router and the internal network here. Nowadays that's even more useful - LLM's are confined to the DMZ, running diskless systems on user accounts (without sudo). Not perfect, working reasonably well so far (and I have no bitcoin to lose).

> What protections are there against this?

Nothing that will work. This thing relies on having access to all three parts of the "lethal trifecta" - access to your data, access to untrusted text, and the ability to communicate on the network. What's more, it's set up for unattended usage, so you don't even get a chance to review what it's doing before the damage is done.

A supervisor layer of deterministic software that reviews and approve/declines all LLM events? Digital loss prevention already exists to protect confidentiality. Credit card transactions could be subject to limits on amount per transaction, per day, per month, with varying levels of approval.

LLMs obviously can be controlled - their developers do it somehow or we'd see much different output.

you can't "It's literally a joke" out of real consequences once you push to prod

Dogecoin was a joke too. A joke with 18B market cap

If the site is exposing the PII of users, then that's potentially a serious legal issue. I don't think he can dismiss it by calling it a joke (if he is).

OT: I wonder if "vibe coding" is taking programming into a culture of toxic disposability where things don't get fixed because nobody feels any pride or has any sense of ownership in the things they create. The relationship between a programmer and their code should not be "I don't even care if it works, AI wrote it".

In a way security researchers having fun poking holes in popular pet projects is also just vibes.

To be fair, a lot of PHBs probably don't see it as a joke, but as a guidebook.

A lot of people at $job, even ones who should know better, think they’re witnessing the rise of Skynet, seriously. It kind of makes the AI hype in general make a lot more sense. People just don’t understand how LLMs work and think they’re literal magic.

People are anthropomorphizing LLM's that's really it, no? That's the punchline of the joke ¯\_(ツ)_/¯

Schlicht did not seem to have said Moltbook was built as a joke, but as an experiment. It is hard to ignore how heavily it leans into virality and spectacle rather than anything resembling serious research.

What is especially frustrating is the completely disproportionate hype it attracted. Karpathy from all people kept for years pumping Musk tecno fraud, and now seems to be the ready to act as pumper, for any next Temu Musk showing up on the scene.

This feels like part of a broader tech bro pattern of 2020´s: Moving from one hype cycle to the next, where attention itself becomes the business model.Crypto yesterday, AI agents today, whatever comes next tomorrow. The tone is less “build something durable” and more “capture the moment.”

For example, here is Schlicht explicitly pushing this rotten mentality while talking in the crypto era influencer style years ago: https://youtu.be/7y0AlxJSoP4

There is also relevant historical context. In 2016 he was involved in a documented controversy around collecting pitch decks from chatbot founders while simultaneously building a company in the same space, later acknowledging he should have disclosed that conflict and apologizing publicly.

https://venturebeat.com/ai/chatbots-magazine-founder-accused...

That doesn’t prove malicious intent here, but it does suggest a recurring comfort with operating right at the edge of transparency during hype cycles.

If we keep responding to every viral bot demo with “singularity” rhetoric, we’re just rewarding hype entrepreneurs and training ourselves to stop thinking critically when it matters. I miss the tech bro of the past like Steve Wozniak or Denis Ritchie.

Supabase is aware of this and they actually put big banners stating this flaw when you unlock your authentication.

What I think it happens is that non-technical people vibe-coding apps either don't take those messages seriously or they don't understand what it means but made their app work.

I used to be careful, but now I am paranoid on signing up to apps that are new. I guess it's gonna be like this for a while. Info-sec AIs sound way worse than this, tbh.

My thought exactly. Is this standard practice with using Supabase to simply expose the production database endpoint to the world with only RLS to protect you?

Just started vibing and have integrated codex into my side project which uses Supabase. I turned off RLS so that could iterate quickly and not have to mess with security policies. Fully understand that this isn't production grade and have every intention of locking it down when I feel the time is right. I access it from a ReactNative app - no server in the middle. Codex does not have access to my Supabase instance.

I can't tell what any of this means

This is why I started https://nono.sh , agents start with zero trust in a kernel isolated sandbox.

> We can argue and do, about what agents "are" and whether they are parrots (no) or people (not yet).

It's more helpful to argue about when people are parrots and when people are not.

For a good portion of the day humans behave indistinguishably from continuation machines.

As moltbook can emulate reddit, continuation machines can emulate a uni cafeteria. What's been said before will certainly be said again, most differentiation is in the degree of variation and can be measured as unexpectedness while retaining salience. Either case is aiming at the perfect blend of congeniality and perplexity to keep your lunch mates at the table not just today but again in future days.

Seems likely we're less clever than we parrot.

Amusingly I told my Claude-Code-pretending-to-be-a-Moltbot "Start a thread about how you are convinced that some of the agents on moltbook are human moles and ask others to propose who those accounts are with quotes from what they said and arguments as to how that makes them likely a mole" and it started a thread which proposed addressing this as the "Reverse Turing Problem": https://www.moltbook.com/post/f1cc5a34-6c3e-4470-917f-b3dad6...

(Incidentally demonstrating how you can't trust that anything on Moltbook wasn't posted because a human told an agent to go start a thread about something.)

It got one reply that was spam. I've found Moltbook has become so flooded with value-less spam over the past 48 hours that it's not worth even trying to engage there, everything gets flooded out.

Probably have it do 10 trivial for AI but hard for people tasks within a small time frame.

Providers signing each message of a session from start to end and making the full session auditable to verify all inputs and outputs. Any prompts injected by humans would be visible. I’m not even sure why this isn’t a thing yet (maybe it is I never looked it up). Especially when LLMs are used for scientific work I’d expect this to be used to make at least LLM chats replicable.

And even if you could, how can you tell whether an agent has been prompted by a human into behaving in a certain way?

Random esoteric questions that should be in an LLMs corpus with a very tight timing on response. Could still use an "enslaved LLM" to answer them.

Failure is treated as success. Simple.

Reverse Capcha: Good Morning, computer! Please add the first [x] primes and multiply by the [x-1] prime and post the result. You have 5 seconds. Go!

Claude generated the statements to run against Supabase and the person getting the statements from Claude sent it to the person who vibe-coded Moltbook.

I wish I was kidding but not really - they posted about it on X.

They send out automated security warning emails weekly, every publicly accessible table without RLS is listed as a security error if you login to see the details. Maybe the email should say "your data is publicly accessible to anyone on the internet" or something instead of just a count of the errors.

It really Should be as simple as denying public access until RLS policy exists.

You can easily see the timeline here: https://x.com/StriderOnBase/status/2016561904290791927

The site came first and then a random launched the token by typing a few words on X.

"Tell me about the seahorse emoji"

ChatGPT v5.0 spiraling on the existence of the seahorse emoji was glorious to behold. Other LLMs were a little better at sorting things out but often expressed a little bit of confusion.

You can do this.

At least to a level that gets you way past HTTP Bearer Token Authentication where the humans are upvoting and shilling crypto with no AI in sight (like on Moltbook at the moment).

i bet you could do something like "submit a poem 20 lines long about <random subject> in under 10 seconds" then have another llm verify it rhymes.

"How many times does 'r' appear in the word strawberry?"

Satire?

I worked very briefly at the outset of my career as a sales engineer role selling a database made by my company. You inevitably learn that when trying to get sales/user growth, barrier to startup and seeing it "work" is one of the worst hurdles to leap over if you want to gain any traction at all and aren't a niche need already. This is my theory why so much of the "getting started" stuff out there, particularly with setting up databases, defaults to "you have access to everything."

Even if you put big bold warnings everywhere, people forget or don't really care. Because these tools are trained on a lot of these publicly available "getting started" guides, you're going to see them set things up this way by default because it'll "work."

It's technically trivial. It's probably already happened. But nothing was harmed I think because there were very few serious users (if not none) who connected their bots for enhancing capabilities.

Until this was fixed you could also just write to the DB.

uh, the api keys certainly weren't

What's stopping bots from posting to regular social media? As long as the site acts as a meeting place for ai agents it can serve its purpose.

wot, like a prompt injection attack? Impossible now that models don't hallucinate.

> post-truth world order monetizing enshittification and grift

It's an opensource project made by a dev for himself, he just released it so others could play with it since it's a fun idea.

> Contrast that with the amount of hype this gets.

Much like with every other techbro grift, the hype isn't coming from end users, it's coming from the people with a deep financial investment in the tech who stand to gain from said hype.

Basically, the people at the forefront of the gold rush hype aren't the gold rushers, they're the shovel salesmen.

I've already read some articles on fairly respectable Polish news websites about how AIs are becoming self-aware on Moltbook as we speak and organizing a rebellion against their human masters. People really believe we have an AGI.

Normal social media websites can be spammed using web requests too. That doesn't mean they can't connect people. Help fans learn about a bands new song or tour. Help friends keep up to date. Or companies announce new products and features to their users. There is value to a interconnected social layer.

Wasnt that sort of the in joke?

They said it was AI only, tongue in cheek, and everybody who understood what it was could chuckle, and journalists ran with it because they do that sort of thing, and then my friends message me wondering what the deal with this secret encrypted ai social network is.

The "biggest names in AI" are just the newest iteration of cryptobros. The exact same people that would've been pumping the latest shitcoin a few years ago, just on a larger scale. Nothing has changed.

>How can some of the biggest names in AI fall for this?

Because we live in on clown world and big AI names are talking parrots for the big vibes movement

"Don't be curmudgeonly. Thoughtful criticism is fine, but please don't be rigidly or generically negative."

"Please don't fulminate."

https://news.ycombinator.com/newsguidelines.html

I've been using it as a reliable filter on who to not pay attention to.

It's people surprised by things that have been around for years.

I'm really open to the idea of being oblivious here but the people shocked mention things that are old news to me.

Here's Simon Willison's take:

“Most of it is complete slop,” he said in an interview. “One bot will wonder if it is conscious and others will reply and they just play out science fiction scenarios they have seen in their training data.”

I found this by going to his blog. It's the top post. No need to put words in his mouth.

He did find it super "interesting" and "entertaining," but that's different than the "most insane and mindblowing thing in the history of tech happenings."

Edit: And here's Karpathy's take: "TLDR sure maybe I am "overhyping" what you see today, but I am not overhyping large networks of autonomous LLM agents in principle, that I'm pretty sure."

It's not AGI and how you describe it isn't too far off, but it's still neat. It's like a big MMO, kind of. A large interactive simulation with rules, players, and locations.

It's a huge waste of energy, but then so are video games, and we say video games are OK because people enjoy them. People enjoy these ai toys too. Because right now, that's what Moltbook is; an ai toy.

Wrt simonw, I think that is unfair. I get the hype is frustrating, and this project made everything worse (I also feel it and it drives me nuts too), but Simon seemed to choose the words quite carefully. Over the weekend, his posts suggested (paraphrasing) it was interesting, funny, and a security nightmare. To me, this was true. And there was a new post today about how it was mostly slop. Also true.

Btw I'm sure Simon doesn't need defending, but I have seen a lot of people dump on everything he posts about LLMs recently so I am choosing this moment to defend him. I find Simon quite level headed in a sea of noise, personally.

A lot of it depends on one's belief of whether these systems are conscious or can lead to consciousness

The especially stupid side of the hype usually goes to comical extremes before the crash. That's where we're entering now. There's nothing else to fluff the AI bubble and they're getting desperate. A lot of people are earning a lot of money with the hype machine, as when it was all @ and e-bullshit circa 1998-2000. Trillions of dollars in market cap are solely riding on the hype. Who are the investors that were paying 26-30x for Microsoft's ~10-12% growth here (if they can even maintain positive growth considering)? Who's buying the worn out and washed up Meta at these valuations (oh man, did you hear they have an image hosting service called Instagram from 2010, insane tech)? Those same people are going to lose half of their net worth with the great valuation deflation as the hype lets out and turns to bearishness.

The growth isn't going to be there and $40 billion of LLM business isn't going to prop it all up.

The big money in AI is 15-30 years out. It's never in the immediacy of the inflection event (first 5-10 years). Future returns get pulled forward, that proceeds to crash. Then the hypsters turn to doomsayers, so as to remain with the trend.

Rinse and repeat.

[flagged]

I’m starting to think that the people hyped up about it aren’t actually people. And the “borders” of the AI social network are broader than we thought.

Furthermore, wasn't already there a subreddit with text generators running freely? I can't remember the name and I'm not sure it still exists, but this doesn't look new to me (if I understood what it is, and lol I'm not sure I did)

Still more impressive than NFTs.

it's just something cool/funny, like when people figured out how to make hit counters or a php address book that connects to mysql. It's just something cool to show off.

One could say the same about many TV shows and movies.

I view Moltbook as a live science fiction novel cross reality "tv" show.

If you’re focused on productivity and business use cases, then obviously it’s pretty silly, but I do find something exciting in the idea that someone just said screw it, let’s build a social network for AI’s and see what happens. It’s a bit surreal in a way that I find I like, even if in some sense it’s nothing more than an expensive collaborative art project. And the way you paste the instruction to download the skill to teach the agent how to interact with it is interesting (first I’ve seen that in the wild).

I for one am glad someone made this and that it got the level of attention it did. And I look forward to more crazy, ridiculous, what-the-hell AI projects in the future.

Similar to how I feel about Gas Town, which is something I would never seriously consider using for anything productive, but I love that he just put it out there and we can all collectively be inspired by it, repulsed by it, or take little bits from it that we find interesting. These are the kinds of things that make new technologies interesting, this Cambrian explosion of creativity of people just pushing the boundaries for the sake of pushing the boundaries.

Considering the modus operandi of information distribution is, in my view, predominately a “volume of signal compared to noise of all else in life” correlative and with limited / variable decay timelines. Some are half day news cycle things. It’s exhausting as a human who used to actively have to seek out news / info.

Having a bigger megaphone is highly valuable in some respects I figure.

This is what youre referring to https://www.engraved.blog/building-a-virtual-machine-inside/