Jellyfin LLM/"AI" Development Policy
jellyfin.org174 points by mmoogle 7 hours ago
174 points by mmoogle 7 hours ago
I know this is nothing new, but it's insane that we need policies like "When talking to us you have to use human words, not copy pasted LLM output" and "You must understand the code you're committing."
When I was young, I used to think I'd be open minded to changing times and never be curmudgeonly, but I get into one "conversation" where someone responds with ChatGPT, and I am officially a curmudgeon.
Brazen usage of LLM output is a disrespect to the target audience to begin with. If I'm being expected to employ the mental capital needed to understand the context and content of your writings, I at the very least expect that you did the same when actually authoring it.
It also feels like using one of those cereal encoder wheels, to some degree. If someone sends me 10 paragraphs of output from chatGPT, and they only wrote a sentence to prompt it, then the output is really just a re-encoding of the information in the original prompt.
Quite literally - if they sent me the text of the prompt I could obtain the same output, so the output is just a more verbose way of stating the prompt.
I find it really disrespectful to talk to people through an LLM like that.
Generally speaking, a person can write a long rambling email without much effort. It takes some work to distill it down to keep the meaning without the verbosity.
If anything, AI should be used to take the long rambling email and send off the shorter distilled version.
I hope it becomes as accepted as it is to stick cameras in random people's faces: generally seen as rude, and bad actors who do it anyway are desperate and considered as such.
I am capable of copying and pasting shit into an LLM, do not give me its output and don't insult me by pretending the output is your own work.
They’ll self-sort pretty quickly. The ChatGPT people will talk to the ChatGPT people and be happy about it. I think it’ll work out.
> LLM output is expressly prohibited for any direct communication
I would like to see this more. As a heavy user of LLMs I still write 100% of my own communication. Do not send me something an LLM wrote, if I wanted to read LLM outputs, I would ask an LLM.
I’m glad they have a carve out for using LLMs to translate to, or fix up English communications. LLMs are a great accessibility tool that is making open source development truly global. Translation and grammar fix up is something LLMs are very, very good at!
But that is translation, not “please generate a pull request message for these changes.”
Machine translation is best used on the receiving end. Let me decide if I want to run your message through a machine, or read it with my own skills.
"I just used it to clean up my writing" seems to be the usual excuse when someone has generated the entire thing and copy pasted it in. No one believes it and it's blatantly obvious every time someone does this.
Not sure what you're talking about. Quite often I've written out a block of information and have found chunks of repeats or what would be hard to interpret by other stuck here or there. I'll stick it in an llm and have it suggest changes.
Simply put you seem to live in a different world where everyone around you has elegant diction. I have people I work with that if I could I would demand they take what they write and ask "would this make sense to any other human on this planet".
There are no shortages of people being lazy with LLMs, but at the same time it is a tool with valid and useful purpose.
Sometimes I ramble for a long time and ask an LLM to clean it up. It almost always slopifies it to shreds. Can't extract the core ideas, matches everything to the closest popular (i.e. boring to read) concept, etc.
Using software for translation is fine as long as the original source is also present for native speakers to check and any important information that is machine translated should be read by humans to test
It doesn’t hurt, but honestly machine translation (using LLMs) is so insanely good now. It usually does a better job than people.
Better to use Google Translate for this than ChatGPT. Either ChatGPT massively changes the text and slopifies it, or people are lying about using it for translation only because the outputs are horrendous. Google Translate won't fluff out the output with garbage or reformat everything with emoji.
"Translate this from X to X, don't change any meaning or anything else, only translate the text with idiomatic usage in target language: X"
Using Google Translate probably means you're using a language model in the end anyways behind the scenes. Initially, the Transformer was researched and published as an improvement for machine translation, which eventually led to LLMs. Using them for translation is pretty much exactly what they excel at :)
Google Translate uses GPTs under the hood. GPT was invented by Google’s machine translation team. I think you are misunderstanding my point.
Yep. If you don't know the language, it's best not to pretend you do.
I've done this kind of thing, even if I think it's likely they speak English. (I speak zero Japanese here.) It's just polite and you never know who's going to be reading it first.
> Google翻訳を使用しました。問題が発生した場合はお詫び申し上げます。貴社のウェブサイトにコンピュータセキュリティ上の問題が見つかりました。詳細は下記をご覧ください。ありがとうございます。
> I have found a computer security issue on your website. Here are details. Thank you.
Why would you want to use a chat bot to translate? Either you know the source and destination language, in which case you'll almost certainly do a better job (certainly a more trustworthy job), or you don't, in which case you shouldn't be handling translations for that language anyway.
Same with grammar fixes. If you don't know the language, why are you submitting grammar changes??
For translating communications like "Here is my PR, it does x, can you please review it", not localisation of the app.
No, I think GP means grammar fixes to your own communication. For example if I don't speak Japanese very well and I want to write to you in Japanese, I might write you a message in Japanese, then ask an LLM to fix up my grammar and check my writing to make sure I'm not sounding like a complete idiot.
I have read a lot of bad grammar from people who aren't very good at the language but are trying their best. It's fine. Just try to express yourself clearly and we figure it out.
I have read text where people who aren't very good at the language try to "fix it up" by feeding it through a chat bot. It's horrible. It's incredibly obvious that they didn't write the text, the tone is totally off, it's full of obnoxious ChatGPT-isms, etc.
Just do your best. It's fine. Don't subject your collaborators to shitty chat bot output.
Agreed. Humans are insanely good at figuring out intent and context, and running stuff through an LLM breaks that.
The times I've had to communicate IRL in a language I don't speak well, I do my best to speak slowly and enunciate and trust they'll try their best to figure it out. It's usually pretty obvious what you're asking lol. (Also a lot of people just reply with "Can I help you?" in English lol)
I've occasionally had to email sites in languages I don't speak (to tell them about malware or whatever) and I write up a message in the simplest, most basic English I can. I run that through machine translation that starts out with "This was generated by Google Translate" and include both in the email.
Just do your best to communicate intent and meaning, and don't worry about sounding like an idiot.
> Humans are insanely good at figuring out intent and context
I wish that was true.
You seem to be judging business communications by weird middle-class aesthetics while the people writing the emails are just trying to be clear.
If you think that every language level is always sufficient for every task (a fluency truther?), then you should agree that somebody who writes an email in a language that they are not confident in, puts it through an LLM, and decides the results better explain the idea they were trying to convey than they had managed to do is always correct in that assessment. Why are you second guessing them and indirectly criticizing their language skills?
Running your words through ChatGPT isn't making you clear. If your own words are clear enough to be understood by ChatGPT, they're clear enough to be understood by your peers. Adding ChatGPT into the mix only ensures opportunity for meaning to be mangled. And text that's bad enough as to be ambiguous may be translated to perfectly clear text that reflects the wrong interpretation of your words, risking misunderstandings that wouldn't happen if the ambiguity was preserved instead of eliminated.
I have no idea what you're talking about with regard to being a "fluency truther", I think you're putting words into my mouth.
Eh, na dawg, I'll have to reject a lot of what you've typed here.
LLMs can do a lot of proof checking on what you've written. Asking it to check for logical contradictions in what I've stated and such. It will catch were I've forgot things like a 'not' in one statement so one sentence is giving a negative response and another gives a positive response unintentionally. This kind of error is quite often hard for me to pick up on, yet the LLM seems to do well.
I completely agree. I let LLMs write a ton of my code, but I do my own writing.
It's actually kind of a weird "of two minds" thing. Why should I care that my writing is my own, but not my code?
The only explanation I have is that, on some level, the code is not the thing that matters. Users don't care how the code looks, they just care that the product works. Writing, on the other hand, is meant to communicate something directly from me, so it feels like there's something lost if I hand that job over to AI.
I often think of this quote from Ted Chiang's excellent story The Truth Of Fact, The Truth of Feeling:
> As he practiced his writing, Jijingi came to understand what Moseby had meant: writing was not just a way to record what someone said; it could help you decide what you would say before you said it. And words were not just the pieces of speaking; they were the pieces of thinking. When you wrote them down, you could grasp your thoughts like bricks in your hands and push them into different arrangements. Writing let you look at your thoughts in a way you couldn’t if you were just talking, and having seen them, you could improve them, make them stronger and more elaborate.”
But there is obviously some kind of tension in letting an LLM write code for me but not prose - because can't the same quote apply to my code?
I can't decide if there really is a difference in kind between prose and code that justifies letting the LLM write my code, or if I'm just ignoring unresolved cognitive dissonance because automating the coding part of my job is convenient.
To me, you are describing a fluency problem. I don't know you or how fluent you are in code, but what you have described is the case where I have no problem with LLMs: translating from a native language to some other language.
If you are using LLMs to precisely translate a set of requirements into code, I don't really see a problem with that. If you are using LLMs to generate code that "does something" and you don't really understand what you were asking for nor how to evaluate whether the code produced matched what you wanted, then I have a very big problem with that for the same reasons you outline around prose: did you actually mean to say what you eventually said?
Of course something will get lost in any translation, but that's also true of translating your intent from brain to language in the first place, so I think affordances can be made.
Relevant: https://noslopgrenade.com
What do you recommend if I've been regularly producing blog-length posts in Slack for years, no LLM present? It's where I write man...should I quit that out? I try to be information dense...
Yeah I use LLMs to show me how to shorten my emails because I can type for days. It helps a lot for when I feel like I just need a short concise email but I still write it all myself.
Yeah I do the same. I’ve seen great results from writing out a large slack, copying it into ChatGPT and saying “write this more succinctly”.
Then, of course, I review the output and make some manual edits here and there.
That last thing is the key in both written communication and in code, you HAVE to review it and make manual edits if needed.
I see this on Reddit a lot. Someone will vibe code something then spam a bunch of subreddits with LLM marketing text. It’s all low effort low quality sooo.
I mean this is how spammers have always worked. If spam were high quality and useful people wouldn't complain about it.
yeah, you could ask a LLM, but are you sure you know what to ask?
like in that joke with the mechanic which demands $100 for hitting the car once with his wrench
Same can go for LLM code. I don't want to review your code if it was written by an LLM
I only use LLM to write text/communication because that's the part I don't like about my work
I think at some point we will need a "PEP-8" for LLM / AI code contributions document that is universally reusable and adoptable per project, call it an "Agent Policy" or what have you, that any agent worth its Salt should read before touching a codebase and warn the user that their contributions might not be accepted or what have you, depending on project policy, just like we have GPL, BSD, MIT, etc it would probably make sense to have it, especially for those of us who are respectful to a projects needs and wishes. I think there's definitely room for sane LLM code / vibe coded code, but you have to put in a little work to validate your changes, run every test, ensure that you understand the output and implications, not just shove a PR at the devs and hope they accept it.
A lot of the time open source PRs are very strategic pieces of code that do not introduce regressions, an LLM does not necessarily know or care, and someone vibe coding might not know the projects expectations. I guess instead of / aside from a Code of Conduct, we need a sort of "Expectation of Code" type of document that covers the projects expectations.
> that any agent worth its Salt should read before touching a codebase and warn the user that their contributions might not be accepted
Are you talking about some agent that is specific for writing FOSS code or something? Otherwise I don't see why we'd want all agents to act like this.
As always, it's the responsibility of the contributor to understand both the code base and contributing process, before they attempt to contribute. If they don't, then you might receive push-back, or have your contribution deleted, and that's pretty much expected, as you're essentially spamming if you don't understand what you're trying to "help".
That someone understands this before contributing, is part of understanding how FOSS works when it's about collaborating on projects. Some projects have very strict guidelines, others very lax, and it's up to you to figure out what exactly they expect from contributors.
I'm not sure when this policy was introduced, but fairly recently Jellyfin released a pretty major update that introduced a lot of bugs and performance issues. I've been watching their issue tracker as they work through them and have noticed it's flooded with LLM generated PRs and obviously LLM generated PR comments/descriptions/replies. A lot of the LLM generated PRs are a mishmash of 2-8 different issues all jumbled into a single PR.
I can see how frustrating it is to wade through those and they are distracting and taking time away from them actually getting things fixed up.
We've had these thoughts for a while, especially relating to clients, but that is exactly what prompted this - a huge number of pure-vibe-coded "fixes performance" PRs that have been a nightmare to wade through.
I have lately taken to this approach when I raise bugs:
1. Fully human-written explanation of the issue with all the info I can add
2. As an attachment to the bug (not a PR), explicitly noted as such, an AI slop fix and a note that it makes my symptom go away.
I've been on the receiving end of one bug report in this format and I thought it was pretty helpful. Even though the AI fix was garbage, the fact that the patch made the bug go away was useful signal.
The open for anyone PR model might be at risk now. How can maintainers be expected to review unlimited slop coming in. I can see a lot of open source just giving up on allowing community contribution. Or maybe only allowing trusted members to contribute after they have demonstrated more than passing interest in the project.
It has been at risk for a long time, now it is in doubt.
Think of a scenario like
Attacker floods you with tons of AI slop to make your overloaded and at risk of making mistakes. These entries should have just enough basis in reality to avoid summary rejection.
Then the attacker puts in useful batch of code that fixes issues and injects a tricky security flaw.
If there's not a lot going on the second part is hard to pull off. But if you ruin the SnR it becomes more likely.
That's not going to be the scenario (IMO). After the AI slop comes in, everything in the queue is going to be triaged as garbage to clear it.
The key to get better quality AI PR is to add high quality Agents.md file to tell the LLM what are the patterns, conventions, etc.
We do that internally and I cant overstate how much better the output is even with small prompts.
IMO things like "dont put abusive comments" as a policy is better in that file, you will never see comment again instead of fighting with dozen of bad contributions.
There was a discussion recently on the Wikimedia wikitech-l discussion list, and one participant had a comment I appreciated:
>I'm of the opinion if people can tell you are using an LLM you are using it wrong.
They continued:
>It's still expected that you fully understand any patch you submit. I think if you use an LLM to help you nobody would complain or really notice, but if you blindly submit an LLM authored patch without understanding how it works people will get frustrated with you very quickly.
<https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists...>
I suspect the vast number of individuals in developing countries currently spamming LLM commits to every open source project on earth, and often speak neither the project or programming language are not going to pay much attention to this policy. It’s become a numbers game of automation blasting “contributions” at projects with name recognition and hoping you sneak one in for your resume/portfolio.
Policy is not put in place to prevent anything. Policy is put in place so that you have a sign to point at while you lock a PR thread.
In other words, you are responsible for the code you submit (or cause to be submitted via automated PRs), regardless of how fancy your tools are.
That said I understand calling it out specifically. I like how they wrote this.
Related:
> https://news.ycombinator.com/item?id=46313297
> https://simonwillison.net/2025/Dec/18/code-proven-to-work/
> Your job is to deliver code you have proven to work
Most of these seem to be applicable to any development. Don't submit PRs that you can't explain. I would hope they have that standard for all submissions.
This has actually come up in our internal discussions while we were drafting this, and the truth is, yea, this applies to "normal" PRs and such as well. But we weren't having any sort of problem with someone who has no understanding of the code at all coming in and making extensive changes. That simply wasn't possible. But LLMs enable someone with no knowledge to submit an extensive, on-the-surface-sensible PR that then needs literally hours of review work and testing.
Seems perfectly legit and hopefully it will help creating new contributors that learn and understand what the AI helped them generate.
This seems fair, tbh. And I fully agree on the policy for issues/discussions/PRs.
I know there will probably be a whole host of people from non-English-speaking countries who will complain that they are only using AI to translate because English is not their first (or maybe even second) language. To those I will just say: I would much rather read your non-native English, knowing you put thought and care into what you wrote, rather than reading an AIs (poor) interpretation of what you hoped to convey.
Although: "An exception will be made for LLM-assisted translations if you are having trouble accurately conveying your intent in English."
I am quite obviously blind, but I still stand by my sentiment. I would rather have a "bad" but honest PR body than a machine translated one where the author isn't sure about what it says. How will you know if what it says is what you meant?
I think the spirit of the policy also allows you to write your own words in your own language and have an AI translate it.
(But also, for a majority of people old fashioned Google Translate works great).
(Edit: it's actually a explicit carveout)
> LLM/"AI"
love the "AI" in quotes
It's incongruous to me to put "AI" in scare quotes while allowing it to be used. It is intelligent.
It's plain as day that there's no intelligence whatsoever in LLMs. Time and again they fall flat on their face with tests that no human would ever fail (like the "how many r's are in strawberry" thing), because they can't actually understand anything. I think it's perfectly fair to put "AI" in scare quotes.
I find people rarely have useful definitions for intelligence and the ontological units clustered around the term change significantly from person to person.
That said, LLMs have a single specific inductive bias: Translation. But not just between languages, between ontologies themsleves. Whether it’s 'Idea -> Python' or 'Intent -> Prose,' the model is performing a cross-modal mapping of conceptual structures. This does require a form of intelligence, of reasoning, just in a format suitable to a world so alien to our own that they're mutually unintelligble, even if the act of charting ontologies is shared between them.
This is why I think we’re seeing diminishing returns, it is that we’re trying to 'scale' our way into AGI using a map-maker/navigation system. Like asking google maps to make you a grocery list, rather than focusing on its natural purpose in being able to tell you where you can find groceries. You can make a map so detailed it includes every atom, but the map will never have the agency to walk across the room. We are seeing asymptotic gains because each extra step toward 'behavioral' AGI is exponentially more expensive when you're faking reasoning through high-dimensional translation.
Whenever I've trained clients in AI use, I've tried to strongly recommend using GenAI as a "Learning Accelerator" as opposed to a "Learning Replacement".
GenAI can be incredibly helpful for speeding up the learning process, but the moment you start offloading comprehension, it starts eroding trust structures.
>LLM output is expressly prohibited for any direct communication
One more reason to support the project!!
What's the grief with squashing commits? I do it all the time when I'm working on stuff so that I don't have to expose people to my internal testing. So long as the commit(s) look fine at the end of the day, I don't see what the deal is there.
Seems like both "show your working" and "make it easier for us to review" are the reasons. Seems reasonable to me.
"Commit 1: refactor the $THING to enable $CAPABILITY"
"Commit 2: redirect $THING2 to communicate with $THING1"
"Commit 3: add error handling for $EdgeCase" --- long explanation in commit body
A single commit with no commentary just offloads the work to the maintainers. It's their project so their rules.
As a user, I am like this decision.
I very much like the no LLM output in communication. Nothing is worse than getting huge body of text the sender clearly hasn't even read. Then you either have to ignore it or spend 15 minutes explaining why their text isn't even relevant to the conversation.
Sort of related, Plex doesn't have a desktop music app, and the PlexAmp iOS app is good but meh. So I spent the weekend vibe coding my own Plex music apps (macOS and iOs), and I have been absolutely blown away at what I was able to make. I'm sure code quality is terrible, and I'm not sure if a human would be able to jump in there and do anything, but they are already the apps I'm using day-to-day for music.
Good AI policies (like this one) can be spotted since the TLDR is "Don't submit shitty code". As such, good AI policies should be replaced by "Contribution policies" that says "Don't submit shitty code".
I think the gist and the "virality" of this policy is:
1) we accept good quality LLM code
2) we DO NOT accept LLM generated human interaction, including PR explanation
3) your PR must explain well enough the change in the description
Which summed together are far more than "no shitty code". It's rather no shitty code that YOU understand
> 1) we accept good quality LLM code
there is no such thing as LLM code. code is code, the same standards have always applied no matter who or what wrote it. if you paid an indian guy to type out the PR for you 10 years ago, but it was submitted under your name, its still your responsibility.
I don't agree at all. There's a huge difference between "someone wrote this code and at least understands the intention and the problem it's trying to solve" and "the chat bot just generated this code, nobody understands what the intention is". I'm comfortable having a conversation with a human about code they wrote. It's pointless to have a conversation with a human about code they didn't write and don't understand.
The quality of "does the submitter understand the code" is not reflected in the text of the diff itself, yet is extremely important for good contributions.
I mean, it's frequently the case that guidelines for new situations are really just a reapplication of existing principles. But often specificity is needed so people realize which guidelines are applicable.
> Violating this rule will result in closure/deletion of the offending item(s).
Should just be an instant perma-ban (along with closure, obviously).
> Should just be an instant perma-ban (along with closure, obviously).
This is the internet. Real offenders will just submit the next PR with a new alt account.
its still an added inconvenience and means they cant use AI to farm reputation on a single account
Seems a bit disproportionate. I'd say that's more of a "repeat offender" type of solution.
Submitting a pure slop PR and description is a very high level offense that is obviously not acceptable.
Whats disproportionate is the mountains of slop out there and the amount of people think they can just sling slop for cheap online cred.
Once might just be a script kiddie not knowing any better. More than once is a script kiddie refusing to know any better.
These seem fair, but it's the type of framework that really only catches egregious cases — people using the tools appropriately will likely slip through undetected.
People can write horrible PRs manually just as well as they do with AI (see Hacktoberfest drama, etc).
"LLM Code Contributions to Official Projects" would read exactly the same if it just said "Code Contributions to Official Projects": Write concise PRs, test your code, explain your changes and handle review feedback. None of this is different whether the code is written manually or with an LLM. Just looks like a long virtue signaling post.
Virtue signaling? That seems like an uncharitable reading.
The point, and the problem, is volume. Doing it manually has always imposed a de facto volume limit which LLMs have effectively removed. Which I understand to be the problem these types of posts and policies are designed to address.
The effort to write shitty code is way less when you are using IA, you can create a 1k lines PR with a single prompt. This policy is important because no one is saying "we hate AI" but instead advises developers to use it with responsibility. This is coming in time since many people are using it without understanding problems and not being accountable regarding the contributions.
A large enough difference in degree becomes a difference in kind. Chat bots have vastly inflated the amount of shitty PRs, to the degree that it needs different solutions to manage.
Exactly. We never had a problem with spammy PRs before. Even at the height of Hacktoberfest, the vast majority were painfully obvious and confined to documentation. It was easy and obvious to reject those. But LLMs have really changed the game, and this policy was explicitly prompted by a number of big PRs that were obviously purely vibe-coded and we felt we really needed to get a defined policy out that we could point to and say "no, this is why we're rejecting this".