PyPI in 2025: A Year in Review
blog.pypi.org54 points by miketheman 10 hours ago
54 points by miketheman 10 hours ago
One of the big companies making billions on Python software should step up and fund the infrastructure needed to enable PyPI package search via the CLI, like you could with `pip search` in the past.
I upvoted you because I broadly agree with you, but search is never coming back in the API. They previously outlined the cost involved and there's no way, given how minimal the value it gives more broadly, it's coming back ant time soon. It's basically an abusive vector because of the compute cost.
Serious question: how important is `pip search` to your workflows? I don’t think I ever used it, back when PyPI still had an XMLRPC search endpoint.
(I think the biggest blocker on CLI search isn’t infrastructure, but that there’s no clear agreement on the value of CLI search without a clear scope of what that search would do. Just listing matches over the package names would be less useful than structured metadata search for example, but the latter makes a lot of assumptions about the availability of structured metadata!)
Funding could help, but it still requires PyPI/Warehouse to ship and operate a new public search interface that is safe at internet scale.
They operate a public package hosting interface, how is a search one any harder?
PyPI responses are cached at 99% or higher, with less infrastructure to run.
Search is an unbounded context and does not lend itself to caching very well, as every search can contain anything
Pypi has fewer than one million projects. The searchable content for each package is what? 300 bytes? That's a 200mb index. You don't even need fancy full text search, you could literally split the query by word and do a grep over a text file. No need for elasticsearch or anything fancy.
And anyway, hit rates are going to be pretty good. You're not taking arbitrary queries, the domain is pretty narrow. Half the queries are going to be for requests, pytorch, numpy, httpx, and the other usual suspects.
I wonder how a PyPi search index could be statically served and locally evaluated on `pip search`?
PyPI servers would have to be constantly rebuilding a central index and making it available for download. Seems inefficient
Great work!
Side issue: anyone else seeing that none of the links in the article work? They're all 404s.
Great work Dustin and team!
This seems to suggest once the bubble pops, it will take Python down with it. The next AI winter will definitely replace Lisp with Python.