Pypi has fewer than one million projects. The searchable content for each package is what? 300 bytes? That's a 200mb index. You don't even need fancy full text search, you could literally split the query by word and do a grep over a text file. No need for elasticsearch or anything fancy.
And anyway, hit rates are going to be pretty good. You're not taking arbitrary queries, the domain is pretty narrow. Half the queries are going to be for requests, pytorch, numpy, httpx, and the other usual suspects.
2. apt repositories are cryptographically signed, centrally controlled, and legally accountable.
3. apt search is understood to be approximate, distro-scoped, and slow-moving. Results change slowly and rarely break scripts. PyPI search rankings change frequently by necessity
4. Turning PyPI search into an apt-like experience would require distributing a signed, periodically refreshed global metadata corpus to every client. At PyPI’s scale, that is nontrivial in bandwidth, storage, and governance terms
5. apt search works because the repository is curated, finite, and opinionated
The install side is basically Merkle-friendly (immutable artifacts, append-only metadata, hashes, mirrors).
Search isn’t. Search results are derived, subjective, and frequently rewritten (ranking tweaks, spam/malware takedowns, popularity signals). That’s more like constantly rebasing than appending commits.
You can Merklize “what files exist”; you can’t realistically Merklize “what should rank for this query today” without freezing semantics and turning CLI search into a hard API contract.
The searchable context for a distribution on PyPI is unbounded in the general case, assuming the goal is to allow search over READMEs, distribution metadata, etc.
(Which isn’t to say I disagree with you about scale not being the main issue, just to offer some nuance. Another piece of nuance is the fact that distributions are the source of metadata but users think in terms of projects/releases.)
>The searchable context for a distribution on PyPI is unbounded in the general case, assuming the goal is to allow search over READMEs, distribution metadata, etc.
> assuming the goal is to allow search over READMEs, distribution metadata, etc.
Why would you build a dedicated tool for this instead of just using a search engine? If I'm looking for a specific keyword in some project's very long README I'm searching kagi, not npm.
I'd expect that the most you should be indexing is the data in the project metadata (setup.py). That could be unbounded but I can't think of a compelling reason not to truncate it beyond a reasonable length.
You would definitely use a search engine. I was just responding to a specific design constraint.
(Note PyPI can’t index metadata from a `setup.py` however, since that would involve running arbitrary code. PyPI needs to be given structured metadata, and not all distributions provide that.)
How does the big white search box at https://pypi.org/ work? Why couldn’t the same technology be used to power the CLI? If there’s an issue with abuse, I don’t think many people would mind rate limiting or mandatory authentication before search can be used.
The PyPI website search is implemented using a real search backend (historically Elasticsearch/OpenSearch–style infrastructure) layered behind application logic on Python Package Index. Queries are tokenized, ranked, filtered, logged, and throttled. That works fine for humans interacting through a browser.
The moment you expose that same service to a ubiquitous CLI like pip, the workload changes qualitatively.
PyPI has the /simple endpoint that the CDN can handle.
It’s PyPI philosophy that search happens on the website and pip has aligned to that. Pip doesn’t want to make a web scraper understandably so the function of searching remains disabled
Alpha Omega is proud to share the newly published white paper from Seth Larson of the Python Software Foundation, titled “Slippery Zips and Sticky Tar Pits: Security and Archives.” Seth serves as Python’s Security Developer in Residence, a role sponsored by Alpha Omega, focused on improving the safety and trustworthiness of open source software that powers systems and applications everywhere.
Incident report of a recent attack campaign targeting GitHub Actions workflows to exfiltrate PyPI tokens, our response, and steps to protect your projects.
This largely depends on the ICANN policies and their definitions of Renewal and Registration Grace Periods.
The Renewal period is variable, but the Registration Grace Period is pretty much 30 days everywhere.
The ERRP only covers gTLDs, right? Have you seen any ICANN policies requiring ccTLDs to adopt the same grace periods. As far as I know, ccTLDs can do whatever they want.
ICANN policies only govern global domains. Country domains set their own policies; for example, .eu expiration period is 45 days, not 30.
WHO IS policies also vary wildly, for example .de domains do not show registration date in the WHO IS, so it's not possible to know if a domain was dropped and re-registered.
PyPI now checks for expired domains to prevent domain resurrection attacks, a type of supply-chain attack where someone buys an expired domain and uses it to take over PyPI accounts through password resets.
Excellent idea, and something I tried a little while back.
The `pytest-postgresql` plugin used has the ability to do this natively, but when we tried it out we found that we had other issues with developing on a Linux machine.
reply