> Don't understand how a private company like Astral is leading here. Why is that hard for the Python community to come up with a single tool to rule them all? (I know https://xkcd.com/927/). Like, you could even copy what Go or Node are doing, and make it Python-aware; no shame on that. Instead we have these who-knows-how-long-they-will-last tools every now and then.
Python packaging is (largely) solving problems that Go and Node packaging are not even trying to address.
Specifically simultaneous distribution of precompiled binaries for many different OS and hardware configurations and built-on-demand source distribution of non-Python software to be used as dependencies with as little (ideally none) host setup by the user all installable under a single name/version everywhere.
imagine a world without: failed to build native gem extension
Not the person you're replying to, so I don't know if this is what he had in mind, but with Python packages you can distribute more than just Python. Some packages contain C/C++/Fortran/Rust/others? source code that pip will try to automatically build upon install. Of course you can't expect everyone to have a dev environment set up, so packages can also contain pre-compiled binary for any combination of windows/mac/linux + amd/arm + glibc/musl + CPython/pypy (did I miss any?).
I don't know much about go, and I've only scratched the surface with node, but as far as node goes I think it just distributes JS? So that would be one answer to what Python packaging is trying to solve that node isn't trying to address.
> any combination of windows/mac/linux + amd/arm + glibc/musl + CPython/pypy (did I miss any?).
From a standards perspective, it is a combination of a Python version/implementation, a "platform" and an "ABI". (After all, the glibc/musl distinction doesn't make sense on Windows.)
Aside from CPython/pypy, the system recognizes IronPython (a C# implementation) and Jython (a Java implementation) under the version "tag"; of course these implementations may have their own independent versioning with only a rough correspondence to CPython releases.
The ABI tag largely corresponds to the implementation and version tag, but for example for CPython builds it also indicates whether Python was built in debug or release mode, and from 3.13 onward whether the GIL is enabled.
The platform tag covers Mac, Windows, several generic glibc Linux standards (called "manylinux" and designed to smooth over minor differences between distros), and now also some generic musl Linux standards (called "musllinux"). Basic CPU information (arm vs intel, 32- vs 64-bit etc.) is also jammed in here.
> Some packages contain C/C++/Fortran/Rust/others? source code that pip will try to automatically build upon install.
And in the TS/JS world we have React.Native that has a flexible pluggable model that allows creating XCode projects with autodiscovered dependencies in C, C++, Swift and other languages.
It's also flexible enough to allow third-party products like Sentry to integrate into the build process to upload debug symbols to the Sentry servers on release builds.
So no, Python is really not unique in its requirements.
Come on, React Native is a thing but as a % of JS engineers, the number that will regularly stray into compiled packages is much smaller than with Python. Basically every time you pick up a Maths or ML project which is a huge part of Python's ecosystem, you'll be using C or Fortran code underneath. That library ecosystem in that are is just non-existent in JS, and where it does people are typically using compile-to-WASM anyway.
Pypi doesn't have any issue distributing binaries either these days? The issue is (a) diversity of CPU/GPU microarchitectures inc. CUDA (b) diversity of OS-es.
I've never had to compile a native Node library on a POWER9 GPU cluster, but I have had to compile Python ones.
And the JS world has ESM modules, CommonJS, various polyfills for compatibility, and other quirks that need to be accommodated by the package manager. With native code compilation affected by these settings.
> I don't know much about go, and I've only scratched the surface with node, but as far as node goes I think it just distributes JS?
As a sibling comment posted, Node packages have no issue distributing non-JS assets. Since the very beginning, the "package.json" file had support for the "arch" and "postinstall" fields to distribute precompiled binaries or compile on install, for example using node-gyp. If you split your precompiled binaries and use optional dependencies then only the right package for your machine is downloaded.
In all package managers, distributing native code may be uncommon but it's always supported well enough (Node/npm/yarn, JVM/Gradle, PHP/Composer, Ruby/gem, Rust/Cargo, etc). What's unique about Python is how messy all the packaging handling is compared to other languages.
Python packaging is (largely) solving problems that Go and Node packaging are not even trying to address.