setuptools removal breaking builds

[ipmb]

I've seen numerous builds that were previously working and now fail since v0.14.0 was released with #243

Many have a requirements.txt generated using pip-tools --generate-hashes ... without using the --allow-unsafe flag.

This results in errors like this at build time:

Collecting setuptools (from pip-tools==6.13.0->-r requirements.txt (line 1169))
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    setuptools from https://files.pythonhosted.org/packages/e1/58/e0ef3b9974a04ce9cde2a7a33881ddcb2d68450803745804545cdd8d258f/setuptools-72.1.0-py3-none-any.whl (from pip-tools==6.13.0->-r requirements.txt (line 1169))

[Error: Unable to install dependencies using pip]
The 'pip install' command to install the application's dependencies from
'requirements.txt' failed (exit status: 1).

See the log output above for more information.

ERROR: failed to build: exit status 1
ERROR: failed to build: executing lifecycle: failed with status code: 51

Some packages have implicit dependencies on setuptools. Here's an example of an error I just saw:

Traceback (most recent call last):
  File "/workspace/manage.py", line 23, in <module>
    main()
  File "/workspace/manage.py", line 19, in main
    execute_from_command_line(sys.argv)
  File "/layers/heroku_python/dependencies/lib/python3.11/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line
    utility.execute()
  File "/layers/heroku_python/dependencies/lib/python3.11/site-packages/django/core/management/__init__.py", line 420, in execute
    django.setup()
  File "/layers/heroku_python/dependencies/lib/python3.11/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/layers/heroku_python/dependencies/lib/python3.11/site-packages/django/apps/registry.py", line 91, in populate
    app_config = AppConfig.create(entry)
                 ^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/heroku_python/dependencies/lib/python3.11/site-packages/django/apps/config.py", line 126, in create
    mod = import_module(mod_path)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/heroku_python/python/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/layers/heroku_python/dependencies/lib/python3.11/site-packages/django_q/apps.py", line 3, in <module>
    from django_q.conf import Conf
  File "/layers/heroku_python/dependencies/lib/python3.11/site-packages/django_q/conf.py", line 8, in <module>
    import pkg_resources
ModuleNotFoundError: No module named 'pkg_resources'

This removal seems premature given that major projects in the Python ecosystem assume its presence.

[edmorley]

Hi - sorry for the delayed reply!

The reasons the global setuptools install has been dropped are:

1. The Python venv and ensurepip stdlib modules no longer install setuptools when using Python 3.12+, and the majority of other tools/environments are changing (or have changed some time ago) to also not install it for newer Python versions (eg GitHub Actions, virtualenv, uv, get-pip, python:* Docker images; see links in description of Stop installing setuptools and wheel #243).
2. For build-time setuptools needs, the fallback default build backend will inject setuptools and wheel automatically, so these packages will continue to work.
3. With the switch to venvs in Switch to using a virtual environment for app dependencies #253, even if we install setuptools alongside pip in the pip layer, it won't be usable at either build or run-time by the app dependencies being pip installed into the venv. We would instead have to install it as an app dependency into the venv directly, which wouldn't be ideal.
4. When using Poetry, poetry install --sync removes setuptools from the venv if its not in the lockfile, which means that even if the buildpack installs setuptools itself, it won't actually be available anyway. Whilst this doesn't directly affect apps using pip, ideally we'd have parity between the buildpack installed tooling installed regardless of chosen package manager.
5. It will be less jarring for us to make this change now (whilst our CNBs are still in preview, and so subject to changes) than when they reach general availability.

The recent Poetry work (which also required the pip / venv changes) being the primary trigger for making this change now.

In addition, the design philosophy of this buildpack is to:

• drop the tech debt of the classic buildpack where possible
• target the future state/trajectory of the Python ecosystem at the time when the Heroku CNBs reach general availability (/are useable on Heroku), rather than the historic state of the Python ecosystem or the even the state as of right now.

Regarding the specific cases of build failures in the OP, you are correct that there will be some packages in the wild that may depend on setuptools at run-time (either directly, or its copy of pkg_resources) but that have forgotten to specify it as a dependency.

However:

1. These packages effectively have incomplete metadata - if they require a library at run-time they need to specify it via install_requires or similar (and always should have been doing so, even before Python's venv / ensurepip dropped setuptools; though it's understandable why some did not, given everything appeared to work regardless).
2. These packages are already broken on Python 3.12+. Over time more people will be upgrading to Python 3.12+ (particularly given Python 3.13 is due out next month) which will make the fact it's these packages that are broken more apparent - leading to fixes or forks of these packages. So by the time this CNB GAs, this will be even less of an issue.
3. Use of pkg_resources has also been discouraged/informally deprecated since 2021 and officially deprecated (with warning output) since 2023.
4. Users have a workaround, which is to add setuptools to their requirements.txt.

For example as proof of (2), the django-q package you mention in the OP is broken outside of this CNB already (and has been since the release of Python 3.12):

$ docker run --rm -it python:3.12-slim bash
root@8c05e98c8ba7:/# python -m venv .venv
root@8c05e98c8ba7:/# source .venv/bin/activate
(.venv) root@8c05e98c8ba7:/# pip install -q django-q==1.3.9
(.venv) root@8c05e98c8ba7:/# python -c 'import django_q.conf'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/.venv/lib/python3.12/site-packages/django_q/conf.py", line 8, in <module>
    import pkg_resources
ModuleNotFoundError: No module named 'pkg_resources'

These packages will also be broken when using Poetry with any Python version (even those before 3.12), since Poetry removes setuptools from any venv if it wasn't declared in the lockfile. (And I presume the same will be the case for uv sync.)

It's worth also noting for that specific package, that:

• It is effectively unmaintained (hasn't had a release since 2021, and PRs aren't getting merged)
• It's a lower usage package compared to some. Monthly download stats:
  • https://pypistats.org/packages/django-q -> 99 K
  • https://pypistats.org/packages/django -> 18.8 M
  • https://pypistats.org/packages/djangorestframework -> 10.0 M
  • https://pypistats.org/packages/django-debug-toolbar -> 3.1 M
  • https://pypistats.org/packages/django-environ -> 2.8 M
  • https://pypistats.org/packages/django-heroku -> 420 K (this package is sunset and hasn't had a release since 2018, yet it still has 4x more users than django-q)
• There's already a fork of it available (django-q2, with 78 K downloads/month) which fixes issues like the undeclared setuptools usage, and people have been trying to migrate users to the new project or at least consolidate the two (see Can someone relinquish django-q and allow the django-q2 community to fold in? Koed00/django-q#735).

Now obviously, there will be users that either don't realise there is a working fork, or other legacy broken packages out there for which there aren't fixes or a fork. However, given that:
(a) there is an easy user workaround (adding setuptools to requirements.txt),
(b) these broken packages are typically lower usage and are only going to become more broken over time as new Python versions are released for example (unless they start being maintained)

...then I think in practice it shouldn't be too impactful, and there isn't anything we can or should be doing for it.

Many have a requirements.txt generated using pip-tools --generate-hashes ... without using the --allow-unsafe flag.

I'm very much a proponent of using the hashes mode of pip (I was a contributor/co-maintainer of peep before it was integrated into pip, have contributed hashes mode fixes to pip since, and advocated for using hashes mode when I was at Mozilla), however, using it does mean accepting that issues like this can occur from time to time.

For example, if a requirements.txt file lists a dependency that only has an sdist and later the package maintainer uploads a wheel for that version (or if the package had wheels, but later additional more-specific variants are uploaded), then pip will immediately start trying to use the wheel, triggering a "hash not listed" type error. This isn't just a hypothetical, I've had to fix a number of these in the past, eg:

• Bug 1221157 - Update hash for django-browserid now it has a wheel mozilla/treeherder#1113
• Update package hash for orderedmultidict mozilla/treeherder#3990
• Add new pip hashes for zope.interface mozilla/treeherder#4118
• Add hash for isort wheel package mozilla/treeherder#4720

But for most, including me, it's still worth the trade-offs :-)

It's also primarily advanced users that are using hashes - who are more than capable of making a one-off fix to their requirements file in cases like these. (And us dropping setuptools is very much a one-off type of change, I don't expect any future CNB changes to cause users to need to update hashes again.)

I think perhaps the main issue / point of disconnect here, is that this buildpack is still in preview (and so from our perspective is subject to change, and only has early adopter users who know what they are getting into and are both able and happy to work through any compatibility changes), whereas it sounds like from your perspective, you're using this CNB in a production service with end users who are expecting stability (and I'm presuming may not know or care what a CNB is)?

Does AppPack currently mirror/pin the builder images? If not, perhaps controlling how these are rolled out to your users might help smooth things out until the CNB reaches GA and our ideas of project status/stability are more in alignment?

[ipmb]

Where can I track what is GA vs preview? Yes, we're typically letting users pull the latest heroku/builder:22 (soon 24) and heroku/python buildpacks. It was not clear to me that these aren't intended to be production-ready.

[edmorley]

All things Heroku+CNB are currently "in preview" and not generally available. For now, it's currently not possible to build or deploy CNBs on Heroku (the new CNB build system and rest of platform changes are internal only) - the "preview" is people being able to try CNBs out via Pack CLI or similar.

The preview was announced here:
https://blog.heroku.com/heroku-cloud-native-buildpacks

And the preview status is documented in places like:

• buildpacks-python/README.md

  Lines 12 to 13 in 53eabc7

  	> [!IMPORTANT]
  	> This is a [Cloud Native Buildpack][cnb], and is a component of the [Heroku Cloud Native Buildpacks][heroku-buildpacks] project, which is in preview. If you are instead looking for the Heroku Classic Buildpack for Python (for use on the Heroku platform), you may find it [here][classic-buildpack].

• https://github.com/heroku/cnb-builder-images/blob/0b107913526af0ea15916a56bbb1f0732aa771bc/README.md?plain=1#L5-L6
• https://github.com/heroku/buildpacks/blob/a678c66045e169971127472423dcbedecd4d4154/README.md?plain=1#L12

Prior to the March 2024 preview announcement, iirc repos READMEs and/or GitHub repo description metadata described the projects as "experimental".

The Heroku roadmap item for platform support for CNBs is here:
heroku/roadmap#20

[edmorley]

It was not clear to me that these aren't intended to be production-ready.

So there's a bit of nuance here - all of our CNB buildpacks and the builder images are already capable of producing production-ready OCI images, it's more that the buildpacks don't necessarily support all features yet, and we aren't making the same release-to-release change stability guarantees that we would make for a GA product since we're still refining features and design. For an early-adopter end user who is comfortable reading a changelog and is aware that they are using an "in preview" tool that's subject to change, IMO there's nothing wrong with them using images generated by these buildpacks in production - and we welcome the feedback.

[ipmb]

Thanks for the clarification @edmorley. My understanding is that the non-CNB buildpacks are EOL and no longer supported. So there isn't a supported GA option for Python using Heroku builders?

[edmorley]

No, the non-CNB buildpacks (which we've been colloquially calling "classic buildpacks" to differentiate from CNBs) are still supported - and will be for some time, since the Heroku customer migration to CNBs won't happen overnight. (ie: The buildpacks like: https://github.com/heroku/heroku-buildpack-python)

However, I believe you might be talking about the experimental "shimmed" CNBs that were bundled in the now-sunset heroku/buildpacks:* builder images? These shimmed CNBs were a temporary stop-gap to facilitate the early development of CNBs prior to native CNBs being available for all languages (for example, it's hard to work on an internal build system until there is something that can run on it). These shimmed CNBs were not Generally Available (ie: not in production and not officially supported), and also not usable on Heroku by customers.

So there isn't a supported GA option for Python using Heroku builders?

Not one Heroku CNB or builder image is GA at the moment (or has ever been), regardless of language (this isn't Python specific).

This is what was meant above by:

All things Heroku+CNB are currently "in preview" and not generally available.

[edmorley]

Closing this out, since:

• The cases of run-time breakage are due to incomplete (i.e. broken) metadata in upstream packages, which should be fixed in those packages.
• The number of affected packages is overall very low (and those packages are already broken in most other environments/tooling, when using Python 3.12+ - such as when using venv, virtualenv, get-pip.py, the official Docker Python images, GitHub Actions, Poetry, Pipenv, uv etc). Plus, by the time this buildpack GAs, even fewer packages will be affected in the wild.
• It's easy for end-users to work around packages with incomplete metadata, by specifying an explicit dependency on setuptools in their requirements.txt.
• Even if we wanted to try and work around these broken packages in the Python buildpack, the layer design (that we need/want to use for other reasons) doesn't naturally lend itself to injecting arbitrary dependencies into the app-facing environment. Plus magically injecting packages doesn't work when using a package manager that manages the environment using a lockfile (such as Poetry or uv).