Tests fail with Python 3.12

[gvanrossum]

Whenever I push something to a PR in the cpython main branch (3.12) I get an email from this repo telling me that the 3.12 run failed. Is there a way we can fix it? Here's a link to the failing run: https://github.com/python/pyperformance/actions/runs/4094033809/jobs/7059879010

I'd rather not just disable running with 3.12 (that would presumably take away some useful signal) but maybe we can fix or suppress the failing tests?

[AlexWaygood]

I believe this PR by @hugovk would mean that the tests would still run, but wouldn't be reported as "failures" for now:

• Skip 3.12-dev pending Greenlet support #259

[gvanrossum]

Yeah, that just disables 3.12 altogether, right? Isn't there a way to only disable those tests that depend on greenlet?

[gvanrossum]

IIRC in the past we resolved a similar thing (for a different dependency) by changing the requirements.txt file to reference a specific PR (via some notation I've forgotten that lets you depend on a git repo/branch).

[hugovk]

Yeah, that just disables 3.12 altogether, right?

Yes, the original intent of #259 was to run on 3.12, and if 3.12 fails, then don't fail the whole build. But I couldn't find the right continue-on-error incantation, so instead it disables the whole 3.12 job.

Isn't there a way to only disable those tests that depend on greenlet?

Yes, we can skip them when greelet isn't available. Please see PR #264.

[gvanrossum]

Yeah, #264 feels better. I hesitate to approve it, since I am not familiar with this code base (nor with the sutbleties of GitHub workflows) but maybe @mdboom can take a quick look.

[corona10]

@gvanrossum I proposed another PR that does not touch pyperformance code directly but managing by GHA meta data.

PR: #274 (Expect as working correctly after the PR is merged)
Sample: corona10#1 (comment)

[corona10]

Ah https://github.com/python/pyperformance/pull/259/files#r1096738419 looks similar approach but I didn't notice it

[hugovk]

Looks like the key difference between those, you have continue-on-error: ${{ matrix.experimental }} at the job level, I had it at the step level. Nice work!

[corona10]

Close this issue :) If the issue is needed to be reopened, feel free to open the issue again.