Require OpenSSF Scorecard on one of the mature stages of the incubation process

[cornelius]

The OpenSSF Scorecard defines a set of checks for good practices regarding security of open source projects. There is a tool to automatically check these via a GitHub action.

We should consider adding the OpenSSF Scorecard as a criteria to stage 2 or 3 of the incubation process.

[flomonster]

I ran their test for OSRD and here's the result:

RESULTS
-------
Aggregate score: 5.6 / 10

Check scores:
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|
|  SCORE  |          NAME          |             REASON             |                                               DOCUMENTATION/REMEDIATION                                               |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|
| 9 / 10  | Binary-Artifacts       | binaries present in source     | https://github.com/ossf/scorecard/blob/7b07a8ddf0771c3de0d537ab5295a8c891bcf63f/docs/checks.md#binary-artifacts       |
|         |                        | code                           |                                                                                                                       |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|
| 4 / 10  | Branch-Protection      | branch protection is not       | https://github.com/ossf/scorecard/blob/7b07a8ddf0771c3de0d537ab5295a8c891bcf63f/docs/checks.md#branch-protection      |
|         |                        | maximal on development and all |                                                                                                                       |
|         |                        | release branches               |                                                                                                                       |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|
| 10 / 10 | CI-Tests               | 23 out of 23 merged PRs        | https://github.com/ossf/scorecard/blob/7b07a8ddf0771c3de0d537ab5295a8c891bcf63f/docs/checks.md#ci-tests               |
|         |                        | checked by a CI test -- score  |                                                                                                                       |
|         |                        | normalized to 10               |                                                                                                                       |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|
| 0 / 10  | CII-Best-Practices     | no effort to earn an OpenSSF   | https://github.com/ossf/scorecard/blob/7b07a8ddf0771c3de0d537ab5295a8c891bcf63f/docs/checks.md#cii-best-practices     |
|         |                        | best practices badge detected  |                                                                                                                       |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|
| 10 / 10 | Code-Review            | all changesets reviewed        | https://github.com/ossf/scorecard/blob/7b07a8ddf0771c3de0d537ab5295a8c891bcf63f/docs/checks.md#code-review            |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|
| 10 / 10 | Contributors           | project has 22 contributing    | https://github.com/ossf/scorecard/blob/7b07a8ddf0771c3de0d537ab5295a8c891bcf63f/docs/checks.md#contributors           |
|         |                        | companies or organizations     |                                                                                                                       |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|
| 10 / 10 | Dangerous-Workflow     | no dangerous workflow patterns | https://github.com/ossf/scorecard/blob/7b07a8ddf0771c3de0d537ab5295a8c891bcf63f/docs/checks.md#dangerous-workflow     |
|         |                        | detected                       |                                                                                                                       |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|
| 10 / 10 | Dependency-Update-Tool | update tool detected           | https://github.com/ossf/scorecard/blob/7b07a8ddf0771c3de0d537ab5295a8c891bcf63f/docs/checks.md#dependency-update-tool |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|
| 0 / 10  | Fuzzing                | project is not fuzzed          | https://github.com/ossf/scorecard/blob/7b07a8ddf0771c3de0d537ab5295a8c891bcf63f/docs/checks.md#fuzzing                |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|
| 10 / 10 | License                | license file detected          | https://github.com/ossf/scorecard/blob/7b07a8ddf0771c3de0d537ab5295a8c891bcf63f/docs/checks.md#license                |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|
| 10 / 10 | Maintained             | 30 commit(s) and 0 issue       | https://github.com/ossf/scorecard/blob/7b07a8ddf0771c3de0d537ab5295a8c891bcf63f/docs/checks.md#maintained             |
|         |                        | activity found in the last 90  |                                                                                                                       |
|         |                        | days -- score normalized to 10 |                                                                                                                       |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|
| 10 / 10 | Packaging              | packaging workflow detected    | https://github.com/ossf/scorecard/blob/7b07a8ddf0771c3de0d537ab5295a8c891bcf63f/docs/checks.md#packaging              |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|
| 0 / 10  | Pinned-Dependencies    | dependency not pinned by hash  | https://github.com/ossf/scorecard/blob/7b07a8ddf0771c3de0d537ab5295a8c891bcf63f/docs/checks.md#pinned-dependencies    |
|         |                        | detected -- score normalized   |                                                                                                                       |
|         |                        | to 0                           |                                                                                                                       |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|
| 0 / 10  | SAST                   | SAST tool is not run on all    | https://github.com/ossf/scorecard/blob/7b07a8ddf0771c3de0d537ab5295a8c891bcf63f/docs/checks.md#sast                   |
|         |                        | commits -- score normalized to |                                                                                                                       |
|         |                        | 0                              |                                                                                                                       |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|
| 0 / 10  | Security-Policy        | security policy file not       | https://github.com/ossf/scorecard/blob/7b07a8ddf0771c3de0d537ab5295a8c891bcf63f/docs/checks.md#security-policy        |
|         |                        | detected                       |                                                                                                                       |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|
| ?       | Signed-Releases        | no releases found              | https://github.com/ossf/scorecard/blob/7b07a8ddf0771c3de0d537ab5295a8c891bcf63f/docs/checks.md#signed-releases        |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|
| 0 / 10  | Token-Permissions      | detected GitHub workflow       | https://github.com/ossf/scorecard/blob/7b07a8ddf0771c3de0d537ab5295a8c891bcf63f/docs/checks.md#token-permissions      |
|         |                        | tokens with excessive          |                                                                                                                       |
|         |                        | permissions                    |                                                                                                                       |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|
| 0 / 10  | Vulnerabilities        | 21 existing vulnerabilities    | https://github.com/ossf/scorecard/blob/7b07a8ddf0771c3de0d537ab5295a8c891bcf63f/docs/checks.md#vulnerabilities        |
|         |                        | detected                       |                                                                                                                       |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|

• Branch protection: It's low because we do not require branch to be up to date before merging. We didn't checked this protection for a smoother PR review.
• CII-Best-Practice: This criteria seems to be here only to encourage people using openSSF badges. I don't think it's a very interesting check.
• Fuzzing: We have a score of 0 because our tool is not recognized.
• Pinned-Dependencies: Our dependencies are pinned using Cargo.lock, yarn.lock (both are not recognized by the tool)
  • The documentation request to pin docker base images using hashes which I find really weird. It's supposed to avoid security issues if the docker registry is compromised.
• SAST: We don't have anything setup
• Security policy: Will be setup soon
• Signed releases: Already discussed
• Token-Permissions: I don't know which of our tokens is considered to have too much permission 🤷 (I couldn't find any more logs than the table above)
• We indeed have 21 vulnerabilities linked to our dependencies, we have to fix this.

[cornelius]

It's interesting information and it's good to make this transparent. I would assume that it's not useful to run this at every commit, but once in a while when significant changes are to be expected, e.g. when a new tool has been set up.

We could require a minimal score as a condition to move to the more mature incubation stages, e.g. minimum score 5 for stage 2 and minimum score 8 for stage 3.

[flomonster]

Report of notes taken at the meeting of 03/12/2024

• Threshold of 5/10 (Aggregate score)
• Should be run once for the stage application. Most of the criteria won't change (except vulnerabilities).
• The project can argue why OpenSSF has incorrectly scored certain criteria, in order to increase the score manually.