Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Setup snapshot testing for editoast_derive using insta #9938

Merged
merged 3 commits into from
Dec 16, 2024
Merged

Conversation

leovalais
Copy link
Contributor

@leovalais leovalais commented Dec 4, 2024

closes #7467

Setup snapshot testing library insta in order to unit-test our derive macros. This approach has several benefits:

  • Tests are easy to write, so we can have many
  • When the snapshot changes, insta shows a nice diff, making debugging our changes to the macro easier
  • Facilitates the review: snapshots are committed, so their diff appear in the PR description

Since we may soon have some extensive changes coming to our macros with the new error system, having tests and ease the review process are essential.

A few drawbacks I can think of:

  • The macro expansion need to be determinist: no more HashMaps are allowed
  • We need to refrain as much as possible from using macros in our code gen, otherwise it hinders the readability of the snapshots
  • We have to "accept" diffs manually (either by renaming the file or by using cargo insta, their official interactive snapshot management CLI tool)

Important

Review this PR commit by commit to see how the snapshots evolve when the macro output changes.

@github-actions github-actions bot added area:editoast Work on Editoast Service area:ci Work on Continous Integration Pipeline Service labels Dec 4, 2024
@codecov-commenter
Copy link

codecov-commenter commented Dec 4, 2024

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 96.37681% with 10 lines in your changes missing coverage. Please review.

Project coverage is 79.88%. Comparing base (02ccfc8) to head (8023bd3).
Report is 107 commits behind head on dev.

Files with missing lines Patch % Lines
editoast/editoast_derive/src/model/identifier.rs 50.00% 6 Missing ⚠️
editoast/editoast_derive/src/model.rs 82.60% 4 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@             Coverage Diff             @@
##              dev    #9938       +/-   ##
===========================================
+ Coverage   38.21%   79.88%   +41.66%     
===========================================
  Files         992     1054       +62     
  Lines       92158   105597    +13439     
  Branches     1186      759      -427     
===========================================
+ Hits        35221    84353    +49132     
+ Misses      56486    21202    -35284     
+ Partials      451       42      -409     
Flag Coverage Δ
editoast 73.72% <96.37%> (+0.32%) ⬆️
front 89.24% <ø> (+69.05%) ⬆️
gateway 2.18% <ø> (+<0.01%) ⬆️
osrdyne 3.28% <ø> (ø)
railjson_generator 87.50% <ø> (+0.01%) ⬆️
tests 87.00% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@leovalais leovalais force-pushed the lva/insta branch 3 times, most recently from e2407b9 to 3e0bf44 Compare December 4, 2024 14:23
@leovalais leovalais marked this pull request as ready for review December 4, 2024 14:23
@leovalais leovalais requested review from a team as code owners December 4, 2024 14:23
Copy link
Contributor

@woshilapin woshilapin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have to "accept" diffs manually (either by renaming the file or by using cargo insta, their official interactive snapshot management CLI tool)

I actually don't see this as a drawback. Sure, it's more work for the developer, but we actually always do that even for "normal" unit tests: a expected output changed, we need to review it manually and validate that the new output is valid. So it's actually a good thing. Maybe you meant it's more cumbersome to do that operation of validating a different test output?

In general, this PR is going to bring much more to the experience of dealing with macros...

@flomonster
Copy link
Contributor

Great idea! Thanks for the PR.

@leovalais
Copy link
Contributor Author

Thanks for the feedback! I've added *.snap.new to our .gitignore to avoid accidentally commit snapshots that still require some review (missing cargo insta review).

@Khoyo
Copy link
Contributor

Khoyo commented Dec 6, 2024

We should probably add some documentation about working with insta, either in editoast README or in a new one for editoast_derive

Copy link
Contributor

@woshilapin woshilapin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for digging up about the "macro into function" proposition, even if it ended up not satisfying. And thanks for the additional documentation. Great improvement for editoast_derive in general! ❤️

Allows to easily test macro expansion, track changes, improves
debuggability and ease review.

HashMaps have been eliminated from `Model`'s internals to have a
deterministic expansion. Otherwise snapshots would be flaky.

Signed-off-by: Leo Valais <[email protected]>
Macros confuses `prettyplease` and we want snapshots as clean as
possible.

Signed-off-by: Leo Valais <[email protected]>
Not only does this reduces the complexity of batch operation expansion,
it also allows `prettyplease` to format the code chunk in snapshots.

Signed-off-by: Leo Valais <[email protected]>
@leovalais leovalais added this pull request to the merge queue Dec 16, 2024
Merged via the queue into dev with commit 6ec445a Dec 16, 2024
27 checks passed
@leovalais leovalais deleted the lva/insta branch December 16, 2024 09:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:ci Work on Continous Integration Pipeline Service area:editoast Work on Editoast Service
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add snapshot testing to editoast_derive
5 participants