Skip to content

Commit 7b84d85

Browse files
committed
Fork xRetry into xRetry.v3, in preparation for xUnit.v3 support
xRetry and xRetry.v3 will live side-by-side until xUnit v2 is not something we want to still work on. For now, it's still widely used & there's no v3 support in Specflow or Reqnroll. Many other xUnit extensions also don't support v3 yet, so a full switch over won't be possible for many people. These .v3 packages are currently just copies of the existing code, still targeting xUnit v2. This skeleton will prove CI works, and be the starting point for work towards v3 support.
1 parent d17aee8 commit 7b84d85

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

51 files changed

+1869
-13
lines changed

.github/workflows/cicd.yaml

+16-3
Original file line numberDiff line numberDiff line change
@@ -21,11 +21,11 @@ jobs:
2121
working-directory: build
2222
run: make build
2323

24-
- name: Run Unit Tests (Main)
24+
- name: Run Unit Tests (Main xRetry)
2525
working-directory: build
2626
run: make unit-tests-run-main
2727

28-
- name: Run Unit Tests (Single Threaded)
28+
- name: Run Unit Tests (Single Threaded xRetry)
2929
working-directory: build
3030
run: make unit-tests-run-single-threaded
3131

@@ -37,6 +37,14 @@ jobs:
3737
working-directory: build
3838
run: make unit-tests-run-reqnroll
3939

40+
- name: Run Unit Tests (Main xRetry.v3)
41+
working-directory: build
42+
run: make unit-tests-run-main-v3
43+
44+
- name: Run Unit Tests (Single Threaded xRetry.v3)
45+
working-directory: build
46+
run: make unit-tests-run-single-threaded-v3
47+
4048
- name: Create Nuget Packages
4149
working-directory: build
4250
run: make nuget-create
@@ -104,4 +112,9 @@ jobs:
104112
- name: "Release: xRetry.Reqnroll"
105113
if: startsWith(github.ref, 'refs/tags/xRetry.Reqnroll_v')
106114
working-directory: deploy
107-
run: /bin/bash deploy.sh xRetry.Reqnroll ${{ secrets.NUGET_API_KEY }}
115+
run: /bin/bash deploy.sh xRetry.Reqnroll ${{ secrets.NUGET_API_KEY }}
116+
117+
- name: "Release: xRetry.v3"
118+
if: startsWith(github.ref, 'refs/tags/xRetry.v3_v')
119+
working-directory: deploy
120+
run: /bin/bash deploy.sh xRetry.v3 ${{ secrets.NUGET_API_KEY }}

README.md

+81-3
Original file line numberDiff line numberDiff line change
@@ -15,9 +15,10 @@ Retry flickering test cases for xUnit, Reqnroll and SpecFlow in .NET.
1515

1616
[//]: \# (Src: nugetBadges.md)
1717

18-
| Testing Tool | Plugin | NuGet Package |
19-
|-------------------------------|--------|---------------|
20-
| [xUnit](#usage-xunit) | xRetry | [![xRetry NuGet Version](https://img.shields.io/nuget/v/xRetry)](https://www.nuget.org/packages/xRetry "Download xRetry from NuGet") [![xRetry NuGet Downloads](https://img.shields.io/nuget/dt/xRetry)](https://www.nuget.org/packages/xRetry "Download xRetry from NuGet") |
18+
| Testing Tool | Plugin | NuGet Package |
19+
|-------------------------------|-----------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
20+
| [xUnit v3](#usage-xunit) | xRetry.v3 | [![xRetry.v3 NuGet Version](https://img.shields.io/nuget/v/xRetry.v3)](https://www.nuget.org/packages/xRetry.v3 "Download xRetry.v3 from NuGet") [![xRetry.v3 NuGet Downloads](https://img.shields.io/nuget/dt/xRetry.v3)](https://www.nuget.org/packages/xRetry.v3 "Download xRetry.v3 from NuGet") |
21+
| [xUnit v2](#usage-xunit) | xRetry | [![xRetry NuGet Version](https://img.shields.io/nuget/v/xRetry)](https://www.nuget.org/packages/xRetry "Download xRetry from NuGet") [![xRetry NuGet Downloads](https://img.shields.io/nuget/dt/xRetry)](https://www.nuget.org/packages/xRetry "Download xRetry from NuGet") |
2122
| [SpecFlow](#usage-specflow-3) | xRetry.SpecFlow | [![xRetry.SpecFlow NuGet Version](https://img.shields.io/nuget/v/xRetry.SpecFlow)](https://www.nuget.org/packages/xRetry.SpecFlow "Download xRetry.SpecFlow from NuGet") [![xRetry.SpecFlow NuGet Downloads](https://img.shields.io/nuget/dt/xRetry.SpecFlow)](https://www.nuget.org/packages/xRetry.SpecFlow "Download xRetry.SpecFlow from NuGet") |
2223
| [Reqnroll](#usage-reqnroll-2) | xRetry.Reqnroll | [![xRetry.Reqnroll NuGet Version](https://img.shields.io/nuget/v/xRetry.Reqnroll)](https://www.nuget.org/packages/xRetry.Reqnroll "Download xRetry.Reqnroll from NuGet") [![xRetry.Reqnroll NuGet Downloads](https://img.shields.io/nuget/dt/xRetry.Reqnroll)](https://www.nuget.org/packages/xRetry.Reqnroll "Download xRetry.Reqnroll from NuGet") |
2324

@@ -36,6 +37,83 @@ This is the intended use case of the library.
3637
If you have a test that covers some flaky code, where sporadic failures are caused by a bug,
3738
this library should **not** be used to cover it up!
3839

40+
[//]: \# (Src: xRetry.v3/usage.md)
41+
42+
## Usage: xUnit
43+
44+
Add the [`xRetry.v3` NuGet package](https://www.nuget.org/packages/xRetry.v3 "xRetry.v3 NuGet package") to your project.
45+
46+
### Facts
47+
48+
Above any `Fact` test case that should be retried, replace the `Fact` attribute, with
49+
`RetryFact`, e.g:
50+
51+
```cs
52+
using xRetry;
53+
54+
private static int defaultNumCalls = 0;
55+
56+
[RetryFact]
57+
public void Default_Reaches3()
58+
{
59+
defaultNumCalls++;
60+
61+
Assert.Equal(3, defaultNumCalls);
62+
}
63+
```
64+
65+
This will attempt to run the test until it passes, up to 3 times by default.
66+
You can optionally specify a number of times to attempt to run the test as an argument, e.g. `[RetryFact(5)]`.
67+
68+
You can also optionally specify a delay between each retry (in milliseconds) as a second
69+
parameter, e.g. `[RetryFact(5, 100)]` will run your test up to 5 times, waiting 100ms between each attempt.
70+
71+
### Theories
72+
73+
If you have used the library for retrying `Fact` tests, using it to retry a `Theory` should be very intuitive.
74+
Above any `Theory` test case that should be retried, replace the `Theory` attribute with `RetryTheory`, e.g:
75+
76+
```cs
77+
// testId => numCalls
78+
private static readonly Dictionary<int, int> defaultNumCalls = new Dictionary<int, int>()
79+
{
80+
{ 0, 0 },
81+
{ 1, 0 }
82+
};
83+
84+
[RetryTheory]
85+
[InlineData(0)]
86+
[InlineData(1)]
87+
public void Default_Reaches3(int id)
88+
{
89+
defaultNumCalls[id]++;
90+
91+
Assert.Equal(3, defaultNumCalls[id]);
92+
}
93+
```
94+
95+
The same optional arguments (max attempts and delay between each retry) are supported as for facts, and can be used in the same way.
96+
97+
### Skipping tests at Runtime
98+
99+
In addition to retries, `RetryFact` and `RetryTheory` both support dynamically skipping tests at runtime. To make a test skip just use `Skip.Always()`
100+
within your test code.
101+
It also supports custom exception types so you can skip a test if a type of exception gets thrown. You do this by specifying the exception type to the
102+
attribute above your test, e.g.
103+
104+
```cs
105+
[RetryFact(typeof(TestException))]
106+
public void CustomException_SkipsAtRuntime()
107+
{
108+
throw new TestException();
109+
}
110+
```
111+
112+
This functionality also allows for skipping to work when you are already using another library for dynamically skipping tests by specifying the exception
113+
type that is used by that library to the `RetryFact`. e.g. if you are using the popular Xunit.SkippableFact NuGet package and want to add retries, converting the
114+
test is as simple as replacing `[SkippableFact]` with `[RetryFact(typeof(Xunit.SkipException))]` above the test and you don't need to change the test itself.
115+
116+
39117
[//]: \# (Src: xRetry/usage.md)
40118

41119
## Usage: xUnit

build/Makefile

+36-1
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
XRETRY_VERSION =1.9.0#
22
XRETRY_SPECFLOW_VERSION =1.9.0#
33
XRETRY_REQNOLL_VERSION =1.0.0#
4+
XRETRY_V3_VERSION =1.0.0-alpha1#
45

56
.PHONY: clean
67
clean:
@@ -20,6 +21,9 @@ build: clean
2021
dotnet restore ../
2122

2223
# SpecFlow & Reqnroll plugins must be built before the tests
24+
cd ../src/xRetry.v3 && \
25+
dotnet build -c Release --no-restore -p:Version=$(XRETRY_V3_VERSION)
26+
2327
cd ../src/xRetry.SpecFlow && \
2428
dotnet build -c Release --no-restore -p:Version=$(XRETRY_SPECFLOW_VERSION)
2529

@@ -37,9 +41,21 @@ build: clean
3741

3842
cd ../test/UnitTests.Reqnroll && \
3943
dotnet build -c Release --no-restore
44+
45+
cd ../test/UnitTests.v3 && \
46+
dotnet build -c Release --no-restore
47+
48+
cd ../test/UnitTests.SingleThreaded.v3 && \
49+
dotnet build -c Release --no-restore
4050

4151
.PHONY: unit-tests-run
42-
unit-tests-run: unit-tests-run-main unit-tests-run-single-threaded unit-tests-run-specflow unit-tests-run-reqnroll
52+
unit-tests-run: \
53+
unit-tests-run-main \
54+
unit-tests-run-single-threaded \
55+
unit-tests-run-specflow \
56+
unit-tests-run-reqnroll \
57+
unit-tests-run-main-v3 \
58+
unit-tests-run-single-threaded-v3
4359

4460
.PHONY: unit-tests-run-main
4561
unit-tests-run-main:
@@ -63,6 +79,18 @@ unit-tests-run-reqnroll:
6379
cd ../test/UnitTests.Reqnroll && \
6480
dotnet test --no-build -c Release --logger:trx\;logfilename=../../../artefacts/testResults/UnitTests.Reqnroll.trx
6581

82+
.PHONY: unit-tests-run-main-v3
83+
unit-tests-run-main-v3:
84+
cd ../test/UnitTests.v3 && \
85+
dotnet test --no-build -c Release --logger:trx\;logfilename=../../../artefacts/testResults/UnitTests.v3.trx
86+
87+
.PHONY: unit-tests-run-single-threaded-v3
88+
unit-tests-run-single-threaded-v3:
89+
# Run the single threaded tests with a timeout, as they test deadlock scenarios which would never return if failing
90+
# You can tell if this times out as it returns exit code 124, which make prints as "Error 124"
91+
cd ../test/UnitTests.SingleThreaded.v3 && \
92+
timeout 10 dotnet test --no-build -c Release --logger:trx\;logfilename=../../../artefacts/testResults/UnitTests.SingleThreaded.v3.trx
93+
6694
.PHONY: docs
6795
docs:
6896
cd ../docs && \
@@ -92,6 +120,13 @@ nuget-create:
92120
--no-build \
93121
-c Release \
94122
-o ../artefacts/nuget
123+
124+
dotnet pack ../src/xRetry.v3 \
125+
-p:Version=$(XRETRY_V3_VERSION) \
126+
-p:NuspecFile=xRetry.v3.nuspec \
127+
--no-build \
128+
-c Release \
129+
-o ../artefacts/nuget
95130

96131
.PHONY: ci
97132
ci: lint build unit-tests-run docs nuget-create

docs/Makefile

+12-1
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ repo:
1919
$(call append,ciBadge.md,../README.md,0)
2020
$(call append,nugetBadges.md,../README.md,0)
2121
$(call append,whenToUse.md,../README.md,0)
22+
$(call append,xRetry.v3/usage.md,../README.md,0)
2223
$(call append,xRetry/usage.md,../README.md,0)
2324
$(call append,xRetry.SpecFlow/usage.md,../README.md,0)
2425
$(call append,xRetry.Reqnroll/usage.md,../README.md,0)
@@ -56,5 +57,15 @@ xRetry.Reqnroll:
5657
$(call append,logs.md,../src/xRetry.Reqnroll/README.md,0)
5758
$(call append,issues.md,../src/xRetry.Reqnroll/README.md,0)
5859

60+
.PHONY: xRetry.v3
61+
xRetry.v3:
62+
echo $(GENERATED_DISCLAIMER) > ../src/xRetry.v3/README.md
63+
$(call append,xRetry.v3/header.md,../src/xRetry.v3/README.md,0)
64+
$(call append,ciBadge.md,../src/xRetry.v3/README.md,0)
65+
$(call append,whenToUse.md,../src/xRetry.v3/README.md,0)
66+
$(call append,xRetry.v3/usage.md,../src/xRetry.v3/README.md,0)
67+
$(call append,logs.md,../src/xRetry.v3/README.md,0)
68+
$(call append,issues.md,../src/xRetry.v3/README.md,0)
69+
5970
.PHONY: all
60-
all: repo xRetry xRetry.SpecFlow xRetry.Reqnroll
71+
all: repo xRetry xRetry.SpecFlow xRetry.Reqnroll xRetry.v3

docs/nugetBadges.md

+4-3
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
1-
| Testing Tool | Plugin | NuGet Package |
2-
|-------------------------------|--------|---------------|
3-
| [xUnit](#usage-xunit) | xRetry | [![xRetry NuGet Version](https://img.shields.io/nuget/v/xRetry)](https://www.nuget.org/packages/xRetry "Download xRetry from NuGet") [![xRetry NuGet Downloads](https://img.shields.io/nuget/dt/xRetry)](https://www.nuget.org/packages/xRetry "Download xRetry from NuGet") |
1+
| Testing Tool | Plugin | NuGet Package |
2+
|-------------------------------|-----------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
3+
| [xUnit v3](#usage-xunit) | xRetry.v3 | [![xRetry.v3 NuGet Version](https://img.shields.io/nuget/v/xRetry.v3)](https://www.nuget.org/packages/xRetry.v3 "Download xRetry.v3 from NuGet") [![xRetry.v3 NuGet Downloads](https://img.shields.io/nuget/dt/xRetry.v3)](https://www.nuget.org/packages/xRetry.v3 "Download xRetry.v3 from NuGet") |
4+
| [xUnit v2](#usage-xunit) | xRetry | [![xRetry NuGet Version](https://img.shields.io/nuget/v/xRetry)](https://www.nuget.org/packages/xRetry "Download xRetry from NuGet") [![xRetry NuGet Downloads](https://img.shields.io/nuget/dt/xRetry)](https://www.nuget.org/packages/xRetry "Download xRetry from NuGet") |
45
| [SpecFlow](#usage-specflow-3) | xRetry.SpecFlow | [![xRetry.SpecFlow NuGet Version](https://img.shields.io/nuget/v/xRetry.SpecFlow)](https://www.nuget.org/packages/xRetry.SpecFlow "Download xRetry.SpecFlow from NuGet") [![xRetry.SpecFlow NuGet Downloads](https://img.shields.io/nuget/dt/xRetry.SpecFlow)](https://www.nuget.org/packages/xRetry.SpecFlow "Download xRetry.SpecFlow from NuGet") |
56
| [Reqnroll](#usage-reqnroll-2) | xRetry.Reqnroll | [![xRetry.Reqnroll NuGet Version](https://img.shields.io/nuget/v/xRetry.Reqnroll)](https://www.nuget.org/packages/xRetry.Reqnroll "Download xRetry.Reqnroll from NuGet") [![xRetry.Reqnroll NuGet Downloads](https://img.shields.io/nuget/dt/xRetry.Reqnroll)](https://www.nuget.org/packages/xRetry.Reqnroll "Download xRetry.Reqnroll from NuGet") |

docs/xRetry.v3/header.md

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# xRetry.v3
2+
Retry flickering test cases for xUnit v3.

docs/xRetry.v3/usage.md

+73
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
## Usage: xUnit
2+
3+
Add the [`xRetry.v3` NuGet package](https://www.nuget.org/packages/xRetry.v3 "xRetry.v3 NuGet package") to your project.
4+
5+
### Facts
6+
7+
Above any `Fact` test case that should be retried, replace the `Fact` attribute, with
8+
`RetryFact`, e.g:
9+
10+
```cs
11+
using xRetry;
12+
13+
private static int defaultNumCalls = 0;
14+
15+
[RetryFact]
16+
public void Default_Reaches3()
17+
{
18+
defaultNumCalls++;
19+
20+
Assert.Equal(3, defaultNumCalls);
21+
}
22+
```
23+
24+
This will attempt to run the test until it passes, up to 3 times by default.
25+
You can optionally specify a number of times to attempt to run the test as an argument, e.g. `[RetryFact(5)]`.
26+
27+
You can also optionally specify a delay between each retry (in milliseconds) as a second
28+
parameter, e.g. `[RetryFact(5, 100)]` will run your test up to 5 times, waiting 100ms between each attempt.
29+
30+
### Theories
31+
32+
If you have used the library for retrying `Fact` tests, using it to retry a `Theory` should be very intuitive.
33+
Above any `Theory` test case that should be retried, replace the `Theory` attribute with `RetryTheory`, e.g:
34+
35+
```cs
36+
// testId => numCalls
37+
private static readonly Dictionary<int, int> defaultNumCalls = new Dictionary<int, int>()
38+
{
39+
{ 0, 0 },
40+
{ 1, 0 }
41+
};
42+
43+
[RetryTheory]
44+
[InlineData(0)]
45+
[InlineData(1)]
46+
public void Default_Reaches3(int id)
47+
{
48+
defaultNumCalls[id]++;
49+
50+
Assert.Equal(3, defaultNumCalls[id]);
51+
}
52+
```
53+
54+
The same optional arguments (max attempts and delay between each retry) are supported as for facts, and can be used in the same way.
55+
56+
### Skipping tests at Runtime
57+
58+
In addition to retries, `RetryFact` and `RetryTheory` both support dynamically skipping tests at runtime. To make a test skip just use `Skip.Always()`
59+
within your test code.
60+
It also supports custom exception types so you can skip a test if a type of exception gets thrown. You do this by specifying the exception type to the
61+
attribute above your test, e.g.
62+
63+
```cs
64+
[RetryFact(typeof(TestException))]
65+
public void CustomException_SkipsAtRuntime()
66+
{
67+
throw new TestException();
68+
}
69+
```
70+
71+
This functionality also allows for skipping to work when you are already using another library for dynamically skipping tests by specifying the exception
72+
type that is used by that library to the `RetryFact`. e.g. if you are using the popular Xunit.SkippableFact NuGet package and want to add retries, converting the
73+
test is as simple as replacing `[SkippableFact]` with `[RetryFact(typeof(Xunit.SkipException))]` above the test and you don't need to change the test itself.

docs/xRetry/header.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,2 @@
11
# xRetry
2-
Retry flickering test cases for xUnit.
2+
Retry flickering test cases for xUnit v2.

src/xRetry.v3/BlockingMessageBus.cs

+54
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
using System.Collections.Concurrent;
2+
using Xunit.Abstractions;
3+
using Xunit.Sdk;
4+
5+
namespace xRetry.v3
6+
{
7+
/// <summary>
8+
/// An XUnit message bus that can block messages from being passed until we want them to be.
9+
/// </summary>
10+
public class BlockingMessageBus : IMessageBus
11+
{
12+
private readonly IMessageBus underlyingMessageBus;
13+
private readonly MessageTransformer messageTransformer;
14+
private ConcurrentQueue<IMessageSinkMessage> messageQueue = new ConcurrentQueue<IMessageSinkMessage>();
15+
16+
public BlockingMessageBus(IMessageBus underlyingMessageBus, MessageTransformer messageTransformer)
17+
{
18+
this.underlyingMessageBus = underlyingMessageBus;
19+
this.messageTransformer = messageTransformer;
20+
}
21+
22+
public bool QueueMessage(IMessageSinkMessage rawMessage)
23+
{
24+
// Transform the message to apply any additional functionality, then intercept & store it for replay later
25+
IMessageSinkMessage transformedMessage = messageTransformer.Transform(rawMessage);
26+
messageQueue.Enqueue(transformedMessage);
27+
28+
// Returns if execution should continue. Since we are intercepting the message, we
29+
// have no way of checking this so always continue...
30+
return true;
31+
}
32+
33+
public void Clear()
34+
{
35+
messageQueue = new ConcurrentQueue<IMessageSinkMessage>();
36+
}
37+
38+
/// <summary>
39+
/// Write the cached messages to the underlying message bus
40+
/// </summary>
41+
public void Flush()
42+
{
43+
while (messageQueue.TryDequeue(out IMessageSinkMessage message))
44+
{
45+
underlyingMessageBus.QueueMessage(message);
46+
}
47+
}
48+
49+
public void Dispose()
50+
{
51+
// Do not dispose of the underlying message bus - it is an externally owned resource
52+
}
53+
}
54+
}

0 commit comments

Comments
 (0)