You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Objectives
Re-enable test_service.ipynb in tox notebooks to test the evaluation service
Todos
enable a redis container for the notebooks workflow
pass the redis container host(?) to the notebook in an environment variable redis_ip
uncomment the test_service.ipynb
Minimal testing requirements
N/A - this change enables an existing test to run
Additional context checks.yml notebooks have recently been enabled by Jeremy / @hagrid67 in PR #61 @fnberta would you mind taking a look at enabling redis in the workflow please? (Or I'll take a look in a few days)
The text was updated successfully, but these errors were encountered:
I think we have plans to offer the evaluation service as a key feature of flatland, so that people can publish their benchmarks. We discussed at the last dev meeting starting work by the end of the year; Manuel is doing some planning. So I'd suggest that it's worth putting the effort into building some tests around it. The existing evaluator uses redis; this comes from the AICrowd architecture.
The notebook is one I built 2-3 years ago to run a simple test to start an evaluator service and then run a client against it. It "manually" starts and stops processes in the background so I'm sure it could be improved; eg we might want to run Redis in one container, and the evaluator in another, and then run the client in the foreground.
Until then, it seems fairly low effort to get this existing test working on github CI. It seems to run OK on my local Ubuntu 22.04 with redis running.
Objectives
Re-enable
test_service.ipynb
intox notebooks
to test the evaluation serviceTodos
redis
container for the notebooks workflowredis_ip
test_service.ipynb
Minimal testing requirements
N/A - this change enables an existing test to run
Additional context
checks.yml
notebooks have recently been enabled by Jeremy / @hagrid67 in PR #61@fnberta would you mind taking a look at enabling
redis
in the workflow please? (Or I'll take a look in a few days)The text was updated successfully, but these errors were encountered: