local kubernetes setup

[savingoyal]

build wheel -

python setup.py sdist bdist_wheel

install resulting wheel -

python -m pip install --force-reinstall dist/...whl

deploy using

metaflow-dev up

tear down using

metaflow-dev down

use using

metaflow-dev shell

access UI using

metaflow-dev ui

[josephsirak]

This is getting pretty close.

• There are a couple of indentation issues plus a couple of configuration issues

• The other main feedback I have is on the services picker . I believe minio shouldn't be optional similar to postgres. I know that there is a use case for running just the metadata-service, which only needs the postgres database, but for anyone who is trying to use the metaflow-ui or just run flows without adding additional wiring for the datastore so they use S3 or set the --datastore=local flag, they will run into hard to debug issues if they didn't chose to install minio. Also, the metaflow-ui has an explicit dependency on it.

[josephsirak]

Got an error with make shell when trying to run a flow

metaflow.exception.MetaflowException: Unable to locate METAFLOW_PROFILE 'local' in '/.../metaflow/.devtools')

looks like the metaflow config file is stored as metaflow_config.json instead of config_local.json

[savingoyal]

interesting - looks like some commits are missing

[savingoyal]

@saikonen i believe your commits are missing

[josephsirak]

Looks like event publishing works, but I don't think this is wired up properly to trigger workflows when it receives events:

{"level":"error","ts":1740419561.4127948,"logger":"argo-events.eventbus-controller","caller":"sensor/controller.go:72","msg":"reconcile error","namespace":"default","senso
r":"demoforeachflow","error":"eventbus default not found","stacktrace":"github.com/argoproj/argo-events/pkg/reconciler/sensor.(*reconciler).Reconcile\n\t/home/runner/work/
argo-events/argo-events/pkg/reconciler/sensor/controller.go:72\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/s
igs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/
home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controlle
r).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/inte
rnal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}

[savingoyal]

This is getting pretty close.

• There are a couple of indentation issues plus a couple of configuration issues
• The other main feedback I have is on the services picker . I believe minio shouldn't be optional similar to postgres. I know that there is a use case for running just the metadata-service, which only needs the postgres database, but for anyone who is trying to use the metaflow-ui or just run flows without adding additional wiring for the datastore so they use S3 or set the --datastore=local flag, they will run into hard to debug issues if they didn't chose to install minio. Also, the metaflow-ui has an explicit dependency on it.

thanks! re: minio being optional - the expectation is that more often than not, the user will choose the deploy everything (we can clarify that in services picker) - it's only the developers (and tests) who need to cherry pick.