Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor the vllm specific code to become model server agnostic #383

Open
Tracked by #362
ahg-g opened this issue Feb 21, 2025 · 4 comments
Open
Tracked by #362

Refactor the vllm specific code to become model server agnostic #383

ahg-g opened this issue Feb 21, 2025 · 4 comments
Assignees

Comments

@ahg-g
Copy link
Contributor

ahg-g commented Feb 21, 2025

Currently the epp hardcodes the metrics to vllm. To support other model servers, we should make this logic configurable via flags to allow integrations with any model server adhering to the model server protocol as defined in https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/docs/proposals/003-endpoint-picker-protocol/README.md

@ahg-g
Copy link
Contributor Author

ahg-g commented Feb 21, 2025

/assign @BenjaminBraunDev

@k8s-ci-robot
Copy link
Contributor

@ahg-g: GitHub didn't allow me to assign the following users: BenjaminBraunDev.

Note that only kubernetes-sigs members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign @BenjaminBraunDev

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@BenjaminBraunDev
Copy link
Contributor

Commenting to allow assignment!

@ahg-g
Copy link
Contributor Author

ahg-g commented Feb 26, 2025

/assign @BenjaminBraunDev

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants