-
Notifications
You must be signed in to change notification settings - Fork 7k
[docs] [llm] [template] update ray version + use anyscale image for the gpt-oss deployment #59125
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: examples-deployment-serve-llm
Are you sure you want to change the base?
[docs] [llm] [template] update ray version + use anyscale image for the gpt-oss deployment #59125
Conversation
…image instaed of dockerfile Signed-off-by: Aydin Abiar <aydin@anyscale.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review
This pull request updates the Ray and vLLM versions in several documentation tutorials and switches the gpt-oss tutorial to use a pre-built Anyscale image instead of a Dockerfile, which is a great improvement for user experience.
I've found a few inconsistencies in the gpt-oss tutorial documentation (README.md and notebook.ipynb) where some values seem to have been copied from another tutorial. I've left specific comments with suggestions to fix them.
Additionally, I noticed that in the "Custom Dockerfile" examples across several of the updated markdown and notebook files, the commented-out image_uri still refers to the old version 2.49.0. While these are not part of the diff, it would be good to update them to 2.52.0 for consistency.
Overall, the changes look good and improve the documentation.
doc/source/serve/tutorials/deployment-serve-llm/gpt-oss/README.md
Outdated
Show resolved
Hide resolved
doc/source/serve/tutorials/deployment-serve-llm/gpt-oss/README.md
Outdated
Show resolved
Hide resolved
doc/source/serve/tutorials/deployment-serve-llm/gpt-oss/notebook.ipynb
Outdated
Show resolved
Hide resolved
doc/source/serve/tutorials/deployment-serve-llm/gpt-oss/notebook.ipynb
Outdated
Show resolved
Hide resolved
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: Aydin Abiar <62435714+Aydin-ab@users.noreply.github.com>
Signed-off-by: Aydin Abiar <aydin@anyscale.com>
doc/source/serve/tutorials/deployment-serve-llm/gpt-oss/README.md
Outdated
Show resolved
Hide resolved
doc/source/serve/tutorials/deployment-serve-llm/gpt-oss/notebook.ipynb
Outdated
Show resolved
Hide resolved
kunling-anyscale
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
looks good, left a few suggestions
…ok.ipynb Co-authored-by: kunling-anyscale <kunling@anyscale.com> Signed-off-by: Aydin Abiar <62435714+Aydin-ab@users.noreply.github.com>
Signed-off-by: Aydin Abiar <aydin@anyscale.com>
Updating to a more recent release of ray where ray serve llm is more stable with vllm. Also using an anyscale image instead of a dockerfile in the anyscale service config file for gpt-oss. Previously, we used a dockerfile because there was compatibility issues with vllm but this has been fixed. Building from a dockerfile is a pretty bad experience for a user (can take ~4-5 min for the service to spin up)