Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add vLLM to local-apps #744

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Add vLLM to local-apps #744

wants to merge 5 commits into from

Conversation

mgoin
Copy link
Contributor

@mgoin mgoin commented Jun 7, 2024 •
edited
Loading

vLLM is a high-throughput and memory-efficient open-source serving engine for LLMs.

vLLM is fast with:

  • State-of-the-art serving throughput
  • Efficient management of attention key and value memory with PagedAttention
  • Continuous batching of incoming requests
  • Fast model execution with CUDA/HIP graph
  • Quantization: FP8, GPTQ, AWQ, SqueezeLLM, FP8 KV Cache
  • Optimized CUDA kernels

Transparent Logo:
vllm-logo-text-light

@krampstudio krampstudio mentioned this pull request Sep 2, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@osanseviero osanseviero Awaiting requested review from osanseviero

@SBrandeis SBrandeis Awaiting requested review from SBrandeis SBrandeis is a code owner

@gary149 gary149 Awaiting requested review from gary149 gary149 is a code owner

@Wauplin Wauplin Awaiting requested review from Wauplin Wauplin is a code owner

@julien-c julien-c Awaiting requested review from julien-c julien-c is a code owner

@pcuenca pcuenca Awaiting requested review from pcuenca pcuenca is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@mgoin @krampstudio