Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GitHub Marketplace Models for Remote Inference #98

Closed
potatoqualitee opened this issue Sep 9, 2024 · 4 comments
Closed

GitHub Marketplace Models for Remote Inference #98

potatoqualitee opened this issue Sep 9, 2024 · 4 comments
Assignees
@a1exwang

Comments

@potatoqualitee
Copy link

Hello,
How can I make GitHub models work with AI Toolkit?

image

image

Perhaps it's bc I cannot append the model name to the body as the example suggests.

curl -X POST "https://models.inference.ai.azure.com/chat/completions" \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $GITHUB_TOKEN" \
    -d '{
        "messages": [
            {
                "role": "system",
                "content": "You are a helpful assistant."
            },
            {
                "role": "user",
                "content": "What is the capital of France?"
            },
            {
                "role": "assistant",
                "content": "The capital of France is Paris."
            },
            {
                "role": "user",
                "content": "What about Spain?"
            }
        ],
        "model": "gpt-4o-mini"
    }'
@swatDong
Copy link
Contributor

The latest pre-release of AI Toolkit has integrated GitHub models in model catalog.
(cc @a1exwang if I missed anything)

Please try:

  1. Install latest pre-release
    image

  2. Open Model Catalog
    image

  3. Add and try GitHub models in playground
    image

@potatoqualitee
Copy link
Author

Oh! Didn't realize these were from GitHub. Does this count against our github.com/marketplace models limits?

https://docs.github.com/en/github-models/prototyping-with-ai-models#rate-limits

Can people who don't have preview access to the model marketplace access it via AI Toolkit?

@swatDong
Copy link
Contributor

Oh! Didn't realize these were from GitHub. Does this count against our github.com/marketplace models limits?

https://docs.github.com/en/github-models/prototyping-with-ai-models#rate-limits

Can people who don't have preview access to the model marketplace access it via AI Toolkit?

Opening model in AI Toolkit shares the same access&limit as opening on GitHub.
If signing in an account in VSCode without preview access, you cannot chat with those models in AI Toolkit.
image

@potatoqualitee
Copy link
Author

Merci 👍🏼

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@a1exwang a1exwang

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@swatDong @potatoqualitee @a1exwang