Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[gguf] Add descriptions to quantization types #615

Merged
merged 9 commits into from
Apr 10, 2024
Merged

[gguf] Add descriptions to quantization types #615

Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
f2690a9
[gguf] Describe data & quant types
mishig25 Apr 9, 2024
fdb78f1
cleaner & shorter description
mishig25 Apr 10, 2024
7e83f0b
chore
mishig25 Apr 10, 2024
b1aeff5
Q4_0, Q4_1, Q5_0, Q5_1, Q8_0, Q8_1
mishig25 Apr 10, 2024
94e0aed
cleaner & shorter
mishig25 Apr 10, 2024
1c1ef0e
src comments
mishig25 Apr 10, 2024
fc27b4c
add src comments
mishig25 Apr 10, 2024
240f0df
better js object
mishig25 Apr 10, 2024
0d425f9
re-export
mishig25 Apr 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
chore
  • Loading branch information
@mishig25
mishig25 committed Apr 10, 2024
commit 7e83f0bbd202f6bea6ab6e692c577f5d570f89ad
2 changes: 1 addition & 1 deletion packages/gguf/src/quant_descriptions.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ export const QUANT_DESCRIPTIONS: Record<GGMLQuantizationType, string> = {
[GGMLQuantizationType.Q3_K]: `3-bit quantization (q). Super-blocks with 16 blocks, each block has 16 weights. Block scales (d) is quantized with 6 bits, resulting. 3.4375 bits-per-weight. Weights are obtained by w = d * q.`, // src: https://github.com/ggerganov/llama.cpp/pull/1684#issue-1739619305
[GGMLQuantizationType.Q4_K]: `4-bit quantization (q). Super-blocks with 8 blocks, each block has 32 weights. Block scales (d) & mins (m) are quantized with 6 bits, resulting. 4.5 bits-per-weight are obtained by w = d * q + m.`, // src: https://github.com/ggerganov/llama.cpp/pull/1684#issue-1739619305
[GGMLQuantizationType.Q5_K]: `5-bit quantization (q). Super-blocks with 8 blocks, each block has 32 weights. Block scales (d) & mins (m) are quantized with 6 bits, resulting in 5.5 bits-per-weight. Weights are obtained by w = d * q + m.`, // src: https://github.com/ggerganov/llama.cpp/pull/1684#issue-1739619305
[GGMLQuantizationType.Q6_K]: `6-bit quantization (q). Super-blocks with 16 blocks, each block has 16 weights. Scales (d) are quantized with 8 bits, resulting in 6.5625 bits-per-weight. Weights are obtained by w = d * q.`, // src: https://github.com/ggerganov/llama.cpp/pull/1684#issue-1739619305
[GGMLQuantizationType.Q6_K]: `6-bit quantization (q). Super-blocks with 16 blocks, each block has 16 weights. Block scales (d) is quantized with 8 bits, resulting in 6.5625 bits-per-weight. Weights are obtained by w = d * q.`, // src: https://github.com/ggerganov/llama.cpp/pull/1684#issue-1739619305
[GGMLQuantizationType.Q8_K]: `8-bit quantization (q). Only used for quantizing intermediate results. The difference to the existing Q8_0 is that the block size is 256. All 2-6 bit dot products are implemented for this quantization type. Weights are obtained by w = d * q.`, // src: https://github.com/ggerganov/llama.cpp/pull/1684#issue-1739619305
[GGMLQuantizationType.IQ2_XXS]: "", // todo: add description
Copy link
Collaborator Author

@mishig25 mishig25 Apr 9, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ikawrakow @ggerganov @younesbelkada @FL33TW00D or anyone, I'd greatly appreciate if you can supply any of the the missing descriptions.

You can just post as a comment and I can add/commit it to the file

Copy link
Contributor

@younesbelkada younesbelkada Apr 9, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to ggerganov/llama.cpp#5063 + offline discussion with @FL33TW00D I would say:

Q4_0: Round-to-Nearest group-wise quantization with a blocksize of 32 and 4-bit quantized weights. Block weights are simply given by w = q * s. Legacy quantization method, and not really used by the community as of today.

I would say Q5_0 / Q8_0 is also RTN but for 5 / 8-bit, not sure yet what _1 stands for Q4_1 - I will let others comment on this

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i might got it right for QK_1:

Q4_1: Round-to-Nearest group-wise quantization with a blocksize of 32 and 4-bit quantized weights with an additional term that is added after the de-quantization step. Block weights are simply given by w = q * s + m with m being the minimum of the block. Legacy quantization method, and not really used by the community as of today.

Same comment applies for Q5_1 and Q8_1 I think

[GGMLQuantizationType.IQ2_XS]: "", // todo: add description
Expand Down
Loading