Add quantized qwen2-0.5b

[bil-ash]

Add quantized(q4f16) qwen2-0.5b to the list of supported models. PR must be merged before merging this.

[Neet-Nestor]

Related PRs:

• Add quantized qwen2-0.5b #490
• Add support for quantized qwen2-0.5b web-llm-chat#44
• Add support for quantized qwen2-0.5b binary-mlc-llm-libs#128

[CharlieFRuan]

Thanks a lot for the contribution. Some minor changes, one on consistency on naming, and one on the required MB after calculation

[CharlieFRuan]

Suggested change

	vram_required_MB: 500,//rough estimate
	vram_required_MB: 944.62,

[CharlieFRuan]

Suggested change

	model_id: "Qwen2-0.5B-Instruct-q4f16-MLC",
	model_id: "Qwen2-0.5B-Instruct-q4f16_1-MLC",

[bil-ash]

By the way, what is the formula for calculating VRAM required?
Also, how does web-llm acquire VRAM- I mean does it take up the entire VRAM specified by vram_required_MB at initialization or does it take up limited amount of VRAM and then acquire more if required?

[CharlieFRuan]

Thanks for making the changes!

For VRAM, it is mainly three parts: model size, intermediate buffer size (for various matrix multiplications, etc.), and KV cache size. The sum of the first two is estimated during python -m mlc_llm compile and shown in the output. For KV cache size, it is context_window_size * head_dim * num_kv_heads * num_layers * 2 (K and V) * 2 (num bytes for f16, 4 for f32).

The vram_required_MB is merely an estimation, so it does not play a role at all in runtime. I believe it takes up most (if not all) of the required VRAM at initialization. Therefore, prefill chunk size would play a role in limiting the amount of memory required for long prompts -- it would simply chunk them to keep the intermediate buffer size the same.

[bil-ash]

Thanks for making the changes!

For VRAM, it is mainly three parts: model size, intermediate buffer size (for various matrix multiplications, etc.), and KV cache size. The sum of the first two is estimated during python -m mlc_llm compile and shown in the output. For KV cache size, it is context_window_size * head_dim * num_kv_heads * num_layers * 2 (K and V) * 2 (num bytes for f16, 4 for f32).

The vram_required_MB is merely an estimation, so it does not play a role at all in runtime. I believe it takes up most (if not all) of the required VRAM at initialization. Therefore, prefill chunk size would play a role in limiting the amount of memory required for long prompts -- it would simply chunk them to keep the intermediate buffer size the same.

So, VRAM=model+ intermediate buffer +KV cache.
So, for this case, please specify all the three components.
I am asking this because I would like to have Qwen2-0.5B but with 32k context. So, if we reduce the prefill chunk size to 1k, the calculation would be something like-
VRAM for 32k context= model+ (intermediate buffer)/2 + 8*KV cache
and may be there won't be much of a difference in VRAM

[CharlieFRuan]

The head_dim etc. can be found in the mlc-chat-config.json, so it is ctx * 64 * 2 * 24 * 2 * 2 bytes. For 4K, it is 48MB, so the other two components is 944.62 - 48 = 896.62. For 32K, it would be 8 * 48MB. I'm personally not sure whether QWen2 0.5B would support such a long context.

[bil-ash]

Thanks for the info

[CharlieFRuan]

I just ran compile again and got following info:

• For 1024 chunk size: 896.62 MB (Parameters: 265.12 MB. Temporary buffer: 631.50 MB)
• For 2048 chunk size: 1528.12 MB (Parameters: 265.12 MB. Temporary buffer: 1263.00 MB)

I guess I used the wrong estimation, it should be 1528.12 + 48 MB instead since the wasm you uploaded uses 2048 chunk size