apepkuss79's picture
Update README.md
1b1045f verified
metadata
base_model: Qwen/Qwen2.5-32B-Instruct
license: apache-2.0
license_link: https://huggingface.co/Qwen/Qwen2.5-32B-Instruct/blob/main/LICENSE
model_creator: Qwen
model_name: Qwen2.5-32B-Instruct
quantized_by: Second State Inc.
language:
  - en
pipeline_tag: text-generation
tags:
  - chat

Qwen2.5-32B-Instruct-GGUF

Original Model

Qwen/Qwen2.5-32B-Instruct

Run with LlamaEdge

  • LlamaEdge version: v0.14.3

  • Prompt template

    • Prompt type: chatml

    • Prompt string

      <|im_start|>system
      {system_message}<|im_end|>
      <|im_start|>user
      {prompt}<|im_end|>
      <|im_start|>assistant
      
  • Context size: 128000

  • Run as LlamaEdge service

    wasmedge --dir .:. --nn-preload default:GGML:AUTO:Qwen2.5-32B-Instruct-Q5_K_M.gguf \
      llama-api-server.wasm \
      --model-name Qwen2.5-32B-Instruct \
      --prompt-template chatml \
      --ctx-size 128000
    
  • Run as LlamaEdge command app

    wasmedge --dir .:. --nn-preload default:GGML:AUTO:Qwen2.5-32B-Instruct-Q5_K_M.gguf \
      llama-chat.wasm \
      --prompt-template chatml \
      --ctx-size 128000
    

Quantized with llama.cpp b3751