Edit model card
MidnightMiqu

Midnight-Miqu-103B-v1.0 - EXL2 4.0bpw

This is a 4.0bpw EXL2 quant of sophosympatheia/Midnight-Miqu-103B-v1.0

Details about the model and the merge info can be found at the above mode page.

Quant Details

This is the script used for quantization.

#!/bin/bash

# Activate the conda environment
source ~/miniconda3/etc/profile.d/conda.sh
conda activate exllamav2

# Define variables
MODEL_DIR="models/sophosympatheia_Midnight-Miqu-103B-v1.0"
OUTPUT_DIR="exl2_midnight103b"
MEASUREMENT_FILE="measurements/midnight103b.json"

BIT_PRECISION=4.0
CONVERTED_FOLDER="models/Midnight-Miqu-103B_exl2_4.0bpw"

# Create directories
mkdir $OUTPUT_DIR
mkdir $CONVERTED_FOLDER

# Run conversion commands
python convert.py -i $MODEL_DIR -o $OUTPUT_DIR -nr -om $MEASUREMENT_FILE
python convert.py -i $MODEL_DIR -o $OUTPUT_DIR -nr -m $MEASUREMENT_FILE -b $BIT_PRECISION -cf $CONVERTED_FOLDER
Downloads last month
12
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Collection including Dracones/Midnight-Miqu-103B-v1.0_exl2_4.0bpw