mistralai

/

Mistral-Nemo-Instruct-2407

like 1.1k

mistral-demo ./nemo/
Traceback (most recent call last):
  File "/home/cloudyu/.local/bin/mistral-demo", line 8, in <module>
    sys.exit(mistral_demo())
  File "/home/cloudyu/.local/lib/python3.10/site-packages/mistral_inference/main.py", line 207, in mistral_demo
    fire.Fire(demo)
  File "/home/cloudyu/.local/lib/python3.10/site-packages/fire/core.py", line 143, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/cloudyu/.local/lib/python3.10/site-packages/fire/core.py", line 477, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/cloudyu/.local/lib/python3.10/site-packages/fire/core.py", line 693, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/cloudyu/.local/lib/python3.10/site-packages/mistral_inference/main.py", line 191, in demo
    generated_words.append(tokenizer.decode(encoded_prompts[i] + x))
  File "/home/cloudyu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py", line 224, in decode
    return "".join(self._decode_all(tokens, special_token_policy=self._special_token_policy))
  File "/home/cloudyu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py", line 203, in _decode_all
    raise ValueError(f"Special tokens not allowed in this context: {list(group)}")
ValueError: Special tokens not allowed in this context: [1]

mistral_common 1.3.1
mistral_inference 1.3.0

I spent whole day to install packages and test new models
both mamba-codestral-7B-v0.1 and Mistral-Nemo-Instruct-2407 don't work.

https://huggingface.co/mistralai/mamba-codestral-7B-v0.1/discussions/4#6699047cd77a8b28703e029c

I'm having the same issue with special tokens even running the function calling with texts in Portuguese example in the README:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[16], line 33
     30 tokens = tokenizer.encode_chat_completion(completion_request).tokens
     32 out_tokens, _ = generate([tokens], model, max_tokens=256, temperature=0.35, eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id)
---> 33 result = tokenizer.decode(out_tokens[0])
     35 print(result)

File /usr/local/lib/python3.10/dist-packages/mistral_common/tokens/tokenizers/mistral.py:148, in MistralTokenizer.decode(self, tokens)
    147 def decode(self, tokens: List[int]) -> str:
--> 148     return self.instruct_tokenizer.decode(tokens)

File /usr/local/lib/python3.10/dist-packages/mistral_common/tokens/tokenizers/sentencepiece.py:199, in InstructTokenizerBase.decode(self, tokens)
    198 def decode(self, tokens: List[int]) -> str:
--> 199     return self.tokenizer.decode(tokens)

File /usr/local/lib/python3.10/dist-packages/mistral_common/tokens/tokenizers/tekken.py:224, in Tekkenizer.decode(self, tokens)
    223 def decode(self, tokens: List[int]) -> str:
--> 224     return "".join(self._decode_all(tokens, special_token_policy=self._special_token_policy))

File /usr/local/lib/python3.10/dist-packages/mistral_common/tokens/tokenizers/tekken.py:203, in Tekkenizer._decode_all(self, tokens, special_token_policy)
    201 if is_special:
    202     if special_token_policy == SpecialTokenPolicy.RAISE:
--> 203         raise ValueError(f"Special tokens not allowed in this context: {list(group)}")
    204     elif special_token_policy == SpecialTokenPolicy.KEEP:
    205         decoded.extend(self._all_special_tokens[t] for t in group)

ValueError: Special tokens not allowed in this context: [9]

Thanks for the notice! Should be fixed with mistral_inference >= 1.3.1 with this PR: https://github.com/mistralai/mistral-inference/pull/199

Can you run

pip install --upgrade mistral_inference and try again?

I too am having issues running this.

pip install --upgrade mistral_inference was successfully updated but failed to fix. Tested with Q8_0 and f16.gguf with 128GB of RAM, more than enough to do the job.

Traceback (most recent call last):
File "C:\text-generation-webui-main\text-generation-webui-main\modules\ui_model_menu.py", line 248, in load_model_wrapper
shared.model, shared.tokenizer = load_model(selected_model, loader)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\text-generation-webui-main\text-generation-webui-main\modules\models.py", line 94, in load_model
output = load_func_maploader
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\text-generation-webui-main\text-generation-webui-main\modules\models.py", line 275, in llamacpp_loader
model, tokenizer = LlamaCppModel.from_pretrained(model_file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\text-generation-webui-main\text-generation-webui-main\modules\llamacpp_model.py", line 85, in from_pretrained
result.model = Llama(**params)
^^^^^^^^^^^^^^^
File "C:\text-generation-webui-main\text-generation-webui-main\installer_files\env\Lib\site-packages\llama_cpp_cuda\llama.py", line 358, in init
self._model = self._stack.enter_context(contextlib.closing(_LlamaModel(
^^^^^^^^^^^^
File "C:\text-generation-webui-main\text-generation-webui-main\installer_files\env\Lib\site-packages\llama_cpp_cuda_internals.py", line 54, in init
raise ValueError(f"Failed to load model from file: {path_model}")
ValueError: Failed to load model from file: models\Mistral-Nemo-Instruct-2407-f16.gguf

Exception ignored in: <function LlamaCppModel.__del__ at 0x0000027C09ABBC40>
Traceback (most recent call last):
File "C:\text-generation-webui-main\text-generation-webui-main\modules\llamacpp_model.py", line 33, in del
del self.model
^^^^^^^^^^
AttributeError: 'LlamaCppModel' object has no attribute 'model'