mirror of
https://github.com/mudler/LocalAI.git
synced 2024-06-07 19:40:48 +00:00
bc8f648a91
The default sampler on some models don't return enough candidates which leads to a false sense of randomness. Tracing back the code it looks that with the temperature sampler there might not be enough candidates to pick from, and since the seed and "randomness" take effect while picking a good candidate this yields to the same results over and over. Fixes https://github.com/mudler/LocalAI/issues/1723 by updating the examples and documentation to use mirostat instead. |
||
|---|---|---|
| .. | ||
| llava | ||
| mistral | ||
| mixtral | ||
| phi-2.yaml | ||
| README.md | ||
Advanced configuration
This section contains examples on how to install models manually with config files.
Prerequisites
First clone LocalAI:
git clone https://github.com/go-skynet/LocalAI
cd LocalAI
Setup the model you prefer from the examples below and then start LocalAI:
docker compose up -d --pull always
If LocalAI is already started, you can restart it with
docker compose restart
See also the getting started: https://localai.io/basics/getting_started/
You can also start LocalAI just with docker:
docker run -p 8080:8080 -v $PWD/models:/models -ti --rm quay.io/go-skynet/local-ai:master --models-path /models --threads 4
Mistral
To setup mistral copy the files inside mistral in the models folder:
cp -r examples/configurations/mistral/* models/
Now download the model:
wget https://huggingface.co/TheBloke/Mistral-7B-OpenOrca-GGUF/resolve/main/mistral-7b-openorca.Q6_K.gguf -O models/mistral-7b-openorca.Q6_K.gguf
LLaVA
Setup
cp -r examples/configurations/llava/* models/
wget https://huggingface.co/mys/ggml_bakllava-1/resolve/main/ggml-model-q4_k.gguf -O models/ggml-model-q4_k.gguf
wget https://huggingface.co/mys/ggml_bakllava-1/resolve/main/mmproj-model-f16.gguf -O models/mmproj-model-f16.gguf
Try it out
curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "llava",
"messages": [{"role": "user", "content": [{"type":"text", "text": "What is in the image?"}, {"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg" }}], "temperature": 0.9}]}'
Phi-2
cp -r examples/configurations/phi-2.yaml models/
curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "phi-2",
"messages": [{"role": "user", "content": "How are you doing?", "temperature": 0.1}]
}'
Mixtral
cp -r examples/configuration/mixtral/* models/
wget https://huggingface.co/TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUF/resolve/main/mixtral-8x7b-instruct-v0.1.Q2_K.gguf -O models/mixtral-8x7b-instruct-v0.1.Q2_K.gguf
Test it out
curl http://localhost:8080/v1/completions -H "Content-Type: application/json" -d '{
"model": "mixtral",
"prompt": "How fast is light?",
"temperature": 0.1 }'