LocalAI

History

Ettore Di Giacinto bc8f648a91 fix(doc/examples): set defaults to mirostat (#1820 ) The default sampler on some models don't return enough candidates which leads to a false sense of randomness. Tracing back the code it looks that with the temperature sampler there might not be enough candidates to pick from, and since the seed and "randomness" take effect while picking a good candidate this yields to the same results over and over. Fixes https://github.com/mudler/LocalAI/issues/1723 by updating the examples and documentation to use mirostat instead.		2024-03-11 19:49:03 +01:00
..
llava	Delete examples/configurations/llava/README.md	2023-11-18 15:01:39 +01:00
mistral	docs(examples): Add mistral example (#1214 )	2023-10-25 20:56:12 +02:00
mixtral	docs(mixtral): add mixtral example (#1449 )	2023-12-16 17:44:43 +01:00
README.md	docs(phi-2): add example (#1525 )	2024-01-01 10:51:47 +01:00
phi-2.yaml	fix(doc/examples): set defaults to mirostat (#1820 )	2024-03-11 19:49:03 +01:00

README.md

Advanced configuration

This section contains examples on how to install models manually with config files.

Prerequisites

First clone LocalAI:

git clone https://github.com/go-skynet/LocalAI

cd LocalAI

Setup the model you prefer from the examples below and then start LocalAI:

docker compose up -d --pull always

If LocalAI is already started, you can restart it with

docker compose restart

See also the getting started: https://localai.io/basics/getting_started/

You can also start LocalAI just with docker:

docker run -p 8080:8080 -v $PWD/models:/models -ti --rm quay.io/go-skynet/local-ai:master --models-path /models --threads 4

Mistral

To setup mistral copy the files inside mistral in the models folder:

cp -r examples/configurations/mistral/* models/

Now download the model:

wget https://huggingface.co/TheBloke/Mistral-7B-OpenOrca-GGUF/resolve/main/mistral-7b-openorca.Q6_K.gguf -O models/mistral-7b-openorca.Q6_K.gguf

LLaVA

Setup

cp -r examples/configurations/llava/* models/
wget https://huggingface.co/mys/ggml_bakllava-1/resolve/main/ggml-model-q4_k.gguf -O models/ggml-model-q4_k.gguf
wget https://huggingface.co/mys/ggml_bakllava-1/resolve/main/mmproj-model-f16.gguf -O models/mmproj-model-f16.gguf

Try it out

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
     "model": "llava",
     "messages": [{"role": "user", "content": [{"type":"text", "text": "What is in the image?"}, {"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg" }}], "temperature": 0.9}]}'

Phi-2

cp -r examples/configurations/phi-2.yaml models/

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
     "model": "phi-2",
     "messages": [{"role": "user", "content": "How are you doing?", "temperature": 0.1}]
}'

Mixtral

cp -r examples/configuration/mixtral/* models/
wget https://huggingface.co/TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUF/resolve/main/mixtral-8x7b-instruct-v0.1.Q2_K.gguf -O models/mixtral-8x7b-instruct-v0.1.Q2_K.gguf

Test it out

curl http://localhost:8080/v1/completions -H "Content-Type: application/json" -d '{
     "model": "mixtral",
     "prompt": "How fast is light?",                                                                                    
     "temperature": 0.1 }'