mirror of
https://github.com/mudler/LocalAI.git
synced 2024-06-07 19:40:48 +00:00
bc8f648a91
The default sampler on some models don't return enough candidates which leads to a false sense of randomness. Tracing back the code it looks that with the temperature sampler there might not be enough candidates to pick from, and since the seed and "randomness" take effect while picking a good candidate this yields to the same results over and over. Fixes https://github.com/mudler/LocalAI/issues/1723 by updating the examples and documentation to use mirostat instead.
30 lines
686 B
YAML
30 lines
686 B
YAML
name: phi-2
|
|
context_size: 2048
|
|
f16: true
|
|
gpu_layers: 90
|
|
mmap: true
|
|
trimsuffix:
|
|
- "\n"
|
|
parameters:
|
|
model: huggingface://TheBloke/phi-2-GGUF/phi-2.Q8_0.gguf
|
|
temperature: 0.2
|
|
top_k: 40
|
|
top_p: 0.95
|
|
seed: -1
|
|
|
|
mirostat: 2
|
|
mirostat_eta: 1.0
|
|
mirostat_tau: 1.0
|
|
template:
|
|
chat: &template |-
|
|
Instruct: {{.Input}}
|
|
Output:
|
|
completion: *template
|
|
|
|
usage: |
|
|
To use this model, interact with the API (in another terminal) with curl for instance:
|
|
curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
|
|
"model": "phi-2",
|
|
"messages": [{"role": "user", "content": "How are you doing?", "temperature": 0.1}]
|
|
}'
|