LLM on the laptop

Variant 1 โ€“ direct installation of Ollama

Install Ollama:

curl -fsSL https://ollama.com/install.sh | sh

Install the language model:

ollama pull gemma4:12b

Optimization for the LLM in the file /etc/systemd/system/ollama.service:

Add to the content:

[Service]
Environment="OLLAMA_NUM_THREADS=6"
Environment="OLLAMA_FLASH_ATTENTION=1"
Environment="OLLAMA_KV_CACHE_TYPE=q8_0"
Environment="OLLAMA_KEEP_ALIVE=-1"
LimitMEMLOCK=infinity

Restart the Ollama service:

sudo systemctl daemon-reload
sudo systemctl restart ollama

Result: slow

Variant 2: LM Studio

Result: slow

Variant 3: Ollama in Docker