Installing a local instance of an LLM - Post 1 of 3

In this post I will basically mention two LLMs that I am going to install and thereafter comment on them.

Most AI apps seem to use python to run. So once you have installed 

1. Install Ubuntu linux version 22 (Jammy Jellyfish). 

a. Go to https://ubuntu.com/download 

b. Get the iso file

c. Copy it to a flash disk using Etcher (https://etcher.io) 

d. Insert flash disk, reboot, hit f2 on the keyboard; enter BIOS and tell your PC to boot off the flash disk. Take it from there. 

e. Install PHP 3.10

sudo apt-get install python3.10 


2. Install either or both of GPT4ALL or privateGPT.

GPT4ALL: https://github.com/nomic-ai/gpt4all

PrivateGPT: https://github.com/imartinez/privateGPT

For example

mkdir git

cd git

git clone https://github.com/imartinez/privateGPT

git clone https://github.com/nomic-ai/gpt4all


3. Once those are downloaded ...

cd git/privateGPT 


or

cd git/gpt4all

then

pip install 'pygpt4all==v1.0.1' --force-reinstall

sudo apt install libxcb-cursor0

python3.10 -m pip install -r requirements.txt

Note that this last step almost always fails with some error or dependency. I find that the following commands usually fix it.

Note also that the dependencies, LLM files and libraries are usually stored in ~/.cache/nomicai/ or similar.

4. Fix its nonsense

sudo apt-get install python3-dev

python3.10 -m pip install --upgrade pip

sudo apt install build-essential

python3.10 -m pip install hnswlib

python3.10 -m pip install chromadb

then

python3.10 -m pip install chroma-migrate

chroma-migrate

python3.10 -m pip install -r requirements.txt

Popular posts from this blog

Testing ChatGPT on coding - a script to make CSV to ICS

 So I gave ChatGPT a task of writing a CSV to ICS convertor, to take a spreadsheet of events and turn it into individual calendar file invitations. The idea is you can make a calendar with lots of event items in a spreadsheet, and then output ICS that you can just double-click to add to your calendar automatically. So here's the spreadsheet: "Event Name","Event Start Date","Event End Date","Event Start Time","Event End Time","event description or notes","URL applicable" "Meeting with John","2024-06-10","2024-06-10","09:00","10:00","Discuss project updates.","http://example.com" And here's the script ChatGPT made. It seems to get confused with things like Daylight Savings and/or Timezone but you can fix that manually. It took some debugging and I had to copy/paste my Mac's manpage for DATE into ChatGPT, but it got it eventually. The only
Read more

Ollama3 works - running a local LLM

Do this: curl -fsSL https://ollama.com/install.sh | sh ollama create llama3  It will then download the llama3 model, about 4.7 gb to run it: ollama run llama3  if you want it to read and summarise a textfile called textfile.txt: ollama run llama3 "Read the text file $(cat textfile.txt) and summarise the findings in one paragraph of no more than 300 words"  And here's a gui for it: https://github.com/amithkoujalgi/ollama-pdf-bot The system runs as a service (daemon) on Linux. CPU usage is high but not bad and response time much better than localGPT. It stores the model files (large LLM blobs) in  /usr/share/ollama/.ollama/models The models can be dowloaded from huggingface.io or https://ollama.com/library
Read more

What is a "token" , what is "temperature", and what is "n-shot"?

 TL;DR: token is a meaning unit; temperature is randomness. Token: In the context of AI and natural language processing (NLP), a "token" refers to the smallest unit of text or language that a model can understand. Tokens are usually words or subwords, but they can also be characters or even smaller units, depending on the tokenization method used. Word Tokens: In most cases, a token corresponds to a word in a sentence. For example, the sentence "ChatGPT is helpful" would be tokenized into three word tokens: ["ChatGPT", "is", "helpful"]. Subword Tokens: Some models use subword tokenization methods like Byte-Pair Encoding (BPE) or WordPiece to split words into smaller units. For example, "unhappiness" might be tokenized into ["un", "happiness"]. Character Tokens: In some cases, especially for languages with complex characters or when dealing with very short text, tokens can be individual characters. For ex
Read more