Four options so far

So this is an update on my experimentation with LLMs.

There are applications that let you access the models (LLMs) and models themselves. Each model is good at some things and bad at others. For example, ChatGPT is said to be bad at maths, whereas LLaMa is said to be better. Your mileage may vary. 

So far, I've experimented with the following apps:

- PrivateGPT (with a range of models)

- LocalGPT (with a range of models)

- ChatGPT (with GPT-3.5)

- GPT4All (with a range of models)

The following table documents the pro-con analysis:



Depending on your purpose, I'd suggest you use ChatGPT if you have a mere goal of getting information. If however you need summaries of actual documents (a repository), I suggest you look at the others. GPT4All is easiest to use and install. However, I noticed that it really tended to hallucinate and give inaccurate answers. The way it works is that you install the app and then it asks you which LLM you want to download to answer queries. It supports about 8 models. Of those, Falcon LLM was generally the fastest but it ignored the local repository and rather just made stuff up. LLaMA LLM was able to read the local repository but had a censorship tendency on anything controversial. There is also a "hermes" version of LLaMA but it is actually slower than the normal 2.7b LLaMA. So don't use it if you need speed. Also, GPT4All didn't seem to use the GPU. I checked up and it seems that it can't. However, its performance was much better than LocalGPT's, and LocalGPT was/is known to not use the GPU.

LocalGPT and PrivateGPT are very similar. However, of the two of them, I was only able to get one to use the GPU, and it had a limitation in that it (stupidly) tried to load everything into the VRAM in order to do so... in other words, the entire MODEL had to go into VRAM. Since my VRAM is quite small, I had to use a smaller model. (GPT Mini).

The more difficult options, PrivateGPT and LocalGPT, will be discussed in a subsequent post, but I suggest you simply go straight to PrivateGPT which seems to be a fork/clone of LocalGPT. I could not get LocalGPT to work with the GPU.


Popular posts from this blog

Testing ChatGPT on coding - a script to make CSV to ICS

 So I gave ChatGPT a task of writing a CSV to ICS convertor, to take a spreadsheet of events and turn it into individual calendar file invitations. The idea is you can make a calendar with lots of event items in a spreadsheet, and then output ICS that you can just double-click to add to your calendar automatically. So here's the spreadsheet: "Event Name","Event Start Date","Event End Date","Event Start Time","Event End Time","event description or notes","URL applicable" "Meeting with John","2024-06-10","2024-06-10","09:00","10:00","Discuss project updates.","http://example.com" And here's the script ChatGPT made. It seems to get confused with things like Daylight Savings and/or Timezone but you can fix that manually. It took some debugging and I had to copy/paste my Mac's manpage for DATE into ChatGPT, but it got it eventually. The only
Read more

Some rough notes on FreedomGPT and PrivateGPT

 The following are rough notes. I'll tidy them up when I get the steps 100% right. https://www.freedomgpt.com/ https://huggingface.co/mosaicml/mpt-30b https://files.mistral-7b-v0-1.mistral.ai/mistral-7B-v0.1.tar use case: wordpress viruses; recognising one https://www.icloud.com/shortcuts/8bcc2979d6e34a3d90a48c3f77f96d68 openai - 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 hugg tok - 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 https://developer.nvidia.com/blog/nvidia-releases-open-source-gpu-kernel-modules/ https://github.com/eosphoros-ai/DB-GPT privateGPT fix: sudo apt-get install python3-dev python3.10 -m pip install --upgrade pip sudo apt install build-essential python3.10 -m pip install hnswlib python3.10 -m pip install chromadb then python3.10 -m pip install chroma-migrate chroma-migrate python3.10 -m pip install -r requirements.txt pip install pycc How to download the models: Visit the Code Llama repository in GitHub and follow the instructions in the README to run the down
Read more

What is a "token" , what is "temperature", and what is "n-shot"?

 TL;DR: token is a meaning unit; temperature is randomness. Token: In the context of AI and natural language processing (NLP), a "token" refers to the smallest unit of text or language that a model can understand. Tokens are usually words or subwords, but they can also be characters or even smaller units, depending on the tokenization method used. Word Tokens: In most cases, a token corresponds to a word in a sentence. For example, the sentence "ChatGPT is helpful" would be tokenized into three word tokens: ["ChatGPT", "is", "helpful"]. Subword Tokens: Some models use subword tokenization methods like Byte-Pair Encoding (BPE) or WordPiece to split words into smaller units. For example, "unhappiness" might be tokenized into ["un", "happiness"]. Character Tokens: In some cases, especially for languages with complex characters or when dealing with very short text, tokens can be individual characters. For ex
Read more