Building my own LLM

Yesterday I started building my own LLM because I want to avoid licensing issues for basic chatbot usages. It's super slow training these things. I won't post code here because I plan to commercialise it, but here's the progress screenshot:



I've set it to use the following parameters. Sadly my GPU is rather small so it is taking ages.

# Training hyperparameters
TEMPERATURE=0.7
TOP_K=40
TOP_P=0.95
REPEAT_PENALTY=1.1
MAX_NEW_TOKENS=200

BLOCK_SIZE=32
BATCH_SIZE=8
NUM_EPOCHS=5
LEARNING_RATE=0.001
CHECKPOINT_INTERVAL=1
USE_GPU=true

# File and directory settings
MODEL_DIR=models/
CORPUS_DIR=repo/
LOG_DIR=logs/
BACKUP_DIR=bak/
TOKENIZER_PATH=

# Internal settings
EVAL_INTERVAL=100
EVAL_ITERS=100
SEED=1337
INPUT_FILE=input.txt



Popular posts from this blog

Recent Experiments testing ChatGPT's limits and some useful prompts

Introduction ChatGPT has a lot of people excited, but it deserves this level of excitement. It has a number of faults which I list below - some with workarounds - but in general it is amazingly good. It is absolutely not for having fun chatting to a robot, it's really more like a brilliant coworker who is incredibly stupid at some quite mundane tasks, but incredibly good at more tricky tasks. Summary of findings so far.  1. API vs Interactive Version - a caution 2. Coding Strengths and Weaknesses - be very careful but very useful anyway 3. Shell Scripting - amazingly good, probably what it's best at 4. Text Summarisation - fiddly but useful 5. Spreadsheet Analysis - annoying, maybe do by hand 6. Multi-Step Instruction Handling - very annoying 7. Spellchecking and writing style/grammar - ok overall but fiddly 8. File Compatibility and Formatting - not great 9. APA References from DOIs - good 10. Reference Accuracy - bad, double-check its work 11. Tone and Style Adaptations - ex...
Read more

Testing ChatGPT on coding - a script to make CSV to ICS

 So I gave ChatGPT a task of writing a CSV to ICS convertor, to take a spreadsheet of events and turn it into individual calendar file invitations. The idea is you can make a calendar with lots of event items in a spreadsheet, and then output ICS that you can just double-click to add to your calendar automatically. So here's the spreadsheet: "Event Name","Event Start Date","Event End Date","Event Start Time","Event End Time","event description or notes","URL applicable" "Meeting with John","2024-06-10","2024-06-10","09:00","10:00","Discuss project updates.","http://example.com" And here's the script ChatGPT made. It seems to get confused with things like Daylight Savings and/or Timezone but you can fix that manually. It took some debugging and I had to copy/paste my Mac's manpage for DATE into ChatGPT, but it got it eventually. The only...
Read more

Deepseek

 TLDR: faster than chatgpt but slightly more buggy and the server is apparently overloaded. It didn't quite do as well on the examination/test question marking as ChatGPT, it inferred marks where there were not meant to be any and counted past the limit of 6, giving full marks when I clearly made minor errors that should have cost me 1-2 marks. Transcript follows. Hi. The purpose of this exercise is to test your capabilities. Please write an algorithm in php which counts from 1 to 10 and prints the result to stdout. Certainly! Below is a simple PHP algorithm that counts from 1 to 10 and prints the result to   stdout : php Copy <?php for ( $i = 1 ; $i <= 10 ; $i ++ ) { echo $i . PHP_EOL ; } ?> Explanation: The   for   loop initializes a variable   $i   to 1. The loop continues as long as   $i   is less than or equal to 10. In each iteration, the value of   $i   is printed using   echo . PHP_EOL   is a...
Read more