Update on what ChatGPT is bad at doing.

OMG ChatGPT o1 is terrible. And 4.5 is not great. I keep going back to 4.o. At least it knows how to code.

I've experimented now for about a year with giving ChatGPT tasks. The below criticisms apply to ChatGPT 4o. I recently got access to ChatGPT 4.5 which is better at coding and following instructions (it at least takes the instruction to give steps one at a time, seriously, whereas ChatGPT 4o almost always ignores that). Anyway, this is what 4o is bad at:

  • Making, interpreting, processing, editing spreadsheets. You have to convert the spreadsheet to CSV first (you lose all formulas and charts). It is ok at interpreting small data sets, e.g. 20-30 bits of simple information e.g. amount of money spent per month, with each month listed. It's no good at, for example, creating a Nett Present Value spreadsheet/analysis.
  • Editing long documents or just searching documents for specific errors. Even basic requests like checking for double spaces do not work. In one case it saw a poem and thought that it was grammatically incorrect and had incorrect capitalisation because it did not recognise that it was a poem. It is OK at suggesting alternative wording but seems to prefer active voice American-style writing. It gets confused about academic passive-voice style and wants to change it.
  • Generating table grids with correct contents. I tried asking it to create a Word file with a bingo sheet. Nope. It generates a generic table (in Word) and some of the bingo sheets have duplicate clues (they're supposed to be random and unique). Even though I told it the squares must be square, it ignored that. Only way to achieve this was to ask it to do it in HTML and embed a javascript randomiser. That worked well. You can see the result at https://www.ostrowick.co.za/bingo/. 
  • Interpreting PDFs. It really cannot do it properly, in the end. You have to convert the PDF to plain text and clean it up first by making chapter breaks clear with say, underscores, and deleting running headers and footers and page numbers. Even then it doesnt like long documents. I tried giving it a book to summarise in this manner. Nope, I had to do it chapter by chapter. It doesn't understand pagination or chapter headings. If you ask it for page numbers of errors it struggles to give accurate answers. If there's a table or graphic in the PDF, good luck with that.
  • Code line numbers. Even if you give it a flat text file with code in it, it cannot tell you exactly what line number a bug is in. It tells you within about 10 or so lines, e.g. if the bug is in like 63 it might tell you the bug is in line 77. I suspect it is expecting UNIX linefeeds/carriage returns and gets confused by DOS ASCII. I'll have to test this theory.
  • Generating images. It passes images to Dall-e which is a pretty lousy image generator. It does not generate vectors. Forget vector diagrams with labels, such as flowcharts and corporate graphics for powerpoints. It is useless at that. It is also heavily censored and won't generate images of politicians. I asked it to generate a satirical cartoon and it flat refused. Pretty sure satire is fair use.
  • Generating powerpoints. It can generate powerpoints but they are very low quality copy/paste jobs with bullets mindlessly divided between pages/slides for no strong reason. No branding/background graphics etc., not even an attempt. No contents page, etc. 
  • Advanced Mathematics. It still makes mathematical errors. My test is to ask for the rotation by volume of a sine wave of 180 degrees. It does not generally give the right answer, it messes around for ages and you have to argue with it. 
  • Following instructions - specifically the instruction to give steps one at a time. I have to tell it approximately every half hour to give me step-by-step instructions for debugging code or installing complex software. It generally runs ahead and gives 3-20 steps without a pause. The trouble is, it assumes it will all work as planned, announcing at the end of the instructions, "WHY THIS WORKS". And so if you want to report a failure at say, step 2 out of 20, you have to scroll up screeds of instructions to find step 2 and click on it to reply and say "hey no it failed at this step with such and such error". I also find it quite arrogant with the way it says "why this works" when it gives an IT solution. It almost never works. I often have to go back to it and reprimand it for arrogance.

Popular posts from this blog

Recent Experiments testing ChatGPT's limits and some useful prompts

Introduction ChatGPT has a lot of people excited, but it deserves this level of excitement. It has a number of faults which I list below - some with workarounds - but in general it is amazingly good. It is absolutely not for having fun chatting to a robot, it's really more like a brilliant coworker who is incredibly stupid at some quite mundane tasks, but incredibly good at more tricky tasks. Summary of findings so far.  1. API vs Interactive Version - a caution 2. Coding Strengths and Weaknesses - be very careful but very useful anyway 3. Shell Scripting - amazingly good, probably what it's best at 4. Text Summarisation - fiddly but useful 5. Spreadsheet Analysis - annoying, maybe do by hand 6. Multi-Step Instruction Handling - very annoying 7. Spellchecking and writing style/grammar - ok overall but fiddly 8. File Compatibility and Formatting - not great 9. APA References from DOIs - good 10. Reference Accuracy - bad, double-check its work 11. Tone and Style Adaptations - ex...
Read more

Testing ChatGPT on coding - a script to make CSV to ICS

 So I gave ChatGPT a task of writing a CSV to ICS convertor, to take a spreadsheet of events and turn it into individual calendar file invitations. The idea is you can make a calendar with lots of event items in a spreadsheet, and then output ICS that you can just double-click to add to your calendar automatically. So here's the spreadsheet: "Event Name","Event Start Date","Event End Date","Event Start Time","Event End Time","event description or notes","URL applicable" "Meeting with John","2024-06-10","2024-06-10","09:00","10:00","Discuss project updates.","http://example.com" And here's the script ChatGPT made. It seems to get confused with things like Daylight Savings and/or Timezone but you can fix that manually. It took some debugging and I had to copy/paste my Mac's manpage for DATE into ChatGPT, but it got it eventually. The only...
Read more

Deepseek

 TLDR: faster than chatgpt but slightly more buggy and the server is apparently overloaded. It didn't quite do as well on the examination/test question marking as ChatGPT, it inferred marks where there were not meant to be any and counted past the limit of 6, giving full marks when I clearly made minor errors that should have cost me 1-2 marks. Transcript follows. Hi. The purpose of this exercise is to test your capabilities. Please write an algorithm in php which counts from 1 to 10 and prints the result to stdout. Certainly! Below is a simple PHP algorithm that counts from 1 to 10 and prints the result to   stdout : php Copy <?php for ( $i = 1 ; $i <= 10 ; $i ++ ) { echo $i . PHP_EOL ; } ?> Explanation: The   for   loop initializes a variable   $i   to 1. The loop continues as long as   $i   is less than or equal to 10. In each iteration, the value of   $i   is printed using   echo . PHP_EOL   is a...
Read more