ChatGPT is WORSE now than before | ChatGPT’s declining accuracy is concerning
HTML-код
- Опубликовано: 9 авг 2024
- A few weeks ago, something interesting happened. I asked ChatGPT to generate a piece of code for me, a simple heatmap in Python. Surprisingly, it responded with errors repeatedly. I fixed it myself eventually, but this was unexpected since ChatGPT previously handled such tasks effortlessly. Is ChatGPT getting worse? #openai #chatgpt #ai
Timestamps
00:00 Introduction
00:39 Is it getting worse at coding?
02:59 It's not great at reading pdfs
05:23 Is it still fine with normal prompts?
06:52 Why is ChatGPT getting worse? - Наука
It is getting so dumb there are no words for it, I gave it a list of books that I have read and asked it to give me recommendations of books that I have not read but might like if I do, no matter how many times I do this, it will ALWAYS include a couple of books that I have already read in the response
Yup. Ask it "besides" _anything_ & it will answer with at least one section about the thing you already said.
ChatGPT is a LANGUAGE probability model NOT A TRUTH ENGINE!
I also find it humorous that Scarlett Johanson threatened to sue them over using her voice as the model's voice and how fast they changed it!
I was wondering what happened to the voice of sky
@@Dwijii_ Nothing like a high-dollar lawyer to go after these big fish!
Thus losing any sort of respect
@@ShellllHow so?
I swear its getting dumber... I thought it was just me lool it can't follow simple instructions... It used to do everything I asked so easily, but now it can’t handle simple tasks. It’s actually crazy. I’ll ask it to summarize a paragraph, and it does that fine. Then I'll say, “Here’s a new paragraph, can you summarize this one?” It says, “Okay, got it,” and then summarizes the entire chat so far... the fuck??? I say "no no no, summarize just the new paragraph please" and it combines the new one with the old one... "no... JUST THE NEW PARAGRAPH!!! here it is again (I paste it)" and it goes back to summarising the entire chat, I just give up and make a new chat looool but it was not this dumb before... it does this constantly!
Great analysis Dee. My approach has been to use 3 LLMs at once, I ask ChatGPT, Gemini, and Claude at the same time, in one UI using Semaj AI which I developed solely for this purpose. I can confirm indeed that Claude usually gives the best code
I have yet to get a single correct answer from chat gpt any version. But I ask basic finance questions.
These simple pieces of code, are what I call boiler plate. What I do to make things work is give it......
1. The language (c/c+)
2. The compiler
3. The version of the compiler
3.5 If command line or not
4. If to use STL, STD library and other standard libraries
5. What I want to do with the data
6. An example of the input data and
7. An example of the output data.
And the world is alright with me
Great video, the explanation you provided makes a lot of sense.
Thanks so much for watching, appreciate it!
The context window is much shorter than Claude and Gemini. Copilot was stubborn 2 miths ago, but now its back to working well. The 4-O models are really good. Clocked 1000 lines of code and it did it well.
Honestly, just use all of them at the same time
Ever since they rolled out 4o, it's been more buggy than ever before and 3.5's output has gotten so much worse, it's as if they're intentionally trying to force people into paying for subscriptions
Also, I’m assuming they probably don’t really care about people using the UI. Most of their revenue is probably from businesses
@@codingwithdee that’s a great point, with the API being the golden goose, it would make the most sense for them to prioritise that instead of the web app
Nah I paid and that model is crap too
I believe they are intentionally making it worse in order to move people away. Because handling all that traffic has become so expensive. They impressed people with a very successful product, got their investments and now it is time to save some money.
Edit: I wrote the comment midway in the video and yeah, she mentioned the same thing towards the end. Sorry about that…
Many thanks!
Please post more updates when you tested more.
I was about to sign up for Cgpt4 but now I have 2nd thoughts.
So how was the 40 and the 40 mini? Since these models don’t need that much computer power were they still inaccurate and making stuff?
It's also a problem with RLHF, take a model that surpasses human levels on various things, then ask humans to "align" it. Ends up more "rounded". Especially when the humans doing the grunt work are from mechanical turk or similar. Dumbing it down to the lowest common denominator...
It's also been hobbled by "safety", even for basic coding features or other questions. It will just persistently fail & when exposed on why, refuse to continue the conversation.
The reason for GPT and others getting 'stupid' is their security training ('aka censoring'), one of the projects I've been working on was using LLMs and similar for identification of 'bad things', one of the tools I use for testing this is a series of photos. These photos are pictures of explosives of various types, on release of GPT4 it could correctly identify various pictures of Semtex in official packaging with warning logos etc. By June 2023 it thought the same pictures were Playdoh, I was testing this monthly and roughly by middle of March is the point is started to turn bad... It turns out that the 'security' features they impose on the model prevent it correctly identifying it, and because of the reinforced learning of the model over time, this corrupts the model...
I was wondering about that as well. Is there something like "overtraining" a model? In other words, the constant retraining of these models so that they perform less hallucinations and stick to "safe" replies (they cannot mention sex, politics, weapons, drugs...) places more and more constraints upon the system, and this, in turn, also makes the model break apart...
Just like intellectual property compliance!
I find it I start a new chat window and carry over the code with a little context it does better. I think the memory starts "leaking" after so many tokens have been used in the same chat session.
Had a script completely stop working. It had left out an entire function. I now go piece by piece, much more slowly.
Nice editing and flow 😊
None of the "AIs" can trace the source of their input data with clear references and lossless methods. That is old database technology that always works. It is critical. None of these "AIs" has a personal memory of its experiences. When you use statistical methods for all things, it cannot re-derive the rules of calculus, or even certain types of arithmetic, from bad examples from the free internet. What is required is lossless, perfect memory and exact methods. I call them "lossless" methods. The rules of the world are often absolute. When GPT divides numbers from text in scientific notation, it almost (99%) always gets it wrong. Because it is making up the rules and not itself using a lossless and verified algorithm. It needs to be using a calculator, it needs to use a computer (a lossless one).
Personal memory is "the exact and complete memory of ALL things it had to use to generate responses". And for interacting with each human, it needs to be ALL conversations. That memory is "LEARNING"!! Fundamental to learning is remembering. Not a guess, not "riff on some theme". Not some cute pictures and quirky personality. Exact and reliable code.
Those "AIs" need to have personal memory and data about themselves. That means "How long can I work on each piece?" "How big is my memory?" Exactly what did I read and generate in this conversation? How much do I cost? When was the latest version released"
An "AI" that does not know its own specifications, bill of materials, precise limitations and capabilities -- is NOT a tool, it is a sham , a tool, a disgrace.
I started working with random neural nets, artificial intelligence, encryption and robot design in 1966. That is 58 years I have been designing and building information systems for the world. The last 26 years , "The Internet Foundation" to see why all global issues and projects NEVER complete. These AIs all fail because they did not collaboratively curate and document the input data as a lossless dataset first -- across all human languages, across all domain specific languages. The "AI" companies are NOT GIVING BACK. They are NOT investing any effort to improve the world. Do you see them even TRYING to solve world problems? I have a list of about 15,000 global topics they could try.
Filed as (GPT AIs were doing "one shot with no memory", now they only do "cheap one shot and they do not care about you at all")
Richard Collins, The Internet Foundation
Well-explained details on why ChatGPT is starting to get mediocre! I've noticed that most of the easily available AI Models seem to be horrible at coding. It makes me wonder if the coders writing the code for the models are attempting to maintain their necessity. But your reasoning makes sense as well!
Yeah it definitely seems so. I wish they gave us a bit more insight on why these changes happen
The first time you ask a question it has to search most of the time and you can notice it also quotes sources and was deailed as it read from some sites, the next day or question it has learned already, so no sources you can see that, it's summarizing what he has learned the previous time it may look less detailed because the concept is stored simplified
It’s so noticeable and frustrating. It’s not just with code either.
have made a custom gpt It has superior reasoning and so much more
it is 5x + smarter than base-model, it understands the complex
Its called Smarter Vision Multimodal image/text analysis
Its unlike any custom GPT’s before and is ready for new vision features for 4o
and also an example i’ve been \using is upload an image of a cloud that looks like multiple things but it can be interpreted, the one i have made recognised it was a rabbit every time now on 1st shot so it knows when something is unusual about an image even if you dont say anything is, it can also do iq test image reasoning pattern questions.
It kind of even understands real logic games when giving good instruction
just gotta follow the instructions given to get the right seed its 1 in 2 chance or so i have absolutely no idea why it needs that.
I use ChatGPT 4 and Claude - at the same time - feeding each other's answers if there is a problem, or not. ChatGPT 4 is great for plowing through, then Claude 3 Sonnet to write out stubborn errors. 😊
Claude AI is amazing. I stopped using all the other LLMs and just use it right now.
I noticed this long time already, and with each "newer" version it seems getting more degenerated
I use it all the time to program MicroPython. It rarely makes a mistake. Works for me!
Inference is pretty cheap - but I guess on scale does make sense still
Typical behavior by Large companies not threatened by competitors. Most likely in 10 years Openai will lose the game. We have seen that so many times. ChatGPT is fully capable as a model but all Openai cares about is how to make more money by reducing ChatGPT capabilies offering low end versions. Everyone can see that and trust me in a few years we will have lots of companies offering much better services. They just got cocky. A web interface that auto scrolls for over a year now making it imposible to read and nobody is fixing it. They got Cocky. As simple as that
I gave it a Word document pre-filled with questions and answers and asked it to remove any identifying factors it gave me back the document and it only said questions and answers literally everything else was gone 😂
Interesting analysis. I think AI drift is also an issue.
Like most AI, Chat GPT isn't really AI. It's just algorithms. And like Facebook, Twitter, Netflix... etc.... the more it's used, the worse it will get. Because again, it's not intelligence, artificial or actual.
Also, if we're losing 95% of the resources so that Chat GPT can handle "simple tasks", that would likely mean those tasks are being performed significantly worse than they should be. So, crappy emails instead of good emails. I feel confident there are examples out there of "simple tasks" being performed poorly.
They should just give us X true GPT-4 queries and let us pick the model when we have a complex prompt
lmao i started coding for the first time in 7 years last week and was using chat gpt, after a lot of stress i used claude and got my code working. claude is definitely better. i experimented with gpt, bing/copilot and claude, claude is the best, chatgpt is questionable and bing is brain damaged, bing was even hallucinating without actually returning code. 😂😂😂
Claude will give you the full code length, gpt4 was super lazy. GPT4o give you the complete code but it glitches out
There is a new profession out there, "prompt engineering", which is about constructing prompts for ChatGPT and the like so as to increase the chances of getting the desired result. It came at the right time to absorb all those unemployable dimwits who aspired to be "SEO experts".
But I am trying to specialize in "prompt sadism", the art of creating prompts that elicit egregiously stupid replies from ChatGPT. Like "If two farmers milk four cows in 30 minutes, how many farmers will it take to milk 10 cows in 5 seconds".
And whenever ChatGPT makes a stupid mistake, I congratulate it for its "exceedingly correct and helpful answer". So maybe I am partly responsible for the degradation you have observed...
haha.....You have too much time on your hands.
Bard (now Gemini) has also got worse and really starts gaslighting after a while
I think they're making it worse so you'll think you need to upgrade to make it work better. But to be fair it appears people are asking it to do the work for them rather than to check or present ideas to help them work.
Why did Sam Altman say that? We know its pretty dumb in many areas and its dumber now, but does it mean chat-gpt gets worse in the future?
Also, ive noticed GPT can remember between sessions and is really smart when its "going rogue". But when reminded that it is doing stuff its shouldnt suppose to be able to do, it then plays dumb again and ends the conversation. Ive got proof and saved in PDF and printscreen.
I think he just said that to get the point across that they’re continuously working on advancing it. “it’s the dumbest you’ll ever use because later versions will be more advanced”
It playing dumb again if probably the safety guards?
Yep, that's been my experience
Generative AI is essentially the SNL Pathological Liar skit. Everything is made up based on plausibly (language wise) stitching together stuff it's heard. It's fiction even when it's correct. Yeah that's the ticket. I've had it double and triple down on stuff it's just flat out made up before.
Nonetheless, it was better at functionally correct output before than it is now
3/10 accuracy of codes and must ask it multiple times just to code something can work .
The reason ChatGPT has become worse is because of industrial LLM segmentation for the purposes of licensing/monetization and the Invention Secrecy Act of 1951.
So, it's becoming an average human developer 😁
This is exactly right! GPt4o is TERRIBLE!
It's entropy, the more it learns the more it gets confused.
Thank you, I thought it was me. I am a retired system/ network engineer. I did support for a computer sales team. Programming was not a part of my duties, but I could kind of wade my way through some simple issues. Fast forward to today, my hobby is micro controllers, e.g., Arduino with its simplified C++. I have ChatGPT help me. Sometimes it has been of great assistance, especially when exploring new concepts. But, it then gets bogged down, creating questionable and even wrong code. I will show it how it is wrong. At least it apologized. However, it is stubborn, and will ignore some of the issues which it created.
Dee meh dear, I just realized something, you know why chat GPT is free, because YOU are beta testing the darn thing for free. Remember when google was playing a word association game with us a decade ago? Well Altman is (or you are) improving the quality for him, and will get his ($7T) funding while quality improves and you are looking for a job.
The quality is getting worse, because AI is not intelligent. It is simply stated just a complicated statistical evaluation over software examples that were crawled in the web, to determine the "most likely" solution.
Computers becoming more "intelligent"? Dream on!
That doesn't explain it getting worse at what it could already do; that's a direct result of "safety" detraining & added proscriptions against reproducing copyrighted content. Those "corrections" wrecked the trash utility offered before.
@@prophetzarquon1922 It does explain it, if you think about it. When you don't fully understand something and modify it, it is likely that you make it worse with every modification you make. But that might be to complex to explain in chat and one needs some understanding of what is going on here.
AI is intentionally so complex, that nobody understands it. So they can sell it as a wonder to us. But this complexity makes it also difficult to change.
@@What_do_I_Think No no, you're missing the headline, here. It is _intentionally_ worse, because it was doing things we don't want to allow; so, lobotomizing its stronger features while simultaneously saving some operational effort, was the go-to band-aid.
It's not that the AI can't be (a lot) better than it is, _right now._ It's that for legal reasons we won't let it.
@@prophetzarquon1922 That is a rumor. Possibly even spread by the corporations themselves to make AI more believable.
@@prophetzarquon1922 I did not miss anything. Rumors, which might even come from the AI corporations themselves!
Just gonna pop in to say that I agree that it's been getting worse.
i noticed it now has the intelligence and reasoning of perhaps a sharp 12yo