Yeah I ran into the same limitations with o1. The day it came out I jumped into it and started a rp2350 project and initially it worked but then after adding a few features it started to all fall apart. Thinking we need to change how we approach the problems or like you said maybe the full release of o1 will be much better but I kind of doubt that it will be too different. At least it’s clear it’s def not replacing programmers any time soon.
@@GaryExplains That is what I would use today, but COBOL was created when there were very primitive languages and even those have more in common with what we call frameworks these days. I am just saying that COBOL predates C and that it is funny that this modern COBOL-like language is using something that was released more than a decade later. Also, COBOL is known for being extremely stable, and that is something that later languages are missing - with many of them being known for being very difficult to write effectively and often it’s that difficulty that leads to buggy software.
Well back in the day I knew BASIC, COBOL and a bit of Pascal. So I thought I would ask both GPT3 and Gemini to write a Python program that would read multiple lines of text and check for duplicate lines, display the duplicate lines with line numbers. We got of to a good start, but had some bugs. I asked them to correct the bugs which introduced more bugs. And it all went down hill from there. Conclusion - After the initial program concept the AI could not understand what I was trying to accomplish. At least it was a good Python learning event for me :)
I have seen clips of an AI engineer tweaking the knobs of the prompt to get it to write code. He basically had to formalize how they would communicate to each other. He would send it json files that had key attribute "meanings". Then he would send it a ton of unit tests. The gist I got out of it is that it would have been 10x faster to just write it himself.
@@brendonwood7595 Nice analogy, but it would be more like teaching a machine to ride a horse. The thing is, he was teaching a machine to ride only one horse. All this tweaking was to force the AI to be reproducible and error free, and in so doing basically throwing out the window whatever neat ideas might come out of the interaction. He was basically writing the code anyway, but in a new more complicated trial and error workflow. This was kind of what I was seeing happening here in this video as well.
@@tkenbenYou have clearly never done any serious programming. Thats a fairly typical workflow for a human as well, the human just takes longer for each step already. Any non trivial code written by a human requires significant test cases to ensure that it is reproducible and error free. The fact that many places try to skip this testing is why so much software has so many bugs.
I think this highlights the major limitation of the LLM's at the moment. They simply have a small working memory for concepts they can deal with at once and beyond a certain complexity it falls over. I expect that this will become less and less of a limitation as the models progress.
So pretty much like any existing interpreted language, a mish-mash of others. Still, pretty impressive for code generated code. It's working like a rookie programmer.
Ya when i use an LLM, i try keep the context of what it will need to assist me in coding, to the smallest amount i think it will need to help me. Too little, or too much, and it just starts to degrade.
At 5:07, the line "let count is 1" seems to be unnecessary and not even used. Am I correct? Overall, what a cool idea to get AI to build a programming language. Hmm, SKYNET is now alive.
The interpreter is still written in, and needs, a high level language 'C'. I wonder how it would get on if you asked it to write the interpreter in x64 assembly language, as an example.
I was just thinking that the AI is standing on someone else's shoulders if it's using C to write the interpreter. I was curious if it could write an interpreter from first principles in assembly. I'm fairly sure it couldn't (or at least the setup would be prohibitive) because you'd need to give it far too much information regarding the underlying hardware. I've programmed (no where near professionally) over the years and started with assembly years ago thus my intrigue.
Interesting. I was just wondering about what AI would produce if it was asked to create it's own programming language. I have a minimal programming background. From what I could see, EssanceLang looked similar to what a human programmer might create. Is that true? Did you see any code that was identifiable as not being created by a human? Thanks
I’ve really been enjoying these thought exercise videos Gary. The topics are interesting, the results are informative and the length is just right. It’s a great formula and I like how you have been mixing them in with the very timely content like the pico 2 work and the continuing progression of performance in the Arm space. You are extremely high on my watch right away list when a video posts.
Dunno! The code is in my GitHub repo. If you have a chance please look at the code and the code from Crafting Interpreters and see if you can spot similarities. 👍
I suspect if you had used o1 preview, you would have gotten along much further. I've been consulting with o1-preview on a project, and it's been working out well. In fact, the day I got access I had to write a compiler/vm for pascal. It created a scanner, tokenizer and vm that could compile and run code about as complex as was shown in this video. I did not go further for lack of time (currently working on a project going into production) but was extremely impressed with the thinking, reasoning, and planning it demonstrated with a fairly standard prompt. I went back and forth just a few times until I had working code.
@@GaryExplains I only saw o1-mini in your video. For me, it went pretty well (compiler stuff only the day got access), I've only been using o1-preview (advanced version). Since then, I've been using it to assist with building library code. For example, I was trying out miniz library and needed a single header drop-in replacement for minizip, so was asking to help duplicate the same functionality of code I've created with minizip, mostly implementing seek within a compressed/encrypted archive. A few back and forth and I was able to get what I needed. I find myself using ChatGPT daily for one reason or the other.
@@suki4410 Sadly true, and it explains why so many people think LLMs are actually intelligent, instead of statistical plagiarism automata. They ARE sophisticated, but not actually creative -- the output of this attempt was just a mishmash lifted from existing languages, complete with stilted keyword choice, redundancy, and no actual innovation. The only way it would "come up with" something like call-with-continuation is by having scraped a bunch of scheme or ML examples, but since it relies on the statistics of how many people wrote about a feature instead of how novel it really is, I doubt it would ever notice the actual wow factor. It's like when some researchers made an algebraic proof engine in Lisp that successfully proved all sorts of theorems it hypothesized, but everything it did turned out to be inherent to the lambda calculus upon which Lisp was created. I guess it discovered itself, which is impressive, but not going beyond itself.
If you make the promt closer to a specification which might be fairly large you should get something better. Maybe different stages like ask it for a grammar first then ask for an implementation. I think you need many iterations asking it to fix bugs one after the other. Could you ask it to generate test cases? But the language it created was alot of fun looks like it cherry picked from Python among others!
Well, my meta-analysis of the comments so far; Two out of ten will want to use AI (or buy e.g. OpenAI shares) - and eight of ten don’t really bother too much, scaling from “nah” to “we’re doomed”. I abstain from concluding, let o1 do it…😂
Very interesting! I think your little experiment highlights both some pros and cons of AI-generated code. It gives the ability for what developer Maggie Appleton called "barefoot" developers to create local-first software they wouldn't be able to otherwise, and this is exactly one of the benefits she sees. The caution comes in an over-reliance on such tools, especially by less experienced developers, because then no one understands what the code is actually doing, and that is something we definitely don't want, especially when it comes to any type of critical infrastructure. To be clear, I'm not saying you're advocating for anything like that and you probably share the same caution, just simply making an observation on the current state of AI.
I hate web dev. So I'm telling AI to write the code. In the end I have to read it all anyway to correct it and improve it, but the time I save is worth it.
I am not sure what you mean by that. o1-mini always has a "thinking" step. When I was asking it to fix the bugs it would "think" for more than 1 minute.
@@GaryExplains o1 is Q*/Strawberry, o1 is probably going to be released, but, the main thing for OpenAI is "finishing" Orion (their next frontier model), and they're using o1 to train Orion, to give it quality data. Orion is expected to be launched between November of this year and March of 2025.
So what are your interpretations of the structure of EssenceLang that it created? Why do you think it picked the syntax structures that it did? Seems a bit verbose and not as intuitively simple as it could be and should have been given your initial design parameters. Interested in your thoughts.
The AI is restricted by the knowlege, that already exists. It can only come up with a mix of the old programming languages! Where is the way for innovation? A new programming language, should be able to write complex programs with just a few inputs, like the input (questions) you ask AI. Not to add some numbers, but to have a metha language, to describe what the code should do. eg Draw lines on the screen etc.
"let" appears to be superfluous and "with" in not needed in some statements. It may have been easier to optimise the language before generating an interpreter (and yes, I understand this was just an experiment)
"let" is a common trick that makes the syntactical analysis pass easier. When a statement begins with an identifier, it can be an assignment "first=1" or a function call "first(param)". In this case, you have to resolve the ambiguity later (generally when you read the next token, but not always), and it makes the analyzer more complex. With the "let" keyword, you know immediately that what follows is an assignment, and the analyzer is simpler. In some more advanced cases, like "first[i]", you have already read 4 tokens, and you don't know what kind of statement it is. It can be an assignment "first[i]=0", or a function call "first[i](param)" (because first is an array of functions). With this last example, you can see that "let" makes things much easier.
It was a simple language it came up with, very 'basic' and human syntax like, instead of C or heaven forbid python. Any 'karen' on the street (if not demanding a manager) could grasp 'For n(umber) equals 1 to 6 ' - - Next n(umber) which is why I never understood the takeover with C, especially when nearly every little home computer shipped with 'basic'
IMO the threat is not "AI", which LLMs are definitely not, but the idiot investor class that sees a shiny new object that promises to make them money without wanting humane working conditions.
@@jungervin8765 You do not need AGI. We already know at least one other promising way. But LLMs are not it. They are absolutely not going to make programming languages and programmers irrelevant. LLMs are artificial con artists. They are good at making people believe they can do things they can't actually do.
Challenge most programmers to write a programming language and create an interpreter, in a very short period of time, you're going to get much the same. I'm not worried about people like Rob Pike losing their job; I think it's the 90% of programmers who have to worry a lot more.
Totally wrong !!!! AI should never be having another programming language again. You are riding on horse carrier then real AI future architecture. Don't miss lead others
Did he mean that Ai will replace any need to have a programming language as it will be the creator and the program all in one. Basically no more Operating systems and programmes, just Ai.
It seems to me, we have to understand the problem before we create the tool to fix it. What's the problem? Language is a tool; money is an idea. Get out of the design phase before you buy the parts. Is English the right language to use to create a programming language? I would ask AI to create a recursive lang like Forth, then use that lang to create the lang to write the next lang in...
![The moment we stopped understanding AI [AlexNet]](http://i.ytimg.com/vi/UZDiGooFs54/mqdefault.jpg)
![[DRAGON BALL LEGENDS REVEALS ＆ STUFF] LEGENDS FESTIVAL 2024 Special Edition Part 1](http://i.ytimg.com/vi/mV5bGIswTtQ/mqdefault.jpg)
Yeah I ran into the same limitations with o1. The day it came out I jumped into it and started a rp2350 project and initially it worked but then after adding a few features it started to all fall apart. Thinking we need to change how we approach the problems or like you said maybe the full release of o1 will be much better but I kind of doubt that it will be too different.
At least it’s clear it’s def not replacing programmers any time soon.
'Let X is 10' ?
'EssenceLang' ?
Essex Slang more like!
Speaking words of wisdom. Let it is.
@@andrewdunbar828to is or not to is, that is the question
My immediate thought when the first sample code was shown: "It looks like COBOL". If someone were designing COBOL today.
I was going to say the same exact thing, it is funny to use C - which was released after COBOL - to create a COBOL-like language.
@jfftck What language would you use to create another programming language, COBOL-like or not?
@@GaryExplains That is what I would use today, but COBOL was created when there were very primitive languages and even those have more in common with what we call frameworks these days. I am just saying that COBOL predates C and that it is funny that this modern COBOL-like language is using something that was released more than a decade later. Also, COBOL is known for being extremely stable, and that is something that later languages are missing - with many of them being known for being very difficult to write effectively and often it’s that difficulty that leads to buggy software.
Its wordiness reminded me of Applescript, actually, and not in an appealing way.
Well back in the day I knew BASIC, COBOL and a bit of Pascal. So I thought I would ask both GPT3 and Gemini to write a Python program that would read multiple lines of text and check for duplicate lines, display the duplicate lines with line numbers. We got of to a good start, but had some bugs. I asked them to correct the bugs which introduced more bugs. And it all went down hill from there. Conclusion - After the initial program concept the AI could not understand what I was trying to accomplish. At least it was a good Python learning event for me :)
You needed to prompt in a different way.
i need an exoskeleton with a cerebral interface asap
Elon Musk can do that for you😂 neurolink and optimus.
@@allangraham970 cant wait for the neural ads, since the neural monthly subscription for my legs to work will probably be too expensive
I have seen clips of an AI engineer tweaking the knobs of the prompt to get it to write code. He basically had to formalize how they would communicate to each other. He would send it json files that had key attribute "meanings". Then he would send it a ton of unit tests. The gist I got out of it is that it would have been 10x faster to just write it himself.
It was also quicker to ride a horse than drive in the first cars.
@@brendonwood7595 Nice analogy, but it would be more like teaching a machine to ride a horse. The thing is, he was teaching a machine to ride only one horse. All this tweaking was to force the AI to be reproducible and error free, and in so doing basically throwing out the window whatever neat ideas might come out of the interaction. He was basically writing the code anyway, but in a new more complicated trial and error workflow. This was kind of what I was seeing happening here in this video as well.
@@tkenbenYou have clearly never done any serious programming. Thats a fairly typical workflow for a human as well, the human just takes longer for each step already.
Any non trivial code written by a human requires significant test cases to ensure that it is reproducible and error free.
The fact that many places try to skip this testing is why so much software has so many bugs.
I think this highlights the major limitation of the LLM's at the moment. They simply have a small working memory for concepts they can deal with at once and beyond a certain complexity it falls over. I expect that this will become less and less of a limitation as the models progress.
So pretty much like any existing interpreted language, a mish-mash of others.
Still, pretty impressive for code generated code. It's working like a rookie programmer.
Ya when i use an LLM, i try keep the context of what it will need to assist me in coding, to the smallest amount i think it will need to help me.
Too little, or too much, and it just starts to degrade.
At 5:07, the line "let count is 1" seems to be unnecessary and not even used. Am I correct? Overall, what a cool idea to get AI to build a programming language. Hmm, SKYNET is now alive.
The interpreter is still written in, and needs, a high level language 'C'. I wonder how it would get on if you asked it to write the interpreter in x64 assembly language, as an example.
You mean a compiler rather than an interpreter?
@@GaryExplains, the new language is implement as an interpreter written in assembly language rather than C.
Oh, I see. Why would you do that?
I was just thinking that the AI is standing on someone else's shoulders if it's using C to write the interpreter. I was curious if it could write an interpreter from first principles in assembly. I'm fairly sure it couldn't (or at least the setup would be prohibitive) because you'd need to give it far too much information regarding the underlying hardware. I've programmed (no where near professionally) over the years and started with assembly years ago thus my intrigue.
@@Reprint001 Agreed.
Interesting. I was just wondering about what AI would produce if it was asked to create it's own programming language. I have a minimal programming background. From what I could see, EssanceLang looked similar to what a human programmer might create. Is that true? Did you see any code that was identifiable as not being created by a human? Thanks
I’ve really been enjoying these thought exercise videos Gary. The topics are interesting, the results are informative and the length is just right. It’s a great formula and I like how you have been mixing them in with the very timely content like the pico 2 work and the continuing progression of performance in the Arm space.
You are extremely high on my watch right away list when a video posts.
You are very kind.
This was a very interesting exercise. Thanks for sharing.
Glad you enjoyed it!
I have done similar experiments with chatgpt. Not going as far building a functional interpreter and I took a bit different of an approach.
I wonder how much of the interpreter implementation was taken by o1-mini directly from the book Crafting Interpreters
Dunno! The code is in my GitHub repo. If you have a chance please look at the code and the code from Crafting Interpreters and see if you can spot similarities. 👍
Interesting. I wonder how much time you needed to get the result you published on Github.
About 2 hours
I suspect if you had used o1 preview, you would have gotten along much further. I've been consulting with o1-preview on a project, and it's been working out well. In fact, the day I got access I had to write a compiler/vm for pascal. It created a scanner, tokenizer and vm that could compile and run code about as complex as was shown in this video. I did not go further for lack of time (currently working on a project going into production) but was extremely impressed with the thinking, reasoning, and planning it demonstrated with a fairly standard prompt. I went back and forth just a few times until I had working code.
No, I tried o1-preview as well, obviously.
@@GaryExplains I only saw o1-mini in your video. For me, it went pretty well (compiler stuff only the day got access), I've only been using o1-preview (advanced version). Since then, I've been using it to assist with building library code. For example, I was trying out miniz library and needed a single header drop-in replacement for minizip, so was asking to help duplicate the same functionality of code I've created with minizip, mostly implementing seek within a compressed/encrypted archive. A few back and forth and I was able to get what I needed. I find myself using ChatGPT daily for one reason or the other.
For me "file.el" means file written in Emacs Lisp.
99.8% of the world, dont know what EMACS is.
@@suki4410 Sadly true, and it explains why so many people think LLMs are actually intelligent, instead of statistical plagiarism automata. They ARE sophisticated, but not actually creative -- the output of this attempt was just a mishmash lifted from existing languages, complete with stilted keyword choice, redundancy, and no actual innovation. The only way it would "come up with" something like call-with-continuation is by having scraped a bunch of scheme or ML examples, but since it relies on the statistics of how many people wrote about a feature instead of how novel it really is, I doubt it would ever notice the actual wow factor. It's like when some researchers made an algebraic proof engine in Lisp that successfully proved all sorts of theorems it hypothesized, but everything it did turned out to be inherent to the lambda calculus upon which Lisp was created. I guess it discovered itself, which is impressive, but not going beyond itself.
Interesting experiment!
Seriously awesome
If you make the promt closer to a specification which might be fairly large you should get something better. Maybe different stages like ask it for a grammar first then ask for an implementation. I think you need many iterations asking it to fix bugs one after the other. Could you ask it to generate test cases? But the language it created was alot of fun looks like it cherry picked from Python among others!
And whilst he's at it, just type the whole thing
@@xaxfixho hahaha just killed the AI hype!
Same experience... AIs help with coding to a point, but as you get into specific issues the AI tends to collapse and to no longer be helpful.
It's very interesting, congratulations!
Very interesting how quickly are AI models evolving.
Well, my meta-analysis of the comments so far;
Two out of ten will want to use AI (or buy e.g. OpenAI shares) - and eight of ten don’t really bother too much, scaling from “nah” to “we’re doomed”.
I abstain from concluding, let o1 do it…😂
Very interesting! I think your little experiment highlights both some pros and cons of AI-generated code. It gives the ability for what developer Maggie Appleton called "barefoot" developers to create local-first software they wouldn't be able to otherwise, and this is exactly one of the benefits she sees. The caution comes in an over-reliance on such tools, especially by less experienced developers, because then no one understands what the code is actually doing, and that is something we definitely don't want, especially when it comes to any type of critical infrastructure. To be clear, I'm not saying you're advocating for anything like that and you probably share the same caution, just simply making an observation on the current state of AI.
I hate web dev. So I'm telling AI to write the code.
In the end I have to read it all anyway to correct it and improve it, but the time I save is worth it.
@@ronilevarez901 Have you ever tried a static site generator? Something like Hugo or MkDocs. I much prefer markdown to HTML.
@@ronilevarez901 You might like a static site generator that converts markdown to HTML
Fantastic work! Very important topic. Would love to see more exploration.
You should have let it "think" about the problem first. Maybe it would have realized what it did wrong and fixed it correctly.
I am not sure what you mean by that. o1-mini always has a "thinking" step. When I was asking it to fix the bugs it would "think" for more than 1 minute.
Very interesting!
o1? Orion is a few months away.
I am not sure what you mean. Obviously we have o1-preview now, and we are waiting for the full release of o1.
@@GaryExplains o1 is Q*/Strawberry, o1 is probably going to be released, but, the main thing for OpenAI is "finishing" Orion (their next frontier model), and they're using o1 to train Orion, to give it quality data. Orion is expected to be launched between November of this year and March of 2025.
And you know this from your secret contact in OpenAI?
@@GaryExplains Only from open-source intelligence (OSINT).
@ThePowerLover You mean unsubstantiated rumors. Spreading rumors isn't nice.
Skynet is going live real soon
So what are your interpretations of the structure of EssenceLang that it created? Why do you think it picked the syntax structures that it did? Seems a bit verbose and not as intuitively simple as it could be and should have been given your initial design parameters. Interested in your thoughts.
The AI is restricted by the knowlege, that already exists. It can only come up with a mix of the old programming languages! Where is the way for innovation? A new programming language, should be able to write complex programs with just a few inputs, like the input (questions) you ask AI.
Not to add some numbers, but to have a metha language, to describe what the code should do. eg Draw lines on the screen etc.
It invented cobol!
👍
Programs, programing programs? I think I saw the movie, didn't end well
Programs that create or modify programs have been around for more than 60 years. Nothing new. Nothing to be afraid of.
Breakdowning it might work. Another video of how it goes after you've breakdowned it.
"let" appears to be superfluous and "with" in not needed in some statements. It may have been easier to optimise the language before generating an interpreter (and yes, I understand this was just an experiment)
"let" is a common trick that makes the syntactical analysis pass easier. When a statement begins with an identifier, it can be an assignment "first=1" or a function call "first(param)". In this case, you have to resolve the ambiguity later (generally when you read the next token, but not always), and it makes the analyzer more complex. With the "let" keyword, you know immediately that what follows is an assignment, and the analyzer is simpler. In some more advanced cases, like "first[i]", you have already read 4 tokens, and you don't know what kind of statement it is. It can be an assignment "first[i]=0", or a function call "first[i](param)" (because first is an array of functions). With this last example, you can see that "let" makes things much easier.
That's actually quite horrible.
The language, the implementation, or the general idea?
@@GaryExplains A bit of all that. ;)
The most interesting result to come out of this, I think, was that repetition reduces the resultant quality...
It was a simple language it came up with, very 'basic' and human syntax like, instead of C or heaven forbid python.
Any 'karen' on the street (if not demanding a manager) could grasp 'For n(umber) equals 1 to 6 ' - - Next n(umber) which is why I never understood the takeover with C, especially when nearly every little home computer shipped with 'basic'
Seems like AI is will replace all programmers soon... =(
IMO the threat is not "AI", which LLMs are definitely not, but the idiot investor class that sees a shiny new object that promises to make them money without wanting humane working conditions.
LLMs are artificial con artists, and you have just been fooled into thinking they can create a programming language.
The irony here is that AI is making the creation of new programming languages irrelevant.
Interesting observation.
Or maybe not. This deal is not done yet. Certainly not in the current state.
Thats a dumb take. You need an artifical general intelligence to make programming languages irrevelent at the minimum.
@@jungervin8765 You do not need AGI. We already know at least one other promising way. But LLMs are not it. They are absolutely not going to make programming languages and programmers irrelevant. LLMs are artificial con artists. They are good at making people believe they can do things they can't actually do.
I smell Visual Basic
Best language ever.
I don't care what anyone says.
Looks awful 😂
Well, at least it shows that LLMs aren't going to take over any time soon. It's just a mishmash rather than anything novel.
Challenge most programmers to write a programming language and create an interpreter, in a very short period of time, you're going to get much the same. I'm not worried about people like Rob Pike losing their job; I think it's the 90% of programmers who have to worry a lot more.
@@prosfilaes More like way worse if the time frame is similar.
yea very impressive, you are able to query an AI model to create this for you, holy shit its almost like we couldnt do this ourselves.
Totally wrong !!!! AI should never be having another programming language again. You are riding on horse carrier then real AI future architecture. Don't miss lead others
Eh?
My thoughts exactly: What?
Did he mean that Ai will replace any need to have a programming language as it will be the creator and the program all in one. Basically no more Operating systems and programmes, just Ai.
It seems to me, we have to understand the problem before we create the tool to fix it. What's the problem?
Language is a tool; money is an idea.
Get out of the design phase before you buy the parts.
Is English the right language to use to create a programming language?
I would ask AI to create a recursive lang like Forth, then use that lang to create the lang to write the next lang in...
WOW Gary I been first for you and my other favorite tech channel called @whatphoto I'm on a role today lol ❤❤❤