Видео

That's the nice thing about learning coding - you learn a ton of other skills along the way (as well as how a bunch of these systems actually work)

Heck yeah! That is awesome - keep going!

Yeah - we will still have our own thoughts/feelings and needs so will need to solve our own problems, whether using AI or some other way.

Could you elaborate? :)

You got it! I think the big thing we need to start differentiating is raw AI models that can do simple input/output and matrix multiplication in the middle VS AI Agents which are really just programs that make use of AI models as part of their workflow. AI Agents is what poses the vast majority of the risk IMO.

Yes, we should strive to not forget these skills for the sake of keeping our dignity and purpose - even if they become "unnecessary" from an economic perspective

I think this type of benchmark will become more and more important. Neural networks and LLM are trained with "infinite" test sets. The energy required to form the models will become ever greater - this benchmark shows how the training amount could be reduced or how inflexible the models are. I think it will be particularly crucial for small LLM on the edge (IOT, smartphones).

🤩 awesome to hear from you Bryan! You all have done an amazing job with this prize! Hey, if you're up for it - would love it if you could share the video on X - could be a good way to introduce more people to the prize and keep engagement going!

You win this award: 🥇 Well done!

The cool thing is that now individual programmers can actually compete with entire companies by building their own stuff much faster! At big companies, the coordination overhead and bureaucracy will keep them slow despite the advances in tech. I think we will see a big shift toward smaller coding agencies

Great point!

Yep! So many things are possible now because of the accelerated pace of development

Best bet to pay the bills is to take advantage of the AI wave and make money with it while the opportunity is there

Absolutely - more and more people will become 'generalists' capable of doing many things - and I think current coders are well equipped to manage/control the AI because they can structure their thoughts well and create clear requirements for the AI

:)) Thank you so much!

Nothing is completely certain, but some things appear safer than others. In fact, some areas may even see a boost thanks to AI. I compare various roles and the AI risk in this video: ruclips.net/video/4KSs29EPd8M/видео.html

Haha this is so true but drives me crazy. I don't understand why companies are still prioritizing formalities over actual ability. Would love to see more "work trials" where people just get to do the job for like a month and then if it works for everyone, they stay.

The funny thing is, even we will never 100% understand our own needs 😂

What I'm referring to is the ability to solve long arithmetic problems. One of his grad students had the idea to do it recursively like the way humans learn, and he threw it away because it wasn't reliable Or fast as calculators or whatever.

Just analyze what's going on. Zoom. Why did you know what it was? The rectangle looked important. You cut the example inputs up and started matching against the output examples. It's an extremely incremental, recursive process that you just can't see in a single pass-through. But if you give it a series of transformations that it can perform, and then let it recursively apply them until it figures it out, it should be able to use the examples the way humans do and find the transformations, and then it's a matter of knowing what transformations are possible. I think these examples draw on human-centric perceptions. Alignment. Zooming. All of these things relate to how humans see the world. Taking things apart. We have hands, and we have done it millions of times. That's the only reason we try it out as a potential transformation. It just comes to our minds. When we see blocks, we see things to pick up and move.

So if I were trying to solve this problem, I would set up a recursive neural network and train it to be able to treat the pixels as objects to manipulate. The output should be a transformation, not a full set of pixels. Reality renders the result of our actions. The thing that has to understand is that objects are rigid though. Or that they can merge. Or whatever. But that expectation in humans is the result of actual real-world experience. These puzzles rely on implicit understandings of how physics works.

What this means is that if there is a certain law of physics that is accounted for in the puzzles that aren't made available for training on but are in the actual test itself, it will be impossible to pass it without full human intuition about how the world works. So it's going to take basically a humanoid robot to be able to know how to solve these things.

AI researchers should listen to Jordan Peterson or learn how to think on their own.

It points out that LLMs are not a path to AGI, as some lunatics think they are

What I love about this challenge is that the data structure is actually sooo simple. Computer vision isn't "looking" at anything like we are, it's analyzing huge arrays of numbers that represent the colors of each pixel. So you can think of this puzzle as a super simplified image. When we use GPT vision up till now, if used a vision model to understand the contents and then pass that to GPT-4. Now with GPT-4o if should be native and pass it in as a compressed version of the image (you can read more on their blogs) but I've only gotten very poor results from using it unfortunately. Still interesting to see that a fine tuned LLM is the current SOTA for Arc.

​@@VoltLover00 speaking this number of languages and not to mention programming languages pretty fluently sounds pretty general to me. A shit ton of human problems get solved by thinking in a language as well. I think human eyes are pretty much like an API as well, considering how we only have a narrow focus point where we can actually see properly.

I guess you don't understand how LLMs work? You can't input an image as a prompt to an LLM

Given your confidence, you should create a solution and claim the prize :) but I assure you, plenty of smart folks have tried all sorts of LLM prompting tricks and vision models for this benchmark and none have worked well at all so far. That's what I find so interesting about it!

Additionally, consider that vision models don't "see" things - they accept huge arrays of numbers representing the colors of each pixel. In that sense, this puzzle's data should be x100 easier to interpret.

It's not a problem with the inputs. You can stick these in as json and the LLM will happily memorize rules. It'll even figure out how to apply the rules it's memorized to slight variations of the problem.

go for the prize!

Which problems? Like half of nlp papers are about prompts or model evals

You have no reason to make such predictions

This has been tried, achieves less than 20% on the public set which is very bad

Would love to see a vision-based approach!

These tasks are not primarily visual -- they can easily be represented via json, or as flattened sequences. The LLMs have no problem memorizing arbitrary manipulations to those sequences, showing that they don't actually have any problem with the input data-type. You don't need CNNs for the network to figure out how to memorize the training set. What they can't do is come up with new, simple combinations of rules that they haven't seen before. The problem isn't that they can't see, it's that they can't think.

You can represent the grids 2d matrix of numbers that denote the colors, you don't need cnn at all.

Yeah that's the worry - Francois expressed this concern on Dwarkesh as well - that someone might create a successful but unsatisfactory solution of just synthesizing a ton of arc-like training data and solving it effectively through memorization. I hope some new ideas come about and take a different approach

You're 100% wrong. ARC is only a challenge because you're not allowed to use LLMs. You have to use a small, local program. GPT-4o is already at 50% accuracy without any training data. Just explain the concept in a prompt, convert the images to text, and it can not only solve them, it can invent computer programs to solve them. And it's the worst it's ever going to be. Claude 3.5 just surpassed GPT4-o's performance this week. Two years ago, it had 0% accuracy. Where will it be next year? Two years from now? Three? Last week, a user on Twitter / X converted these challenges into text-based-squares, built a prompt (less than 32k tokens), and had GPT-4o write python programs to solve them, then submitted those programs to the ARC challenge. GPT-4o's work scored over 50% accuracy, which is higher than MindsAI's 39% currently topping the ARCPrize leaderboard.

@@jonmichaelgalindo Still, the fact that I can sit with my coffee never having seen these puzzles before, and leisurely solve all the ones I have tried so far with relative ease, and yet it takes AI an enormous amount of energy and "compute" just to hit 50% accuracy shows that AGI is still elusive. That said, my mind is the product of 4 billion years of evolution whereas these Chatbots are just getting started. I expect that AGI will be reached within 10-20 years even though we aren't exactly sure what it will take to get there. After all, who predicted 5 years ago that AI would be where it is today in terms of being able to pass medical exams etc...

@@tommiest3769 I'm basically incompetent at these puzzles. Way lower than average. :-( And I'm not stupid. I play several instruments. I'm a lot better at coding than GPT-4. I've self published novels. I enjoy philosophy. I could go on. But these stupid squares never make sense. I get it right after someone tells me the trick and then it seems super obvious, but there's just something not quite right in my head.

Isn't the best way to test whether a system is an AGI to place it in a completely novel environment and ask it to figure out a puzzle for which it has no experience whatsoever? So in some ways, we might need embodiment before this can happen. An example might be an escape room or placing it out in the middle of a deep woods and seeing if it can figure out how to get from point A to point B (e.g. orienteering). Another test for AGI/ASI would be to set it loose on one of The Millennium Prize Problems" in mathematics.

Thanks for watching! I hope we will see a non-memorizarion based solution for Arc!

The model I used was GPT-4o but admittedly I only did text input, not visual. I believe others have tried visual based approaches and had similar results. Will be interesting to see if someone can create an effective solution on the public leaderboard using this approach!