Видео

Oh it give me an 💡 idea. It sounds to me like when I watch my nephew learning to walk, my wife comes and provies him with a walker (hyper parameters space). What doesn't make sense to me is that, are we trying to build the equivalent of a kid, the walker, or wifu... Or a god forbidden amalgamation of the three?

Your approach is good, i think the Gpt 4'o will perform well on the training phase. Did you notice that for the testing part, the shapes of Output can significantly vary How do you think you can guess the right answer , if you have only 2 attempts, for example, based on shape of the input (3,3) several solution are possible as output shape (3,3) (1, 2)(5, 5)(9, 9)(3, 1)(1, 1)(6, 6)(1, 4)(12, 12)(18, 18)(3, 6)(5, 3)(6, 3)(1, 3)(15, 15). Notice also the same key can lead to diffrent output. Thanks for your return

I get that this is a stepping stone, but calling it a test for AGI is just ludicrous. This isn't even close to AGI, it's just a toy.

Excellent talk. Chollet deserves a lot of credit for developing a benchmark before the GPT craze that is so simple and yet for which GPT scaling has only led to modest improvements on. Even if it does fall soon to ad-hoc approaches, lasting as long as it has is no small feat.

Somebody from space is listening, "Oh, poor humans, They still refer to abstraction as a blurred image, hahahahahahah." Abstractions have their own use, but they could be a large source of limitations if we try to imitate that from humans. Abstractions could be human limitations in trying to understand matters. Unless we refer to the AI model as the abstraction layer. I don't know how relevant that could be.

Throwing a million dollars into an AI puzzle fuels substantial momentum toward AI research. And that is awesome. Thank you for differentiating between the output of the process and the process itself. Unfortunately, trending concepts become marketing slogans, such as Samsung Galaxy AI :).

Do not we have enough A.I. to brute force the model we are needing?

Going through the arc training tests, it's as I feared, many many of the problems are based on cultural conventions of human layout of information, human pattern design, or well known cultural patterns, or comp sci cultural knowledge, not to mention the bias in our visual system. There is no way for even a hyper 'intelligent' tabula rasa system to answer these problems correctly, as the answers are not even 'correct'. The winner will likely be the team that hardcodes in all the cultural visual baggage, where the 'answer' may not be strongest 'pattern' by any measure. A hyper intelligent system wouldn't ever score well on ARC, but a shitty human-overtuned system, one implemented probably with CLIP or something, that is actually a relatively dumb search over an exhaustive corpus of hand-coded patterns built to match the creator's ideas for input/output pairs. The creators of this test think they're smart because the search space is so large, but it isn't, they haven't realized in their hubris the number of different types of 'patterns' they're likely to think of, coming out of a few brains, that are actually good and 'fair' questions that require reasoning, is relatively small, and the kaggle competitors will think of most of them, submitting boring entries over and over again, slowly matching the answers in the problem set as they add ideas. (eg. I bet there's a grip of problems like 'game of life' cell automata, or battleship!, or fibonacci tee hee! all that type of crap, in the private test set, that sort of thing, and I bet the private answers are 'not even wrong', in that they're not the simplest or strongest pattern by any measure, other than by human comp-sci nerd culture)

I took a look at the ARC Prize... it... it doesn't make sense? A very smart human scores 22% on ARC... the top AI score is 46% on the leaderboard ALREADY? so... what are you trying to measure here? who is best at 'guess the algorithm I'm thinking of'?

''You are confusing the output of the process with the process itself'' Very nice!

we are not providing the same tools back to the core architecture. such as realtime inference

Chapters (Powered by @danecjensen) - 00:00 - Intelligence, benchmarks, and AI hype in 2023 03:26 - LLMs autoregressive responses 07:36 - LLMs inability to solve CSR ciphers 10:14 - ML models rely heavily on human labor 11:41 - Minskystyle AI vs Makatsstyle AI 15:25 - Intelligence spectrum skilled, operational, efficient 16:42 - AI models should not be evaluated using human exams 17:29 - AIs next level of capabilities and efficiency 18:53 - RKGI AI benchmark for human intelligence 20:49 - Data sets Kaggle, RKGI, ARK 21:45 - Zach offers 1m RKGI solution competition 24:58 - Physicists describe intelligence and abstractions 29:09 - AIs ability to master tasks efficiently 30:50 - Two types of abstraction valuecentric and programcentric 33:48 - Decentralized program search forAGI 34:27 - Program synthesis PS vs machine learning 35:30 - Program synthesis overcomes combinatorial explosion, LMS limitations 36:46 - AI combines chess and discrete search techniques 41:17 - Deep learning components in discrete programs 41:56 - Deep learning for ArcGi program synthesis 43:19 - Program embedding for efficient search 43:45 - Python ArcGGI pipeline improvement 44:55 - LLMs fall short of GI, need breakthroughs 47:50 - Breakthrough in ArcGGI likely to come from outsider, not big labs 51:09 - Experiential learning and causality in childrens learning 52:16 - Humans are capable of few short program synthesis 53:32 - Human cognition works on fundamental level

I love that the ARC AGI prize is $1mil, where the if in fact you can build something which can scale to AGI. The prize money for a business would by in the the billions...

And they don't even drive cars well.

16:00 "If you know hot to drive only in very specific geo fence adreas - thats intelligence". Isnt this a SKILL how you defined that? An INTELLIGENCE is ability to drive anywhere (left, right, town, country ...), and not just at my own 3x3 streets box? (which would be learned skill, where you already tried every possibility)

“See you on the leaderboard for ARC-AGI” is quite a way to end it

Excellent reality check for AI. I bet this video won't have 1 million views like the hype videos.

There's IQ tests for that :D . This also goes to show how flawed IQ tests are. Especially when the children of those who wrote the tests seem to score the highest. (True story)

20:20 Should we also measure agents with less / more shots? How many shots needed to get an agent to a score of 90?

Hey, that's the back of my head I see at the bottom right! Good presentation at a fun conference.

One definition of intelligence is the ability to solve problems in complex situations. There are many problems from the local to a global level that are not being solved well with the current level of intelligence. Our highest priority is creating collective human and digital intelligence. Language models in agentic workflows can extract entities and relationships from text, and merge that knowledge into graph representations. Keeping the human in the loop is important to catch and correct mistakes made by the speech recognition and language models. People can select parts of conversations they have with digital agents to be merged into a global shared graph representation. A global platform can be built pretty much with today's technology that can merge selected parts of millions of simultaneous conversations into a shared world model by the end of this year.

Finally! An AGI-like task

This is really good!

It is not entirely clear that we humans perform some computational magic that is beyond merely relying on pre-learnt patterns. And perhaps it just 'feels like' we reach beyond the training set when we have those eureka moments, or imagine that we just came up with a solution to a novel problem? perhaps in fact we have such a tightly woven mesh of prior training examples that those moments when we arrive at 'eureka' are in fact just inevitable and only appear novel to us because we are unconscious of the combined results of all our prior training?

"Next breakthrough will come from an outsider" What breakthrough dude? Have you noticed that AGI = brain algorithms and that my brain algorithms are here since 2016? Nothing has changed since. It's all working. We are dealing with human idiots that are screwing humanity's future. How can idiots that don't see my 2016 invention save humanity? You are not going to save us, you are wasting the time humans have LEFT on this planet. You have already wasted 8 long years!! No one on this planet can return us 8 wasted years. This is not meant only on the speaker, this is meant generally on the whole world.

# AI Progress and Generalization: A Critical Review ## Chapter 1: The AI Hype of Early 2023 **Timestamp: **0:00** - **1:57**** - Overview of the peak AGI hype in early 2023. - ChatGPT, GPT-4, and Bing Chat were perceived as revolutionary, with claims that AI would drastically increase productivity and replace many jobs. - Despite the hype, the actual impact on employment has been negligible, and fears of mass unemployment were unfounded. ## Chapter 2: Limitations of Large Language Models (LLMs) **Timestamp: **1:58** - **3:38**** - LLMs, including ChatGPT, have inherent limitations such as failing to understand context and falling into pattern matching rather than true comprehension. - These limitations are tied to the fundamental architecture and approach of current AI models, showing little progress over time. ## Chapter 3: Problems with Task Familiarity and Generalization **Timestamp: **3:39** - **6:31**** - LLMs struggle with unfamiliar tasks, performing well only on tasks they have memorized. - Performance issues arise from the extreme sensitivity to phrasing and the inability to generalize from known tasks to new, similar ones. ## Chapter 4: The Misconception of AI Intelligence **Timestamp: **6:32** - **13:51**** - Intelligence should not be equated with task-specific skill; true intelligence involves the ability to handle novel situations. - The speaker argues for a shift from task-based AI evaluations to measuring generalization and adaptability. ## Chapter 5: Redefining Intelligence and Measuring Progress **Timestamp: **13:52** - **19:15**** - Intelligence should be viewed as the ability to synthesize new solutions and adapt to new situations. - The current benchmarks, based on human exams, are inadequate for assessing AI’s true generalization capabilities. ## Chapter 6: The Abstraction Reasoning Corpus (ARC) and Generalization Benchmarking **Timestamp: **19:16** - **24:22**** - Introduction of ARC, a dataset designed to measure an AI's ability to generalize and perform novel tasks. - ARC aims to control for prior knowledge and experience, emphasizing the need for AI to infer solutions rather than relying on memorization. ## Chapter 7: The Role of Abstraction in AI and Human Intelligence **Timestamp: **24:23** - **30:36**** - Abstraction is the key to generalization; intelligence depends on the ability to recognize and apply abstract patterns. - LLMs are currently limited to low-level abstraction and lack the capability to synthesize new models on the fly. ## Chapter 8: Integrating Type 1 and Type 2 Thinking for AGI **Timestamp: **30:37** - **37:41**** - The next step in AI development involves combining Type 1 (intuition, pattern recognition) and Type 2 (logical reasoning) thinking. - Human intelligence excels because it merges these two forms of thinking, and AI needs to follow a similar path. ## Chapter 9: Combining Deep Learning with Program Synthesis **Timestamp: **37:42** - **42:48**** - Future AI advancements will likely involve merging deep learning (Type 1) with program synthesis (Type 2) to handle complex, novel tasks. - This approach could significantly improve AI’s problem-solving capabilities and generalization. ## Chapter 10: Practical Applications and the Future of AI Development **Timestamp: **42:49** - **45:12**** - Practical strategies for improving AI, such as using LLMs to generate and refine programs, show promise in advancing AI generalization. - The importance of innovative thinking and diverse approaches in overcoming current AI limitations. ## Chapter 11: The Need for New Breakthroughs and Intellectual Diversity **Timestamp: **45:13** - **48:36**** - The speaker emphasizes that progress towards AGI has stalled due to a lack of new ideas and intellectual diversity. - The speaker suggests that the next breakthroughs are likely to come from outsiders rather than big tech labs. ## Chapter 12: Future Directions and Closing Thoughts **Timestamp: **48:37** - **53:34**** - The development of AI tests and challenges like ARC 2 is discussed, aiming for more sophisticated and dynamic assessments. - Insights from observing human cognitive development, particularly in children, could inform AI research and the creation of more generalizable AI systems.

I've been a big fan of bongard problems and the Arc test tests similar problem space as bongard problems do. I do think that approaches like Greenblatt's dynamic generation of solutions will push the state of the art forward. I also think that the kind of encoding data the current models use and are trained on isn't fully conducive for this task, and more fine tuned models might help. Also, I think that Arc puzzles should be abstracted further out to three dimensions and more complicated transformations to encode patterns and transformations that even 150+ iq test takers have difficulty recognizing in <30 minutes. Of course constructing that dataset is difficult work. Also, it would be interesting if someone took the arc dataset and constructed an IQ test based on the questions. Given the ease of the questions even in the "hard dataset", possibly timing it like the wonderlic would make sense. I would be curious what the distribution etc of that dataset.

Chollets perception and articulation are unmatched

Is this guy for real?

Great Video !

Refreshing view! ❤

We don't need AGI, we need tools, not gods

but what about AlphaProof, AlphaGeometry 2, Q-star, the upcoming Claude 3.5 Opus and Gemini 2 Ultra (aka gemini-test in lmsys) ?

Excellent and succinct distillation of the current state of the industry and the path forward. Very inspirational!

There is no viable path to AGI with the current hardwares and algorithms.

All hyped with LLM, but we do not need human imitation we need human reasoning. The static nature of llm goes against this basic concept, us humans have to think about solutions, make projections in our minds, is not input output. LLMs are just like hype realistic paintings, they look somehow like humans but they are fixed in time.

this is really helpful/thoughtful just joining the competition and this is an exceptional resource.

Data protection, but data stored in the US? Looking forward to Schrems III... The expressed views on data protection (let's put all in ChatGPT!), AI Safety (not important, AGI is far away!) and AI ethics (who needs regulation!) are not very encouraging.

It will not have any effect, because AGI will never happen. I am an AI engineer, we don’t even have true AI today. It’s just machine learning trained on data we give it. It does not think for itself or come up with anything outside of the data it has ingested

1N73LL1G3NC3 15 7H3 4B1l17Y 70 4D4P7 70 CH4NG3. -- 573PH3N H4WK1NG leads to ARC-AGI, Th(X/C) guys! Genius is the ability to create change -- Me if no one else wrote it! (too lazy to l33tcode) hopefully will lead to ARC-AGG, Merci par avance les mecs! :D Bis because of the quotation! :D

can be a big thing! let's se after playing a bit

1N73LL1G3NC3 15 7H3 4B1l17Y 70 4D4P7 70 CH4NG3. -- 573PH3N H4WK1NG leads to ARC-AGI, Th(X/C) guys! Genius is the ability to create change -- Me if no one else wrote it! (too lazy to l33tcode) hopefully will lead to ARC-AGG, Merci par avance les mecs! :D

genius if nobody did that and performance to be validated on my use-case. when will be released the first Open Source component derived from this initiative ? (sorry if I it is already available but didn't find it yet...)

let's play a bit!

I am new to programming, but this challenge and task really interests me and I'd like to give it a try. Could you create a tutorial on how to submit an entry to the arc challenge? maybe with a model which will produce some minimal results?

Could someone please explain how the AI soccer players in a simulation can go from physically flopping around on the ground to teaching themselves team strategy but AI can't solve these ARC tasks?

i might be tripping but i think this dude cloned his own voice and then layered it into the video. you can hear the typical elevenlabs lisp

Why is the train/evaluation set so small?

Mark Knoop is a very smart guy. Knows a lot about ML for being an outsider in that space and meche background

Excellent.