Видео

@clay1958 BERT and ModernBERT are general-purpose NLP models. They were trained using masked language modeling (MLM), which involves predicting masked tokens in a sequence. Using MLM, the models can learn to understand the context and can capture lots of knowledge due to large pre-training corpora, but you usually want to fine-tune them for your downstream tasks to get the best results. For example, if you want to use ModernBERT for semantic search, you might want a version fine-tuned on MS-MARCO, a popular retrieval dataset with contrastive examples. (This is what I did in my video.) OpenAI's embedding models are already fine-tuned on some internal datasets. They're designed to work well for tasks like semantic search, clustering, and recommendations right out of the box, so you don't need additional training to get started. ModernBERT Embed (huggingface.co/nomic-ai/modernbert-embed-base) is probably a good alternative to OpenAI's embed models at the moment. It's trained on the Nomic Embed datasets and performs on par with OpenAI's text-embedding-3 models on benchmarks like MTEB. Plus, you will also get all the benefits of using open-source models.

@@botsknowbest Thank you for this reply! And fantastic video by the way

Thank you!!

Thank you!!

Thank you so much for your comment! I'm really glad you enjoyed the video!🙂 Regarding your question, I agree that re-running inference with representations like FEN could help prevent illegal moves. We would be basically prompting for a more diverse output, hoping to get a new legal move. Besides FEN, we could also try sampling moves multiple times with high temperatures or selecting the highest-scoring legal move from the output distribution (assuming we have access to logits). But even if that works, I would worry that the model is still misunderstanding the given position and that re-running only produces a legal move at the cost of lower performance. Also, this approach wouldn't address cases where LLMs hallucinate legal moves, like in 5:37. We could also add FEN (or some other representations) to every move and provide more information about each position like that. That should work in theory, but in my previous video about GPT-3.5, I experimented with PGN, FEN, visual formats and chain of thought variants, and I only got a small boost with FEN+PGN. Since this video involved quite a lot of games, I ended up using only PGN to make everything faster and more cost-efficient. But I might revisit other notations in future videos! Hope this makes sense, but let me know if you have any other questions 🙂

Thanks!! Yeah, playing in one session just doesn't work atm🙂Haven’t tried GeoGuessr yet, so thanks for the tip!

Just make it easy for most people so you gotta consider that

@danielkempenaar154 I think it's similar to many other tech jobs now-AI is not necessarily replacing them, but knowing how to use AI effectively can make you much better at your job. For example, software developers who are good at coding and know how to use AI (e.g., Copilot) are doing well now. So, I think you will be fine, but I would suggest not ignoring AI and staying updated on the latest advancements in your field so you can use it if needed. Hope this helps!

Thank you!!

So, I set the temperature parameter to 0, which makes the output deterministic. Thus, re-running the model would be pointless because it would return the same answer. When the model suggested an illegal move, I simply forced it to resign. You can see an example in the 6th game with Isabel (lichess.org/study/IdRA6R7f/JsZGxPjU). If I remember correctly, GPT4o lost its Queen and then tried to play again as if it still had the Queen. However, this was quite rare, and it could even play games lasting over 100 moves, as I showed in the video. Its 'understanding' of the game after 100 moves is a bit questionable, though, and you can see how poorly it plays at that point. I noticed that it is much more likely to struggle and suggest illegal moves when it's losing. It's quite confident in its own play and doesn't like to accept that it might be losing. So maybe you were just playing much better than these bots, and GPTo couldn't keep up with you. 🙂

No. Watch the video

Thanks for your comment! You're absolutely right- AI's perception of the world is fundamentally different from human understanding, and this can become quite dangerous if people start placing blind trust in these systems. Personally, I find it interesting how each new model iteration improves in reasoning and provides more accurate and helpful responses. That being said, I also always analyze its mistakes, which can tell you a lot about the current state of research. When I tested these models on zebra puzzles, several models made basic reasoning mistakes that no human would likely make. As I showed in the video, o1 seems to be a substantial improvement on this front. But would I trust it to drive my car? Definitely not. One last point- it's important to remember that while its output may look plausible, it's still model-generated, and we don't know if the model is describing the true reasoning process behind the final answer. These models, especially the proprietary ones, remain largely a black box. Hopefully, we'll see the development of more transparent models in the future.

Thanks- glad you liked it! 🙂

Thank you!

Thank you!

Thanks for the great question! While getting the answer directly in JSON format would be convenient, not all LLMs can handle this consistently. In my experience, asking for JSON output tends to negatively impact reasoning, and I didn't want this to affect the chain-of-thought generation. Instead, I added "Format the final answer as Answer = ..." at the end of my prompts. That was easy to parse and worked well for almost all models. A few smaller models sometimes didn't comply, but regenerating the output with higher temperatures always fixed the problem. If I were developing a product and needed a more reliable solution, I would first generate the CoT reasoning, then run a separate prompt to extract the answer-perhaps even using constrained decoding to ensure the response fits one of the expected options.

Thanks so much for your comment! I see your point. I think that would definitely be true if we were directly removing the options from the list of possible answer candidates (e.g., allowing only houses 3 and 4 to be selected). But that's not the case, and the model can still choose any of those answers. The thing with language models is that they can be very smart on the one hand and not understand simple logical deductions on the other (e.g., the infamous Reversal Curse). So, even if we tell an LLM that X occupies House 1, the model might not necessarily deduce that Y cannot live there at the same time and eliminate it. It can also happen that it reasons correctly but then forgets about this decision later in the output and still makes a mistake. So, I would say it still makes sense to test models even in such cases. What do you think? Regarding the results, I'm sure there could be many improvements. Like, I didn't calculate a p-value or anything, and it would probably be better to fully separate the results based on the number of constraints. But all models were tested on an identical set of 500 puzzles, so that should make for a good enough comparison :)

Thank you so much! I'm really glad you enjoyed the video, and I really appreciate the subscription and suggestion! 🙂 By ARC, do you mean the Abstraction and Reasoning Corpus? That's a great idea - I really enjoyed reading the 'On the Measure of Intelligence' paper.

@BeeJamin-i5b Thank you!!

@ronnielane6903 Thanks! Yes, definitely give it a try. It improves substantially and rarely makes illegal moves. I'm still amazed that it can grasp positions so well, even after 100+ moves.

Thanks for the suggestions! Yeah, I've tried textual representations for chess before, but it didn't work well. And I assume the same here. But creating a visual map of the moves, as you suggested, sounds like a great idea! GPT-4o with visual inputs is more affordable now, so maybe I should give it a shot!

@@botsknowbest it would be great to see something like that. New sub earned mate. Nice work

@@matiascoco1999 Thank you!!

Thanks so much! I agree with you that it's absolutely reasonable to expect a typical chat interaction. I'm mostly highlighting the fact that using the right input format can lead to very different outcomes and that current LLMs are still very sensitive to variations in prompts. The difference in performance is substantial and almost feels like interacting with two different models. As LLMs get better, I expect this gap to get much smaller. I haven't tried Claude or Grok yet, but I will include them in the follow-up video. 🙂

Thanks for your comment! Not yet, but I plan to include more models in the follow-up video, including Claude, Gemini, and Grok.

Thanks for your comment and these suggestions! The video involved lots of experiments (games), and my goal was to keep the prompts as short as possible to make everything computationally feasible. I actually tried experimenting with CoT, PGN, FEN, and a few visual formats in my previous videos, and I got small improvements when using FEN+PGN. But maybe I should revisit these formats in the follow-up video!

@aminothing-ps4ev Exactly! Like of course its chess skills are far from perfect and Stockfish/AlphaZero is much better...but GPT was never directly trained to play chess!

And it's playing blindfolded!

Thanks!

Thank you!! Glad you liked it 🙂

Glad you liked it! This applies to any task, so it’s always a good idea to spend some time on prompt engineering and try different input formats.

Thank you! Glad it was helpful 🙂

Thanks so much! And thank you for the feedback about links- I wasn't sure if people find it useful or not 🙂

@@botsknowbest we do! I needed that for citing it in my thesis :D

@@dariyanagashi8958 Perfect! 😀All the best with your thesis!

@@botsknowbest thanks 😊

Thank you!

I just did a few tests and random exploration could solve about 50% of 4x4 mazes and 20% of 5by5 mazes. 10x10 has way too many decision points to be solvable by random walk in less than 100 moves. But I want to run more comprehensive tests with random walk next week or so, and will let you know!

Thank you!!

Thanks for the suggestions! If I manage to make another video on this topic, maybe I can test prompting with different maze-solving strategies (like hand on the wall) to see if it knows these concepts and can apply them. A few-shot inference with examples of solving a maze could also be interesting. Yeah, Claude 3 Opus would be great. Setting up these three models took already more time than I wanted 😅, but I will definitely include more models in future tests!

Thanks! Yeah, I agree that summary would not work in this case but simplifying the format is a great idea! GPT4 is quite good with JSON/Python-like formats, so maybe I can try passing the previous moves and available directions like that next time.

Thank you!