Yes this video was made over 6 months ago and they have changed, gpt-3.5-turbo is also not the best at PAL. GPT-4 works better but some of the open source models have also been getting better lately too.
Glad you enjoyed it! There are a few more papers this with cool examples of using prompting to do unconventional things. You might like the video and code on Visual ChatGPT which uses other external tools via prompting as well.
Hey great video, what did you mean when you said the example with reaching to school is incorrect because there were no similar prompts before. I can see it followed the exact same step as the first example.
What would be the approach to find the issue why it is calculating wrongly? Is there a way to look into the NN or we are just changing the prompt and hope for a better result. So my question is how we could "look" into the NN to find the error or i am thinking to easy? Thanks
probing in the NeuralNet it's self is a major research effort. Most likely if it is the model itself that is calculating wrongly, then it just hasn't seen enough examples. If the issue is with PAL then I would suggest you work on modifying the prompt.
Sam, When I first saw the video, the PAL module's importance was unclear to me (mea culpa). Now watching the video again and reading the article, I recognize great potential. Thanks for the clarification.
Thanks for your video, Sam. I hav one doubt whether langchain+LLMs like flan T5 x, T5 xl, flan 20 B & flan UL20B accessing using huggingface tokens ( other than downloading model) is free of service. When we are considering alternative to open ai gpt3 models.
Their API is is not a totally free of service and to be honest I wouldn't use the free version to put something into production, that said I have never maxed out my free API calls so far. There doesn't seem to be a lot of info on their pricing page huggingface.co/pricing this could be that they are in startup mode to get lots of users before charging etc. The main disadvantage I have seen is that some of the bigger models often don't work with the API. I do wonder how long they will keep the Flan UL20B up for free for example. If I find something better I will certainly make a video of it. If you find something else please let me know :D
Thank you very much for your fantastic videos! Can a PAL also be simulated within a LLM? So, through specific prompting the LLM simulates a Python environment in which it does the calculation and then returns the answer? Does that make sense?
Nice and interesting if u want to aks more math question, you used code-davinci-02 in the setup for pal but can u combine it with the normal llm like chatgpt-turbo3.5 ?
Fantastic videos Sam - this playlist has (happily) consumed my entire night. Who needs fresh laundry anyway?!
Hopefully the LLMs can do the laundry soon. :D Thanks for the kind words.
Please note PALChain has now moved to langchain_experimental. Also code-davinci-002 model does not work use gpt-3.5-turbo instead.
Yes this video was made over 6 months ago and they have changed, gpt-3.5-turbo is also not the best at PAL. GPT-4 works better but some of the open source models have also been getting better lately too.
Fantastic video ! I watched it like 3 times over last few days as it makes me think what to append to make LLM useful ... feed us more ! thanks
Glad you enjoyed it! There are a few more papers this with cool examples of using prompting to do unconventional things. You might like the video and code on Visual ChatGPT which uses other external tools via prompting as well.
Very good video on Program-Aided Large Language Models
Hey great video, what did you mean when you said the example with reaching to school is incorrect because there were no similar prompts before. I can see it followed the exact same step as the first example.
What would be the approach to find the issue why it is calculating wrongly? Is there a way to look into the NN or we are just changing the prompt and hope for a better result. So my question is how we could "look" into the NN to find the error or i am thinking to easy? Thanks
probing in the NeuralNet it's self is a major research effort. Most likely if it is the model itself that is calculating wrongly, then it just hasn't seen enough examples. If the issue is with PAL then I would suggest you work on modifying the prompt.
Sam, When I first saw the video, the PAL module's importance was unclear to me (mea culpa). Now watching the video again and reading the article, I recognize great potential. Thanks for the clarification.
unfortunately that model is no longer publicly available on OpenAI.
Makes sense. Even I can't do math without calculator.
+
Thanks for your video, Sam. I hav one doubt whether langchain+LLMs like flan T5 x, T5 xl, flan 20 B & flan UL20B accessing using huggingface tokens ( other than downloading model) is free of service. When we are considering alternative to open ai gpt3 models.
Their API is is not a totally free of service and to be honest I wouldn't use the free version to put something into production, that said I have never maxed out my free API calls so far. There doesn't seem to be a lot of info on their pricing page huggingface.co/pricing this could be that they are in startup mode to get lots of users before charging etc. The main disadvantage I have seen is that some of the bigger models often don't work with the API. I do wonder how long they will keep the Flan UL20B up for free for example. If I find something better I will certainly make a video of it. If you find something else please let me know :D
In March of 2023 OpenAI deprecated the code-davinci-002 model. You can get the first example to run if you replace it with "gpt-3.5-turbo".
great video! Is OpenAI function calling replacing this?
they can if used with REPL etc but function calling has more replaced ReACT etc.
Hey Sam, great Channel, appreciate it!!!!
Hey man, great video. Did you read LMQL paper by any chance?
I did its certainly an interesting idea. I might make a vid on it.
Great job man. I subbed. Lets keep em coming!
Thank you very much for your fantastic videos! Can a PAL also be simulated within a LLM? So, through specific prompting the LLM simulates a Python environment in which it does the calculation and then returns the answer? Does that make sense?
It has been tried and can work to a degree, but often will get small parts wrong and thus hasn't been reliable so far.
Nice and interesting if u want to aks more math question, you used code-davinci-02 in the setup for pal but can u combine it with the normal llm like chatgpt-turbo3.5 ?
Yes but it doesn't always work as well. unfortunately the code-002 model has now been taken down on the OpenAI API. GPT-4 does a pretty good job
Great video. Very helpful
Glad it was helpful!