DSPy Explained!
HTML-код
- Опубликовано: 14 июн 2024
- Hey everyone! Thank you so much for watching this explanation of DSPy! DSPy is a super exciting new framework for developing LLM programs! Pioneered by frameworks such as LangChain and LlamaIndex, we can build much more powerful systems by chaining together LLM calls! This means that the output of one call to an LLM is the input to the next, and so on. We can think of chains as programs, with each LLM call analogous to a function that takes text as input and produces text as output.
DSPy offers a new programming model, inspired by PyTorch, that gives you a massive amount of control over these LLM programs. Further, the Signature abstraction wraps prompts and structured input / outputs to clean up LLM program codebases. DSPy then pairs the syntax with a super novel compiler that jointly optimizes the instructions for each component of an LLM program, as well as sourcing examples of the task.
Here is my review of the ideas in DSPy, covering the core concepts and walking through the introduction notebooks showing how to compile a simple retrieve-then-read RAG program, as well as a more advanced Multi-Hop RAG program where you have 2 LLM components to be optimized with the DSPy compiler! I hope you find it useful!
Please send me a message @CShorten30 on X, I would love to discuss what you are working on and offer any help with DSPy and/or Weaviate!
Useful links:
Join the DSPy Discord! / discord
DSPy GitHub: github.com/stanfordnlp/dspy
DSPy Paper: arxiv.org/abs/2310.03714
DSPy Assertions: arxiv.org/abs/2312.13382
A Guide to LLM Abstractions: www.twosigma.com/articles/a-g...
Jason Liu: Pydantic is all you need: • Pydantic is all you ne...
LlamaIndex Query Engines: docs.llamaindex.ai/en/stable/...
LangChain: js.langchain.com/docs/use_cases
Chapters
0:00 DSPy!
1:37 The DSPy Programming Model
5:07 LLM Programs
8:56 Programming, Not Prompting
13:48 PyTorch Analogy
19:16 The DSPy Compiler
27:50 Metrics
30:13 Teleprompters
30:50 Code Example
50:37 Start using DSPy! 
Наука
I think one of DSPy's most valuable contributions is a standardization of benchmarking. This will help make comparisons between models more transparent as you don't have to worry about biasing the test results because your prompt choice was unoptimized for a particular model.
Love this! 100%
Basically yeah, it will somewhat make it generic, the whole flow.
Thanks for the intro and explanation. One suggestion: could you not zoom in and out so much while going through the demo, because there's a lot of code on the screen and typically viewers' eyes move about to try to make sense of it. Constant zooming would disrupt the process, and tbh I just don't see a point of zooming because the text is already big enough to see without it, and you already can move your cursor or select text to highlight a part.
Ah I'm really sorry about that Xiyang! I am still figuring out screenstudio and I am very sorry for this mistake, I will be sure to correct it in future videos. Thank you so much for sharing this with me!
Code is suspect anyway. Like at 11:19, there's a loop through emails that immediately assigns to emails within the loop body.
@@connorshorten6311 Personally I found it very useful that you zoom regarding the sections or portions you are discussing. This keeps viewer attention in-task, and made it more accessible.
The zooming action is a a bit disturbing while I am focusing on the content. Other than that, your content of your video is awesome!
Agreed! I had to "watch" most of this video with my eyes closed as it was triggering motion sickness. A little bit of zooming might be useful on occasion, but you only need to go in like 10% to emphasize something, not 100%.
I am a deep learning practitioner working with PyTorch and find this video very informative. Thank you Connor!
So happy to hear it! Thank you so much!
Great explanation Connor you put it very well and gave the right bits of information and details
Thanks Karl!! Really appreciate it!
Thanks for this deep dive 🚀
Thanks so much Erika!
I agree with one the comments below, zooming in and moving the cursor and talking very fast makes it bit hard to follow. But I'm doing it sometimes too during the meetings so I know exactly where it comes from ;). Having that said...
Very informative video and explanation. Great job and big thank you! I just subscribed to your channel and looking forward to learn more!
Thanks for the great video, but please slow down. It is hard to follow a tutorial if you talk so fast.
Thanks for sharing. I now have a new way to think about promot chaining. DSPy indeed
Love to hear this! Thanks so much for watching!
Thanks Connor for the deep dive.
In 36:00, when you talk about whether using multiple dspy.ChainOfThought modules or using only one is the same, it is actually NOT the same. If you look closely at the compiled state of your dspy program, every module in the pipeline has its own bootstrapped examples after compilation to maximize correctness in every step. In this case, the second dspy.ChainOfThought module will also have the passages context from the previous one.
Thank you!! Apologies, I understand this now -- thanks so much for documenting this here!
Thanks Connor for getting a video out on this!
Thanks Knox!
Coming from twitter.
first minute is like, its just another layer of abstraction....
8 minutes in: OMG, this would change how I work
So happy to hear that! Haha, love the story of that!
Love the video! I’m not a programmer and I was able to follow along 😅
Excellent explanation!
man, that's an amazing overview, thank you
Means a lot! Thank you so much!
came for twitter! excellent explaination!
Awesome! Thanks so much!
amazing explanation!! thanks for the video 🙏🏻
Means a lot! Thank you so much!
Thanks for the overview!
Thanks for watching!
where is link to code used at 31:09 pls, as it is written :walking through the introduction notebooks showing how to compile a simple retrieve-then-read RAG program
that's an excellent explanation of the stuff. TA!
Thanks for sharing this review about DSPy, it's amazing. I'm very excited about this framework.
However, I don't understand how to create a DSPy system capable of sending different queries in a single request. For example, imagine that the program receives queries={'key_1':'query1', 'key_2':'query2', ...}, snippets={'key_1':'snippets1','key_2':'snippets2' , ...} and returns the response in JSON format answer={'key_1':'answer1', ...,key_n:'answer2'} in a single call. That is, I have been able to design a zero-shot program that is capable of answering different questions in a single call, but it is not at all intuitive when it comes to optimizing it.
P.S. To make the output a dictionary I used 'type_predictors', that is, Output(Basemodel) defines each of the keys key_1,key_2, etc...
great presentation!
Thanks for this video. I watched this on 2x speed and it was very informative
Awesome! Really happy to hear it, thank you so much!
Great vid! If you return to it in the future I'd love to see how dspy handles tool calls in models like GPT-4. I spend so much time "bargaining" with my prompts to consistently call functions XD...
great video! BTW what app/software are you using to zoom in and out in the video like that?
🎯 Key points for quick navigation:
Exciting new framework
Programming model optimization
Graph computation programs
09:55 *Signature, dock string*
10:10 *Prompt optimization, syntax*
10:38 *Input, output fields*
10:52 *Control flow, loops*
11:20 *UAPI, web queries*
12:26 *DSPy Assertions, suggestions*
13:49 *Citation attribution suggestions*
14:16 *Optimization, instructions, examples*
14:59 *DpY as PyTorch*
16:31 *Inductive biases, depth*
17:43 *Intermediate supervision, DpY compiler*
19:22 *Testing with programs*
19:35 *Optimizing instructions and examples*
20:03 *Automatic data labeling*
20:46 *Ending manual prompts*
21:14 *Adapting to new models*
22:49 *Structured output with prompts*
25:33 *Fine-tuning neural networks*
26:26 *Using few-shot examples*
27:51 *Bootstrapping rationales*
28:19 *Evaluating synthetic examples*
Overlapping keywords metrics
LM judge prompt
LM produce metric
37:58 *Deep learning paradigm shift*
38:41 *Data set formatting*
40:03 *Inspect intermediate outputs*
40:59 *Add Chain of Thought*
43:35 *Define optimization metric*
45:13 *Value of reasoning*
45:28 *Inspect parameters*
46:25 *Multi-hop search integration*
Queries connected to final answer
Introduction to multi-hop search
Supervision on intermediate hops
Made with HARPA AI
Connor returning to the original Vid Channel!! Good video as expected
Super thanks for you explaination
First of all, thanks for the amazing content you are putting out.
Small suggestion: The code is visible fine as it is. There is no need to zoom in to the text based on the cursor. This zoom movements, some too fast that we can even see motion blur 😊, actually take the focus out of the video as the eyes tries to focus on text movements and strains the eyes too. Thanks again.
Sorry about that, I have stopped the zooming in the future videos in this series! Please note the code can also be found on GitHub at weaviate/recipes!
Thanks for doing this! Been wanting to wrap my head around DSPy for awhile. Any chance you can upload your notebook example?
Awesome, thank so much so happy to hear it! This notebook is just the DSPY intro notebook with all the comments stripped out! I have linked in the video description and I will try again here although I'm not sure if RUclips is happy with links in comments - colab.research.google.com/github/stanfordnlp/dspy/blob/main/intro.ipynb
Wow glad I found this
Great content, man! Just wondering if you'd have a notebook with theses basic examples provided in python? Thanks
I always thought it was pronounced as D ES Pie.
thanks for the deep dive btw!
Is it possible to run DSPY on local windows environment, say with Mistral 7b model? It fails to for me because of default value of url param, which I do know how to avoid.
Could you please add the notebook you used?
which tool are you using for presentation and zoom animation?
Excited to dig into this!
Thanks so much Greg, really appreciate it! Hope you find it useful! DSPy!
"Supervision on intermediate hops" - we would like to know more about that.
Overall, looks pretty dope. Need to dive deeper.
love this... so dope!!! i guess im on the right track... i have been working on this for 2 years now and coming up with ways to dynamically give the most context and this was one of the concepts... i have a few more tricks added to it.... I'm going to get it all up and then start showing RUclipsrs my papers
welcome back, been a long time
Haha thank you so much!
Could you please create a video showcasing various use cases? While I find the previous version impressive, it's a bit challenging for me to come up with different use cases. I'm currently working on writing a book, and I have a significant amount of unstructured text and data. I'm open to getting some assistance with this. Could you please shed some light on this topic?
Good content.
An analogy is only useful if people already know it.
Might have been helpful to show the standard CNN diagram for image classification.
21:15 😂😂😂
Really appreciate the video in 'dark mode'
🎯 Key Takeaways for quick navigation:
00:00 *🤖 DSPy is a new framework for developing LLM programs, offering control and flexibility over LLM programs with a syntax inspired by PyTorch.*
00:11 *📦 DSPy allows for complex tasks to be broken down into smaller tasks, which can be parallelized and controlled individually.*
00:39 *🛠️ DSPy introduces a new syntax that combines with optimization for better performance in LLM prompts.*
01:20 *🎯 The goal of DSPy is to automatically optimize instructions and examples used in prompts to elicit desired behavior.*
01:46 *💼 DSPy programming model is described as PyTorch meets agent syntax for LLM programs, allowing for the initialization of components and defining forward paths.*
02:26 *✍️ DSPy offers a signature syntax for cleaner code and easier organization of LLM programs.*
03:08 *📝 The shorthand syntax in DSPy simplifies the expression of input and output fields in programs.*
03:50 *🔄 DSPy enables local memory and control flow in LLM programs, allowing for loops and if-else statements.*
05:15 *🔗 Large language model chains, which connect LLMs with themselves, have been popularized by Lang chain and have evolved into more complex structures like graphs.*
06:24 *🧠 Chains help overcome input length limitations and overly complex tasks by breaking them into manageable subtasks.*
07:22 *🌐 Text transformation graphs in LLM programs allow for parallel processing and synchronization of nodes to produce coherent outputs.*
08:31 *📈 LLM programs can be thought of as a more organized and flexible way to build applications with large language models.*
09:11 *🛠️ DSPy is positioned as a full-blown LLM programming language, offering more flexibility and control compared to frameworks.*
10:52 *🔄 Control flow in DSPy allows for the integration of loops and if-else statements in LLM programs.*
12:26 *🔍 DSPy assertions define rules for model behavior, ranging from strict assertions to suggestions for better performance.*
14:03 *⚙️ Optimization is a key feature of DSPy, enabling automatic tuning of instructions and examples for better LLM performance.*
15:12 *🧬 DSPy's design is inspired by PyTorch, with analogies drawn between neural network layers and LLM program components.*
17:00 *📐 Inductive biases and depth in DSPy programs are similar to those in neural networks, adding complexity and functionality.*
18:25 *🎓 In LLM programs, like in neural networks, supervision is typically only applied to the final output, not intermediate steps.*
19:22 *🤖 The DSPy compiler optimizes instructions and examples, using LLMs to improve LLM programs.*
20:32 *📝 Instruction tuning in DSPy aims to end manual prompt tuning by automatically optimizing instructions for different language models.*
22:09 *🔄 Example optimization in DSPy involves selecting and generating examples to improve LLM performance.*
23:57 *📊 Few-shot learning and fine-tuning are key concepts in DSPy, enabling models to learn from a small number of examples or through gradient updates.*
26:26 *📈 Bootstrapping few-shot examples in DSPy helps select the most relevant examples for inclusion in prompts.*
28:05 *📏 Metrics in DSPy assess the quality of synthetic examples, guiding the optimization process.*
30:20 *🎭 Teleprompters in DSPy orchestrate the optimization loop, exploring different instructions and examples to improve metrics.*
31:01 *📝 DSPy offers off-the-shelf compilers for Chain of Thought examples, allowing for easier learning of the syntax.*
31:14 *🚀 Retrieval Augmented Generation (RAG) is a popular LLM chain in DSPy, showcasing the basic syntax.*
31:42 *🧠 In a RAG program, components are initialized and then interact with input data in the forward pass.*
32:10 *✍️ DSPy supports shorthand notation for signatures, making it easier to describe tasks for LLMs.*
33:05 *📚 Shorthand notation is powerful for writing papers and presenting new LLM chains.*
33:47 *🤖 Multi-hop question answering systems in DSPy optimize LLM programs for complex tasks.*
34:29 *🔍 Multi-hop question decomposition is a powerful feature in DSPy, enhancing LLM programs.*
36:09 *🔄 DSPy allows for looping and helper functions in the forward pass of LLM programs.*
37:44 *📊 Hotpot QA dataset is used to benchmark multi-hop question answering in DSPy.*
38:13 *💡 DSPy's new framework requires only a few examples to optimize LLM programs.*
39:22 *🧩 Longer hand notation in DSPy allows for more detailed descriptions of tasks.*
39:50 *🔎 Intermediate outputs in DSPy can be inspected, similar to PyTorch's flexibility.*
41:13 *🧠 Adding Chain of Thought to DSPy prompts improves performance and debugging.*
43:07 *📝 DSPy's Chain of Thought adds reasoning to prompts, enhancing LLM outputs.*
44:32 *🔄 Compiled DSPy programs can be run with new inputs, similar to PyTorch models.*
46:39 *📈 Multi-hop search in DSPy involves breaking complex questions into sub-questions.*
48:02 *🧐 Intermediate components in DSPy programs don't need to be supervised, similar to deep learning.*
49:38 *🎓 DSPy can use teacher models to supervise other models, enhancing learning.*
50:19 *💻 DSPy provides a fun syntax for writing LLM programs and organizing prompts.*
51:28 *🤖 Chain of Thought in DSPy eliminates the need to manually write rationales.*
51:57 *🔄 DSPy allows for quick adaptation to new language models by recompiling programs.*
53:06 *🚀 Local LLM inference with AMA is becoming faster, making DSPy more compelling for fine-tuning models.*
Made with HARPA AI
🍿 Love how all the AI hipsters are slowly discovering finite state machines, strategy patterns, dependency injection, ASTs and Markov Trees. And giving all of these old and proven information systems control flow patterns new names along the way 🪄 truly majestic
As the paper indicated, "DSPy is pronounced dee-ess-pie. It’s the second iteration of our earlier Demonstrate-Search-Predict framework (DSP; Khattab et al. 2022)." So, it should be pronounced as "DS-Py":)
Haha if only I had a time machine.. Sorry about that! I first started studying this with the "DSP" as you mention and for some reason my brain just decided it was "D-S-P-Y", but indeed it is dee-ess-pie... My apologies, I will fix this in future videos / podcasts / etc.
I was curious about what DSPy stands for? finally found it in the paper;)
Thanks
Is this notebook shared somewhere?
Can DSPy handles prompt injections?
Interesting question -- I am not an expert on prompt injections but I have watched Karpathy's video on the topic haha! My intuition is that it would be much harder to pass an adversarial prompt through layers of an LLM program like DSPy describes.
📝 Summary of Key Points:
📌 The video introduces dpy, a programming language for large language models (LLMs) like GPT-3. It allows users to program LLMs by defining components and their interactions in a forward pass. It enables breaking down complex tasks into smaller subtasks using multi-hop question answering as an example.
🧐 Dpy offers optimization capabilities by optimizing instructions and examples used in the prompt to elicit the desired behavior from the LLM. It provides syntax inspired by PyTorch, offering control and flexibility over LLM programs.
🚀 The video discusses the use of signatures to define prompts and structured input-output fields, as well as the use of assertions to control LLM behavior. It mentions the dpy compiler for optimizing instructions and examples, and the use of metrics and teleprompters for optimization.
🚀 The video also mentions the use of a system called DPy, which generates queries based on a given context and question. It retrieves relevant passages, duplicates them if already in the context, and produces the next query to continue answering the question.
🚀 DPy provides a fun syntax, helps organize longer prompts, and eliminates the need to write rationales manually for Chain of Thought prompting. It is flexible in adapting to different language models and can be used with local language models for faster and cheaper inference.
💡 Additional Insights and Observations:
💬 "DPy allows you to program LLMs by defining components and their interactions in a forward pass."
📊 No specific data or statistics were mentioned in the video.
🌐 The video does not reference any external sources or references.
📣 Concluding Remarks:
The video introduces dpy as a powerful programming language for large language models (LLMs) like GPT-3. It highlights the ability to break down complex tasks, optimize instructions and examples, and provides control and flexibility over LLM programs. DPy is also discussed as a system that generates queries based on context and question, offering benefits such as a fun syntax, organization of longer prompts, and flexibility with different language models. Overall, dpy is presented as a valuable tool for programming LLMs with various features and capabilities.
Generated using TalkBud
Summarized with DSPy?
@@connorshorten6311 No, using TalkBud, a web browser extension that has many functions, you should give it a try.
Why do they call it 'compiling'? It's not generating machine code, it is modifying your prompt strings. Unless I'm misunderstanding.
Yeah I think exactly what you described -- the perspective is that the high-dimensional prompt/examples is similar to LLM computing as machine code is to traditional computing. I have seen "LLM Computing" used often, but maybe a cool one haha. Do you like the LLM Operating system papers, I guess in that line of analogical thinking.
@@connorshorten6311 I'm def on board with the "LLM OS" line of though, with some reservations, but anyone developing such an OS should probably not overload words like 'compile'!
This is my biggest gripe w it… they seem to go out of their way to avoid calling what they’re doing prompt optimization and prompts. Instead they say they’re “compiling” a “program.” Guys, it’s auto prompt engineering. Just say it.
From the git repo.
"DSPy can routinely teach powerful models like GPT-3.5 or GPT-4 and local models like T5-base or Llama2-13b to be much more reliable at tasks, i.e. having higher quality and/or avoiding specific failure patterns. DSPy optimizers will "compile" the same program into different instructions, few-shot prompts, and/or weight updates (finetunes) for each LM. This is a new paradigm in which LMs and their prompts fade into the background as optimizable pieces of a larger system that can learn from data. tldr; less prompting, higher scores, and a more systematic approach to solving hard tasks with LMs."
@@user-ke6um5to8c " DSPy optimizers will "compile" the same program into different instructions, few-shot prompts, and/or weight updates (finetunes) for each LM. This is a new paradigm in which LMs and their prompts fade into the background as optimizable pieces of a larger system that can learn from data. tldr; less prompting, higher scores, and a more systematic approach to solving hard tasks with LMs."
nice video and helpful info!
the quick zoom ins/outs is quite distracting tho
1:35 but I like emotional prompting...its more fun!
I think DSPy should have a low code UI layer, similar to autogen studio. That should accelerate its adoption
Agreed! Super cool idea, probably a solid startup opportunity! Go go go haha!
can you share notebook code used for example pls
DSPy has a git repo with extensive colab notebooks.
Super interesting topic, thank your for the heads-up
BUT: speaking fast, mumbling, quickly zooming in and out & scrolling... makes me feel dizzy... opposite of Andrej Karpathy's tweet lol
I had to pause many times to get a glimpse of the code between 2 zooms & scroll
Sorry about that! Made a massive mistake with the ScreenStudio, will fix going forward!
29:07, Rayleigh is pronounced Riley, FYI.
Haha appreciate that!
Thank you for great content, but the zoom in/zoom out effects on video are really annoying! Why not using simple highlighting or just leave the slide as simple as they are?
You're smart Connor. lol
Haha thanks Clint, really appreciate it!
the compiler prompt: be a good boi 😂
Haha exactly, what works for dogs should work for LLMs
1. Dee Ess Pee Why (What you call it)
2. Dee Ess PIE (What I've heard Omar call it)
3. DIS pee (What I want to call it)
Haha, #3 going forward
phrasing langchain and llamaindex as the predecessor of DSPy is really bad introduction . DSPy is a totally different animal than langchain and llamaindex . On the other hand , langgraph is making their own wheel when there is whole bunch of excellent libraries such as celery , can do even complicated graph tasks , which is great for all kinds of graph LLMs tasks .
Getting ride of langchain and llamaindex will get you freedom of using all kinds of excellent python framework and libraries , or you have to locked in with their so called "wrapper ecosystem " .
I guess it depends on how you define "predecessor", but I understand how it could be interpreted that way. I think it is commonly agreed that LangChain and LlamaIndex have had a massive impact on the "Agent" developer tool market, which is how I would classify DSPy as a commercial product. Academically I completely agree that it is more akin to LangGraph (I will need to look into celery I thought that was a distributed messaging queue library in Python). Agreed 100% with the last point. Thanks so much for the comment, appreciate these insights!
@@connorshorten6311 In my opinion, DSPy is more about flow engineering than applying agent-based concepts, as Omar briefly suggested in his interview. With features like chain, group, chord, and prioritization, Celery can handle far more complex task dependencies and has been incredibly useful in my LLMs applications with intricate workflows.
Ultimately, by moving away from agent concepts and concentrating on flow engineering, one can greatly broaden their scope and potential for future advancements.
I never understood the value of langchain
Can I ask where the lockin is for llamaIndex when you air quote 'their ecosystem'? Also I thought most folks were moving to AutoGen or agent swarms off langchain now, no?
those instructions begging the LLM to return JSON hit too close to home
Great vid tnx. Seriously tho can we start naming these things better jeez its not that hard folks. I guess at least they didnt name it co-pilot
Wonderful content thanks for explaining and sharing!! But please please don’t do this zoom in thing. I feel dizzy 😢 I normally watch in 2x but even 1x is not a good experience
“FOR THE LOVE OF GOD ONLY RESPOND WITH JSON !!!”
Haha, I bet that one works pretty well!
++
Calling this "Compiling" is like calling Python a non-toy programming language.
Do you laugh when you go in small tangents?
Super interesting. But boy you move quickly through all this. It's really hard to follow at times.
This is almost exactly the same video as the one by Qdrant. Weird.
Guess which one came out first... 🫣 It's super weird indeed
Dude you mentioned weviate twice and didn’t even mention what the retriever does. If you want to do promotion, dedicate a segment to it. It will come across more professional and sincere without confusing viewers.
i'm getting a headache by the zooming-in and then skipping across the page.
Hi Connor, came here from Twitter. Thanks for the explanation of DSPy. I found the video hard to follow and thought to leave you some feedback. Please don't do any of this zooming in and out. Let the viewer look where they want to look. It's absolutely distracting. Don't include so many different examples. Curate a view things you really would like to show and then spend more time on these. There was a lot of rambling, repeating things, speeding through and changing things on the fly, adding to the confusion. If you boil it down, I think the information I got from watching this 50 min. could have been transported in one of these "DSPy in 1 Minute" videos. Multiple times you did go on this PyTorch/DeepLearning vs. DSPy comparison which makes absolutely no sense. Hope you can improve your presentation style. Looking forward to more in-depth videos about DSPy.
If you read the DSPy docs or the associated paper, you will see several comparisons between DSPy and PyTorch. The ideas/abstractions in PyTorch such as layers, modules, forward pass, are some of the motivations for the abstractions in DSPy. Quoting from the paper, arxiv.org/pdf/2310.03714.pdf, "Inspired directly by PyTorch abstractions, DSPy modules are used via expressive define-by-run computational graphs". I think that's why Connor was making that connection between Pytorch and DSPy.
nice intro. the zoom effects are not needed, only makes the video harder to watch. i cannot finish this video because of that 👎
That zoom in and out is very distracting/annoying. Sorry.
would be a good or even great vid... if it wasn't for this "music" ugh
zooming in and out is distracting
Constant zoom-in and zoom-out is quite too much - so distracting. Didn’t finish the video 😘
please, no more zooming
What’s with the random giggling outbursts? Super distracting.
I want everyone to repeat after me "Strings are not an API", "Strings are not an API", "Strings are not an API". Defining a 'syntax' as a parseable string is idiocy. Why not just call `eval("answer = x + y")`? You could do that for any and all calls in your program. The reason we don't is because you've denied all the functions of an IDE to your developer by insisting they "just know" the right syntax in that string.
It is the whole idea of parsing (understanding) strings using an llm and interpret the content. The structure of a signature is not that difficult. you write your program with strings. Using copilot (if trained of finetuned with DSPy examples) might overcome your hesitation?
But DSPy uses Natural Language strings so you can easily communicate to your LLM what behavior you want from it(it is a teacher, and your string is the lesson plan), and there are several ways you can do it. You are using them as initial prompts for it to generate useful prompts to train or finetune a different model for a specific task. You can pass them as inputfield/outputfield parameters, as a docstring etc. It works for this task. Also, you don't need to worry about the IDE. You write a class to perform a specific task, inside a pipeline, you don't need to re-use the class because it is generalizable, you then put it in your pipeline and maybe pass a parameter or two too the pipline, the output is hopefully a new LLM that does the task you trained it to do with much lower inference costs if you use something like GPT3.5 turbo to fine tune a T5 model.
The delivery is awesome, but just a bit of feedback- the constant zooming is distracting and makes the content hard to follow.