Exploring DSPy, a More Robust and Systematic Approach to Prompt Engineering

Isn't it Time to Transcend the Dark Art of Prompt Engineering?
In my humble opinion, standard practice for prompt engineering is more like alchemy than engineering. The hard coded, long text scripts are more akin to incantations than computer programming. Let's explore DSPy, a framework for more systematic and robust prompt engineering.
For a deeper dive on this topic, have a look at our blog post that complements this RUclips content: / 5e50ee216853
The Google Colab notebook to accompany this video: colab.research.google.com/dri...
Stay updated on AI, Data Science, and Large Language Models by following me on Medium: / johnadeojo
Seeking to develop AI? Let's chat: bit.ly/brainqub3-discovery-call
Keen on mastering LLM development? Get on our course waiting list: www.data-centric-solutions.co...

[@ak8376]

Thank you for this video, this is one of the best video on DSPy!

[@joao.morossini]

Very high quality content! You were spot on when you compared prompt engineering to alchemy hahaha. Great stuff :)

[@Data-Centric]

Glad you liked it!

[@SciumoInc]

Well done.

[@Data-Centric]

Thanks

[@powerspeed1295]

Hi , thank you for the video . Super interesting. I still have a doubt about the real usage of this framework, hopefully you could answer. both you and the paper show quite "straightforward" questions: what I mean is that you are actually passing some questions to the LLM that should rely on its own "memory" to answer your question or to some reasoning capabilities.
But what about a task-specific usage , like " extract the most 3 important points from these text and create a song with that " . How could this task be handled with this framework ? More over , how to help the framework with a first suggestion or how to add constraints about the result?
So in general , it seems to me this framework has not been properly designed for a "task usage" but more for a "question-answer usage" even if the purpose for to create a general framework for the prompt part.

[@Data-Centric]

Thanks for the feedback! To extend the framework to your use case, you could start by constructing a dataset that illustrates the task, such as extracting three key points. The strength of DSPy lies in its ability to self-improve through user feedback and adapt models by recompiling prompts based on collected data. While DSPy shows promise, further demonstrations of its capabilities are necessary to fully realize its potential.

[@powerspeed1295]

@@Data-Centric Thanks. I have very few examples and I would like to avoid to use it in the prompt itself. Moreover in my current solution I did not use any examples and the result is quite good.
So in genereal I see no point to use examples in my case but more to use the right prompt to make sure it uses the rules and constraints I provide. Do you have any suggestion in mind for that ?

[@MTProduto]

Can I configure DSPy to work with GPT Builder?

[@Data-Centric]

I haven't looked into it.