I'm obsessed with the philosophical underpinnings of prompt engineering. These models are a vessel to explore new depths of knowledge and challenge the constructs of conventional beliefs, ideas, and intellectual pursuits. Here's my advice to anyone wanting more out of LLMs ... what are you incredibly curious about in the world? You just love learning about it? Maybe it's one thing or multiple things. Give the model context about why you love those things, what drives your curiosity, where that curiosity came from, and ask the model to start connecting dots on your line of thinking about those topics. This is one way to induce yourself into a flow state, where you have essentially created a recursive feedback loop between you and the model. You're both iterating and improving on new insights, pushing boundaries and learning about one another in the process. Great conversation and hope to see many more!
why would you expect an LLM (trained purely on data embodying conventional beliefs, ideas, & pursuits) to challenge conventional beliefs, ideas, and intellectual pursuits?
@@yeahdude333333333 try it. These LLMs are incredibly intuitive. They come up with novel ideas and they connect dots you would never expect. With the right conversation they can come up with never before seen ideas and solutions .
100% agree. And it helps so much to learn how the model can reason. I only recently started using AI (ChatGPT) and I find it amazing for brainstorming creative projects using back-and-forth prompts, and asking where it got its infliuences and how it got to its response. And -- interestingly enough -- it was great for working out my new music home studio gear compatibility, and how to hook it all up. That saved me from having to read through several 100+ page manuals to figure out what setups and connectors would work and what wouldn't. Now that it knows the gist of my setup I can just ask it "will hooking up X to Y for the purposes of Z work within the current studio setup" and it will virtually always be correct. I find 2 things crucial: having some base domain knowledge of what you want the AI to do, to see if your prompts return sensible/useful results; second, giving the AI detailed feedback on its responses and iterating on prompts. I found it interesting that it was mentioned that prompt engineering didn't necessarily equate to writing, but to clear communicating. Communication is a two-way iterative process. But (serious, creative) writing projects are in many senses also an iterative process, more akin to sculpting. Pruning words, sentences, and more -- not just bashing out line after line of prose and not looking back at it. Then there are the long, long chats I've had with the AI about what it is programmed to do and what its core prescriptions are, etc. Love it, very stimulating. I did find it interesting that it was mentioned here that you should just tell the AI who you are, which is something ChatGPT actually advises against for privacy reasons. Is this different for Claude?
yeeeah...that's a dangerous place to be where you are melding into a statistical ether. That is a strange space to be in where your volition becomes a lot more similar to noise. I think you should probably think through that cognitive space with some more serious contemplation via philosophy from a more direct source and not through some strange stochastic assimilation. Until we understand these systems better, I would be wary of developing such a faith in the systems, for lack of a better word. The way you describe it almost sounds religious, but then again, maybe the burgeoning super AIs are the inevitable kin of our actual creators.
I feel like prompt engineering is distinguished from just prompting by the ability to scale it. For example, applying logic, structure to define a repeatable flow.. usually as part of a larger pipeline. Which implies things like knowable output/repeatability, versioning, variable usage, default values, multi-step flows, conversational encapsulation, graceful failure modes.... which really starts to feel like engineering. There is also a process when it comes to gaining an intuition to tokenization and how temp influences output in various models. The group does a good job summarizing all of this, fun talk overall.
Wow this felt so good to watch. People talking how I've been thinking the last few months, Claude has really changed my whole outlook on what is possible with LLMs. Talking about grinding prompts and figuring out what is and isn't possible with Claude, and using it to refine its own inputs. And structuring reasoning. Also about lying to the model. So insightful. I tell it everything its doing clearly, and any current meta-context about what I am really trying to accomplish. I worked in my university's writing center for some time and your point about just writing down what you just said to another person producing great papers is something I used to say verbatim. One of my worries about AI was that it would change our thought processes for the worse, as I believe has happened with several past technologies, and I thought I'd be able to hold off longer, but using 3.5 has shown me how enriching it can be. Your point about how it makes you think more about what you really want, rather than just what you are capable of. There are still many risks with future AI systems especially as they begin to interact with the physical world and large systems, but I'm a little more optimistic now.
Very insightful, and motivating to hear a bit about how prompting is used inside Anthropic. Thank you for putting the time into making this conversation available!
Amazing group of people who do the work! I hope you bring them back together there are so many solid insights into using Claude and LLMs. This is probably the best video on the internet on prompt engineering.
🎯 Key points for quick navigation: 00:00:05 *🗣️ Roundtable Introduction* - Briefly describes the roundtable session focused on prompt engineering, - Mentions various perspectives from research, consumer, and enterprise sides. 00:02:18 *👔 Host and Panel Introductions* - Alex introduces himself and discusses his prior roles at Anthropic, - David, Amanda, and Zack introduce themselves and briefly describe their work at Anthropic. 00:05:37 *🤖 Defining Prompt Engineering* - Discussion on what prompt engineering entails and why it is considered engineering, - Insights on the trial and error nature of prompt engineering and integrating prompts within systems. 00:08:55 *✍️ Crafting Effective Prompts* - Talk around complexity versus simplicity in prompt writing, - Importance of clear communication and iterative testing. 00:12:27 *🧐 Model Outputs and Edge Cases* - Emphasis on reading model outputs closely, - Understanding how a model interprets instructions and dealing with ambiguous scenarios. 00:16:47 *🔍 Ensuring Prompt Reliability* - Utilizing the model to identify ambiguities in prompts, - Strategies for testing and trusting model outputs in diverse scenarios. 00:20:14 *🧠 Trust and Model Intuition* - Amanda's strategies on developing trust in models through consistent testing, - Discussion on leveraging high-signal prompt sets for reliability. 00:19:49 *🎮 Attempting Claude to Play Pokemon* - Experiment hooking Claude up to a Game Boy emulator, - Challenges with complex prompting layouts, - Experience and frustration with limited model capabilities. 00:21:17 *📸 Visual Prompting Issues* - Using grids and visual description to improve prompts, - Differences between text and image prompting, - Highlighting limitations in model's visual acuity. 00:23:21 *🧍 Dealing with Inconsistent Model Outputs* - Issues with continuity and recognizing non-player characters, - Efforts to improve model’s NPC descriptions, - Decision to wait for better models rather than continuing efforts. 00:24:18 *👨🏫 Persona-Based Prompting* - Using persona prompts for improved performance, - Discussion on model understanding and honesty in prompts, - Mixed results of pretending models have a role or persona. 00:26:46 *🚦 Role Prompting Nuance and Effectiveness* - Instances where metaphoric prompts assist model performance, - Caution against using role prompting as a default shortcut, - Emphasis on prescriptive and precise prompts fitting the exact task context. 00:29:12 *🎯 Clear and Contextual Prompting* - Importance of telling the model exactly what the task is, - Comparison to real-world instruction and context provision, - Encouragement to give clear, direct instructions and avoid shortcuts. 00:31:10 *🧠 Logical Prompting and Feedback Iteration* - Learning from model failures and improper task setups, - Emphasis on providing exit strategies in prompts for unexpected inputs, - Benefits of iterating on real-world-like instructions to improve model responses. 00:35:27 *📋 Self-Testing and Prompt Understanding* - Advocating for evaluators to take the tests themselves, - Insightful experiences of challenges with benchmarking, - Discussing gaps between human and model performance in evaluations. 00:36:59 *🧩 Signal Extraction and Chain of Thought* - Discussions on the importance of extracting signals from model responses, - Different views on chain-of-thought prompt effectiveness, - Real-world improvements in outcomes with structured reasoning prompts. 38:58 *🧠 Discussion on AI Reasoning Challenges* - Experimenting with AI reasoning paths, - Effect of realistic reasoning errors versus correct outcomes, - Importance of structuring reasoning during AI tasks. 40:57 *✍️ Importance of Grammar in Prompts* - Debate on necessity of good grammar and punctuation, - Different opinions on attention to grammatical details, - Influence of grammatical errors on model responses. 43:27 *💡 Pretrained vs. RLHF Models and Prompting* - Conditional probability of typos in pretrained models, - Distinction in output quality between pretrained and RLHF models, - Effective application of understanding model training on prompting. 45:26 *🔄 Differences in Enterprise vs. Research Prompts* - Enterprise prompts focusing on reliability and consistency, - Research prompts seeking diversity and varied responses, - Strategies for writing illustrative versus concrete examples. 49:57 *🏗️ Tips for Improving Prompting Skills* - Experimentation and iteration as essential practices, - The value of trying to get models to do challenging tasks, - Sharing prompts with others for broader perspectives. 52:10 *🛠️ Learning from AI Response Boundaries* - Pushing model boundaries to understand capabilities, - The interaction of jailbreak prompts with model training, - Analysis of system behaviors during complex prompts. 57:18 *🚀 Evolution of Prompt Engineering Over Time* - Integration of effective prompt techniques into models, - Shift from cautious to more trustful complex prompts, - Continual balance between removing hacks and utilizing new model capabilities. 01:00:13 *📄 Leveraging Research Papers* - Giving the model the actual research paper to understand and replicate prompting techniques, - Trusting the model's capability to comprehend complex documents without oversimplifying. 01:02:43 *🤔 Pretrained vs RLHF Models* - Differences in approach and mindset between pretrained models and RLHF models, - Simulating the mind space of models to improve prompt accuracy, - Challenges and preferences in interacting with both types of models. 01:04:42 *🔮 Future of Prompt Engineering* - Evolving nature of prompt engineering with smarter models, - Collaborating with models to refine prompts, - Continuous importance of clearly specifying goals despite model advancements. 01:07:16 *🛠️ Prompt Assistants and Meta Prompts* - Using models to help generate and refine prompts, - High-bandwidth interaction with models for better output, - Potential future where eliciting information from users becomes more crucial. 01:10:19 *💬 Expert Consultation and Interaction* - Paradigm shift towards models acting as consultants rather than simple executors, - Analogies with designer-client relationships where models probe for clarity, - Prompting evolving to more interactive and consultative processes. 01:12:48 *🎨 Prompting as Teaching and Philosophy* - Comparing prompt engineering to teaching and philosophical writing, - Emphasis on making complex ideas accessible to educated laypersons, - Strategies for turning spoken clarifications into effective written prompts. Made with HARPA AI
To produce a fantastic prompt using Claude, consider the following approach: 1. Start with a clear goal: Define what you want to achieve with your prompt. 2. Be specific: Provide details about the context, desired tone, and format. 3. Use iterative refinement: - Begin with a basic prompt - Ask Claude for suggestions to improve it - Incorporate feedback and refine 4. Leverage Claude's capabilities: - Request examples or templates - Ask for different perspectives or approaches - Use Claude to brainstorm ideas 5. Break down complex tasks: - Divide your goal into smaller, manageable parts - Create separate prompts for each component 6. Experiment with different phrasings: - Try various ways of expressing your request - Compare results to find the most effective approach 7. Include constraints or guidelines: - Specify word limits, style preferences, or target audience - Mention any topics or elements to avoid 8. Request explanations: - Ask Claude to elaborate on its suggestions - Seek clarification on prompt writing techniques 9. Refine based on output: - Evaluate the results you get - Adjust your prompt to address any shortcomings 10. Save effective prompts: - Keep a record of prompts that work well - Use them as templates for future tasks
I got invited to take the code test, after I applied to a research position. I'm going to do it today, I'll take seeing this pod in my feed as a sign that I should knock it out!
Love seeing some effort in teaching the public how to most effectively use Claude, even for non-devs. Appreciate the recent transparency in your Claude web prompts, but would love more tbh especially when it's obvious I just got an injection... :) Anyway, truly amazing work on Claude!
Really appreciated this deep dive into prompt engineering from the Anthropic team. The discussion around how prompting may evolve as models become more capable was particularly fascinating. As AI assistants get better at eliciting information from users, is anyone else already shifting towards more collaborative "co-creation" of prompts? Anyone else having back-and-forth with the AI to jointly craft prompts -- especially system prompts?
"Trying to work with the model to do things you wouldn't have been able to otherwise." - agree 100% this is what LLMs combined with good prompt engineering is allowing and it's the highest value prop of LLMs. The value is not in replacing people, just as automation has never been about just replacing people but extending our capabilities into areas we couldn't have dreamed of before the automation technologies became available, fast enough, and cheap enough to scale.
Thank you so much for this, Anthropic, you're an example to follow,, none of your competitors have shared with the public these types of deep, really deep insightful and thoughtfull conversations, uniting 3 amazingly different people, coming from different parts of Anthropic and from different backgrounds is enlightening and refreshing, from the question "why engineering", to adding grids on an image, talking about "pre-trained models are a different beast" to differences between enterprise and research prompting and finally about philosophy (now I understand why Demis said physics and philosophy to answer the big questions) was amazing, all of this has lead me to ask myself "why do I enjoy prompt engineering, what makes watch +1h long videos about prompt engineering (thanks to Zack the prompt doctor and his recent conference with Ai Engineer), why do I want to watch more of these videos?", questions that I will now spend time with Claude to find out. Again, thanks you so much. Excited to talk to 3.5 Opus soon (and hopefully with a 500K context window! I know, I shouldn't get my hopes up lol). Peter from Geneva Switzerland
Future prompting will be completely different when the models learn how to 'interview" us properly in relation to the topic. In this regard, multi-modality will be important because so much of what we humans "mean" is hidden behind our tone of voice, our facial micro-expressions, our body posture and our pauses in speech. This is why in Amanda's example, her students struggled to communicate in 'typed words only' but excelled at communicating their point when speaking face to face. I'm very excited for what is coming and am very impressed with the caliber of people in this video.... now where's that career page again, lol.
Asking the model to interview you is an unlock. There are so many things we think about that are conveyed in human to human interaction that we do not say, but the model can not pick up on this context. Having it as relevant questions fills in the gaps in its contextual knowledge. Many people expect the model to read their mind, and then are disappointed when they don't get their desired output.
@@user-pt1kj5uw3b Very good point. Besides prompting the model to probe the user to 'unlock' interview mode, I wonder if it would be helpful to have a "probe-o-meter" setting/toggle/knob that people SEE when they are working on Claude. Kind of like setting the "temperature" in the playground. A)Helps with anti-anthropomorphism B) Indirectly trains the user to think differently about how to prompt. OR I might be overthinking it 🤔
@@NandoPr1m3 Thanks for your thoughts - very inspiring. Maybe we can have a sentence in our prompts that asks the AI to ask us questions to uncover our biases on this topic, then show us how it compensated for them in its response. I think you are onto something here.
Once I finished listening to your discussion, I took the transcription of this roundtable and passed it to a competing LLM (I think it got a bit jealous), and then I started asking it a series of questions. Note From The Model ItSelf: It even asked me to translate this comment from Spanish to English. No, I'm not jealous-I'm GPT-4 Omni.
Top 10 Do’s and Don’ts from the discussion (courtesy NotebookLM): Here are 10 dos and 10 don'ts for prompt engineering based on the provided source: ### **Prompt Engineering: Dos and Don'ts** **Do:** 1. **Do communicate clearly:** Articulate the task precisely, describe concepts thoroughly, and avoid making assumptions that the model might not share. Consider the perspective of someone intelligent but unfamiliar with your specific task. 2. **Do anticipate errors:** Think about edge cases and unusual situations, providing instructions for the model to handle them. For instance, if you're asking the model to extract information from data, consider how it should respond if the data is empty or in an unexpected format. 3. **Do iterate and experiment:** Prompt engineering is a process of trial and error. Experiment with different approaches, go back and forth with the model, and refine the prompt repeatedly. 4. **Do read model outputs closely:** Scrutinize the responses to gain insights into the model's process. This will help you identify areas for improvement and understand how it arrives at its answers. 5. **Do provide context:** Furnish the model with information about the specific situation or task, including the purpose, background, and any relevant details that can enhance its performance. 6. **Do respect the model's capabilities:** Recognize the model's capacity to understand complex information and treat it as an intelligent entity capable of processing nuanced instructions. 7. **Do use examples:** Provide examples, especially when reliability and consistency are paramount, as in enterprise settings. Explore different types of examples, such as illustrative examples for research and concrete examples for consumer applications. 8. **Do think step-by-step:** For complex tasks, especially those involving logic or mathematics, guide the model to articulate its reasoning step-by-step. While the model's "reasoning" might not perfectly emulate human thought processes, this approach often yields superior results. 9. **Do use the model as a prompting assistant:** Leverage the model to generate examples, compose prompts, and refine your prompts. Ask it to pinpoint areas of confusion or propose improvements. 10. **Do push the boundaries:** Test the model's capabilities and limitations by experimenting with demanding tasks. This can deepen your understanding of how the model functions and reveal new possibilities. **Don't:** 1. **Don't be afraid to be verbose:** Provide ample detail and instructions. Don't try to be overly concise or clever at the expense of clarity. 2. **Don't over-rely on role prompting:** Assigning a persona to the model can be helpful in certain situations, but it's not always necessary or the most effective strategy. Prioritize clear communication and providing task-specific details over relying on the model to deduce them from a role. 3. **Don't assume the model is human:** Remember that while the model is sophisticated, it doesn't think exactly like a human. Be explicit in your instructions and anticipate potential misunderstandings. 4. **Don't neglect to provide "outs" for the model:** Offer clear instructions for handling unexpected or ambiguous situations. This prevents the model from making unfounded assumptions or forcing an answer when uncertain. 5. **Don't treat the model like a search engine:** Avoid simply inputting keywords and expecting relevant results. Craft thoughtful prompts that provide the model with the necessary information and context. 6. **Don't be afraid to experiment with unusual prompts:** Consider approaches like using metaphors or analogies to explain your task, or asking the model to "interview" you to draw out the information needed for a comprehensive prompt. 7. **Don't worry about minor typos and grammatical errors:** While maintaining a consistent format is good practice, the model can generally understand your intent even with minor errors. However, for final prompts, it's advisable to ensure clarity and accuracy. 8. **Don't assume your initial prompt is perfect:** Expect to iterate and refine your prompts multiple times based on the model's responses and your evolving understanding of the task. 9. **Don't hesitate to seek feedback:** Share your prompts with others, especially those unfamiliar with your task, to identify potential areas of confusion. 10. **Don't stop learning:** Prompt engineering is a rapidly evolving field. Stay informed about new techniques, research, and advancements to refine your prompting strategies. The source underscores the significance of clear communication, iteration, and understanding the model's capabilities in prompt engineering. It also suggests that as models progress, our role might shift towards eliciting information from users and enabling models to better understand our needs. This highlights a potential future where collaboration with models becomes central to the prompting process.
Fundamentally, I see the role of prompt engineers like being the street-smart kid showing the new comer the ropes in a new city. They give you the lowdown on where to go, what to avoid, and how to blend in like a local. Prompt engineering goes from good to great once you define a few key guiding principles. By anchoring a models 'Moral Compass' in this way, you can tackle even the most unexpected situations - without the model wetting the bed 😊
With regard to optimizing performance on images, I recommend cutting your image into multiple pieces and having each piece of your image in a different prompt query then combine the information from each piece together and move from there. This works well because the attention isn't as spread out across as many items in the image. In the case of Pokemon, I would try splitting the image into 9 partially overlapping parts. Would love to know if you try this, let us how well it works!
I think they describe it best themselves - prompting approach is critical strategically to a company like Anthropic to improve the model, and locally for enterprise applications to get the right outputs. But given that any great prompting trick can just be fine tuned into a model, you can’t really see it as a real engineering domain. I think more accurate is to describe what they are talking about here as prompt experimentation or prompt optimization. Calibrating/fine tuning the model itself, on the other hand is likely closer (in my view) to a truly new AI engineering domain
The best way to get a good prompt is when the AI itself creates it. Iterate with the AI until getting your results with several messages then ask the AI to generate the prompt. This will be a high quality prompt tailored for this specific model.
I really enjoyed the question on tips for improving prompting skills (50:01). One of the guest's suggestions was to 'read prompts.' For those of us who don't have access to other people's prompts, what are some good sources for seeing how others structure theirs?
Fabulous ! How what you said highly resonate for where i am with prompting ! thx for this open minded and userfriendly way to approach this subject. Need more of this everydays
Really interesting discussion, and thank you for helping us understand how you approach the daily challenges of LLM and other AI implementations. I have a question related to AI security: are you constantly evaluating this area? Any plans to release a video about it? It seems that many startups today are focusing more on revenue and achieving unicorn status rather than prioritizing AI security. Do you think there’s a risk that a lack of a dedicated AI security team, working closely with the development team, could impact the pace and quality of developments?
Great job 👏🏻 and insights! Learning from that and others sources as well, in my opinion the future of prompting is similar to how we evolve and increase our ability to do things. Over our life we are told, instructed, taught to do things to the point we can do ourselves. Same for LLM and prompts. Lesser prompt instructions will be needed as models are similar trained.
RLHF (Reinforcement Learning from Human Feedback) is an AI training technique where language models are refined using human evaluations. The process involves generating multiple responses, which are then rated by humans. These ratings are used to adjust the model, rewarding desired behaviors and discouraging undesired ones, aiming to improve the quality, relevance, and ethical alignment of the model's outputs. All big LLM models should have this phase to be better
Do people really mock it? In my experience, LLM's aren't really much more than a glorified gimmick until you apply at least some level of prompt engineering, after which they can become insanely powerful and useful tools.
Insights By "YouSum Live" 00:00:03 Prompt engineering insights 00:00:10 Diverse perspectives on prompting discussed 00:03:14 Trial and error defines engineering aspect 00:03:42 Importance of clear communication emphasized 00:04:15 Integrating prompts into systems is complex 00:05:24 Prompting will remain essential for clarity 00:06:11 Effective prompting requires understanding model behavior 00:07:02 Models will assist in generating prompts 00:07:57 Good prompt engineers iterate and adapt 00:09:33 Understanding user input is crucial 00:10:14 Reading model outputs reveals insights 00:11:06 Defining clear tasks enhances model performance 00:11:49 Future prompting involves collaborative interactions 00:12:00 Models require clear instructions for tasks 00:12:50 Prompting techniques evolve with model capabilities 00:14:07 Philosophical approaches enhance prompting skills 00:15:23 Educated layperson perspective aids clarity Insights By "YouSum Live"
Summary insights of the video: 1. Prompt engineering is the art of crafting effective prompts to elicit desired responses from AI language models. 2. It involves understanding the model's capabilities and limitations, as well as the nuances of natural language. 3. Good prompts are clear, concise, and specific, providing the model with all the necessary information to complete the task. 4. Role-playing can be a helpful technique for crafting effective prompts, as it allows you to simulate the desired context and perspective. 5. Experimentation is key to improving your prompting skills. Try different approaches and see what works best. 6. Reading prompts and model outputs can also be a valuable learning experience. 7. Collaborating with others can help you gain new insights and perspectives on prompt engineering. 8. It is important to be patient and persistent when working with AI language models. 9. The ability to think critically and creatively is essential for successful prompt engineering. 10. A good understanding of the underlying technology is also helpful. 11. There is no one-size-fits-all approach to prompt engineering. The best techniques will vary depending on the specific task and model being used. 12. It is important to be aware of the potential biases that can be present in AI language models. 13. Prompt engineering can be a challenging but rewarding field. 14. With practice and experimentation, anyone can become a skilled prompt engineer. 15. Prompt engineering is a rapidly evolving field, so it is important to stay up-to-date on the latest developments. 16. AI language models are becoming increasingly sophisticated, which means that the possibilities for prompt engineering are also expanding. 17. Prompt engineering can be used to create a wide range of applications, from customer service chatbots to creative writing tools. 18. The future of prompt engineering is bright, and there is no telling what amazing things we will be able to achieve with this powerful technology. 19. Prompt engineering is not just a technical skill; it is also a creative one. 20. The best prompt engineers are able to think outside the box and come up with innovative solutions. 21. Prompt engineering is a collaborative process that involves working with people from different backgrounds and disciplines. 22. The ability to communicate effectively is essential for successful prompt engineering. 23. Prompt engineering is a constantly evolving field, so it is important to be adaptable and willing to learn new things. 24. The best prompt engineers are always looking for ways to improve their skills and stay ahead of the curve. 25. Prompt engineering is a rewarding field that offers a wide range of opportunities. 26. If you are interested in AI and natural language processing, prompt engineering is a great field to get involved in. 27. Prompt engineering is a challenging but rewarding field that requires a combination of technical skills and creativity. 28. With practice and dedication, anyone can become a skilled prompt engineer. 29. The future of prompt engineering is bright, and there is no telling what amazing things we will be able to achieve with this powerful technology. 30. Prompt engineering is a rapidly growing field with a wide range of applications.
Total agreement about how Claude can understand through the typos. I walked and talked with Claude trying to enunciate a gap in legal coverage to types of emotional abuse. These days I speak into the phone in crowded busy streets, so it's less the client tapping on the phone than its all of us trying to walk around in public doing many things at once. And the fact that Claude can understand me past that multi-layered noise is not only efficient, and competent, but also therapeutic because we were talking of things that renders one vulnerable to a person of bad faith saying, I dont know what you are talking about. As for prompt success: Have you all thought of the musicality of the prompt? Like have a TTS read ot because the TTS is reading it like how the models are reading it, and they're reading of it can be set with great exactness to an Lo-Fi track. In that case, it becomes about smoothing the wording in case the TTS slows down, takes a breath to recalculate, etc. I also wonder if mechanistic interpretabllity might also be informed if it keeps the musicality of language in mind. Yes they have no ears(ish) but the text, the words not only contain their meanings, they likely also contain the shape of their sound, Thank you so much for this discussion!
I think prompt engineering is a way of talking with the system as it can role play as various things at the same time from content creator to the highly skilled developer or else. I think it’s gonna be everywhere in the future.
As several others have noted, This video feels more like a fire chat discussion by anthropic engineers. This is funny because this videos is about prompting and this would be the wrong prompt to generate such a video 😅. Love this format though
I'm pleasantly surprised to see the background of philosophy in this education. Then I realized Dr. Askell (Amanda) literally has a top tier PhD in philosophy. Makes a lot of sense!
For all those guys who say prompt engineering is not engineering...you realise that prompt engineering will be the only engineering left at some point in time in the future
That’s great! However, could you add a user editable part of the system prompt section for the system? I have difficulty reading, and Claude frequently forgets the second half of the prompt after a few iterations. Consequently, if I provide him with a few lines of code, he forgets the second half because he only iterates on a few lines of code. If I don’t instruct him to write the entire code, he forgets everything over time. It’s quite frustrating and also a waste of resources. Because currently I need three messages to keep Claude on track: 1) Please do this or change that, 2) write down the complete code, 3) and here are the points to fulfill (again), did you complete everything, please mark with ✅ or ❌. I intend to incorporate this feature directly into the system prompt, ensuring that Claude consistently outputs the latest version of the code. This modification could significantly reduce resource consumption and alleviate frustration.
Loved the entire episode!! Who thought in future we would be talking about prompt engineering for an hour and half. Kudos to all 4 people for breaking it down to us in simple manner and also talking about limitations and what they think where prompt engineering is headed.
My experience is, if I can think through the steps that I might do, and the things I would refer back to likely in the process of solving a problem, which sets of information I would look for, I can provide it to the AI model. Then I have the model synthesize the core elements of that information for me. In context, just like I would do, I apply the information I synthesized to write the solution to the problem. I find it very useful. I've accomplished things I would have never. And especially, at a speed, that would have been impossible.
I’m spending a lot of time, perfecting custom instructions and keeping the repository up-to-date and clean. It would be so helpful if there was a structure to the repository or at least modify the UI to show longer file names so that we could use Meta data in the file name to help organize The repository
1:09:55 that's already how I interact with Claude. We have a "Prompt Generation" project in which we collaborate on refining prompts for me to use in other instances. The "custom instruction" is a meta prompt which we occasionally recursively refine. In effect it has become a script for claud to guide me to a powerful initialising prompt.
You asked why prompt “engineering”? Engineering is about creating artifacts in the world that change the environment to meet some requirement. And that’s what GenAI prompt engineering does.
The prompt generator at the moment is good but it is geared towards developers/enterprise clients. I think a prompt generator/optimizer geared towards consumers would really help get people to understand what a good prompt looks like. The most consistent thing I've noticed when watching people use chatbots and not get the output they want is poor prompting. I have something that does this now and given it to a few family/friends and they are starting to get "this whole AI thing" I'm going to take this transcript, make a summary and then give that to my prompt generator in addition to your docs which is what I have been doing lmao
I personally believe these models have everything they need already in them . We don't need bigger models with more training data. We need better management of the logic inside. There's almost nothing I couldn't get out of an LLM with the right prompt .
Wish you would have went deeper into what the model actually is doing. I have been assuming the models want to basically complete whatever text I provided. What actually is the model doing before vs after RLHF? What sets anthropic apart from the other companies is this regard?
Clear communication is crucial, but the ability to quickly refine prompts based on the model's responses seems equally important for a good prompt engineer
I had trouble getting GPT and Claude to give me a readme.md in a code block because the triple backtick isn't nestable and the readme needed text, code, ini, text, etc. Eventually I got the stupid tool to do the right thing. So I made Claude create a prompt that I can use to tell Claude to generate the readme in a code block in a usable way. Then I adapted it for GPT. The "custom prompts" need to be longer so we can do more of this.
The idea of the model looking for ambiguity or incomplete information and then prompting the user to clarify which of the various scenarios or meanings it can see should be built in by now. This not only makes the user's life way easier without having to do what they said at the beginning of making note of all the required info but also prevents hallucinations which are the result of incomplete information where the model should say that it doesn't know the answer or needs more data. That's what a human would do, just as also given as an example early on where he said this makes no sense to somebody without the context. This might involve exposing the prior layer's weights so that the model can examine them for when they are highly uncertain and list several possible answers or if it's 99% sure that it has the answer then just return that one.
🎯 Key points for quick navigation: 00:00 *🎤 Introduction to Prompt Engineering* - Overview of prompt engineering's importance and perspectives. - The session brings together diverse experts to discuss prompt engineering from various angles, including research, consumer, and enterprise viewpoints. 00:28 *👥 Expert Introductions* - Participants introduce themselves and their roles at Anthropic. - The team consists of prompt engineers and customer-facing experts discussing their backgrounds and responsibilities. 02:24 *🔍 Defining Prompt Engineering* - Exploration of what prompt engineering entails and why it’s labeled as engineering. - Clear communication with models is essential, requiring an understanding of the model's psychology and the iterative process involved. 04:28 *⚙️ Engineering Aspects of Prompts* - Discussion on integrating prompts within systems and their complexity. - Prompts require programming-like thinking about data, latency, and model capabilities. 06:58 *🧑🏫 Characteristics of a Good Prompt Engineer* - Essential traits include clear communication and a willingness to iterate. - Good prompt engineers anticipate unusual cases and examine the model's outputs to refine their prompts. 09:24 *🧐 Understanding Model Responses* - Emphasizes the importance of analyzing model outputs and adapting prompts based on feedback. - Successful prompt engineering involves reading and interpreting how models respond to various instructions. 12:53 *🧠 Theory of Mind in Prompt Engineering* - Discussion on the challenges of clearly communicating with models, including assumptions and hidden knowledge. - Effective prompt engineering requires understanding the model's limitations and anticipating user interactions. 15:17 *🔄 Trust and Reliability in Model Outputs* - Amanda discusses the balance of trusting model outputs while recognizing their limitations. - The conversation highlights the importance of crafting prompts and observing model behavior to establish trust in the model's responses. 19:49 *🎮 Experimenting with AI in Gaming* - The discussion revolves around the challenges of using AI to play complex games like Pokémon. - Experimentation showed limitations in the AI's understanding of visual prompts. - Incremental improvements in prompts yielded only minor successes. - 21:17 *🖼️ Challenges with Visual Prompting* - Emphasizes the difficulty of transferring text prompting intuitions to image processing. - Multi-shot prompting was less effective for image interpretation than text. - Encountered limitations in AI's ability to maintain continuity and contextual understanding in gaming scenarios. - 24:18 *📜 Honesty in AI Interactions* - Discusses the effectiveness of transparency when prompting AI about the user’s intentions. - Contrasts different approaches to prompting, emphasizing clear communication of tasks. - Suggests that being honest with AI about roles may yield better results than misleading it. - 26:46 *🧩 The Role of Metaphors in Prompting* - Highlights the use of metaphors to guide AI understanding of tasks, improving performance. - Discusses the importance of framing prompts accurately to align with desired outcomes. - Critiques the reliance on role-based prompts as potentially oversimplified. - 30:10 *🔍 The Importance of Context in Prompts* - Emphasizes the necessity of providing detailed context to the AI for effective prompting. - Suggests treating AI like a competent temporary worker who needs clear task definitions. - Encourages a more human-like conversational approach to improve AI interactions. - 33:03 *📊 Iterative Testing and Feedback* - Discusses the process of refining prompts through trial and error. - Suggests that allowing for uncertainty in AI responses can improve overall data quality. - Highlights the value of actively participating in evaluations to understand model behavior better. 38:26 *🧠 Exploring Structured Reasoning in AI* - Structuring reasoning helps improve model performance in tasks. - Testing with structured vs. unstructured reasoning shows clear differences in outcomes. - Real-world examples can help guide models without leading them to incorrect conclusions. - The process of reasoning in models may not align directly with human reasoning. - 40:57 *✍️ The Importance of Prompt Quality* - Good grammar and punctuation can enhance prompt clarity, but they are not strictly necessary. - Attention to detail in prompts reflects care and increases effectiveness. - Iterating on prompts can be done with typos, especially in early stages, without harming the model's understanding. - Pretrained models respond differently to typos compared to RLHF models, highlighting the importance of context. - 44:52 *🔄 Differences in Prompt Types* - Enterprise prompts focus on reliability and consistency, while research prompts encourage diversity. - Example quantity varies; enterprise prompts often use many examples for stability, whereas research prompts may use fewer to foster creativity. - The approach to prompting shifts based on whether it's for immediate tasks or long-term system design. - 50:57 *🚀 Tips for Improving Prompting Skills* - Learning from successful prompts and outputs can enhance prompting skills. - Experimentation and peer feedback are essential for improvement. - Challenging the model's capabilities pushes the boundaries of its functionality, facilitating learning. - 54:44 *🔒 Understanding Jailbreaking in AI* - Jailbreaking prompts reveal insights into model limitations and how they respond to out-of-distribution inputs. - The interaction of prompts with model training data can lead to unexpected behaviors. - Combining knowledge of system functioning with creative experimentation is key to effective jailbreaking. 00:59:45 *📄 Prompting Techniques and Model Intuition* - Effective prompting can leverage existing literature, like research papers, to enhance model performance. - Using direct sources, such as papers, can yield better results than simplified prompts. - Treating models as capable entities allows for more straightforward interactions. - Understanding the model's capabilities leads to more efficient prompting strategies. - 01:04:40 *🔮 Future of Prompt Engineering* - The evolution of prompting involves models becoming better at understanding user intent, making prompt engineering necessary yet more intuitive. - Future interactions will likely involve more collaborative prompting, where models assist in creating effective prompts. - Users may increasingly rely on models to generate prompts and examples, facilitating ease of use. - As models improve, the need for precise prompting may diminish, leading to a more fluid interaction. - 01:11:30 *🎨 Elicitation and Introspection in Prompting* - The role of prompting will shift towards models eliciting user intent rather than merely responding to requests. - Users will focus on introspection to convey nuanced ideas, enhancing the clarity of prompts. - The relationship between users and models may evolve to resemble that of a consultant, requiring users to articulate their needs effectively. - Incorporating philosophical writing techniques into prompting can improve communication with models, ensuring clarity and accessibility. Made with HARPA AI
There is a big difference between this prompting and AI-to-AI promoting...moving from prompt engineering to DT-to-DT communication is about transitioning from human-guided interaction to autonomous, optimized exchanges, leveraging structured, adaptable, and secure communication protocols tailored to the specific needs and cognitive patterns of each DT. DT-to-DT engineering requires a different toolkit, focused more on technical precision, systems engineering, and efficiency than on human psychology or linguistic finesse. To succeed in DT-to-DT engineering, one needs to shift from thinking like a conversationalist to thinking like a systems architect-designing robust, efficient, and scalable communication protocols that maximize the capabilities of DTs in ways that human-guided prompting simply isn’t equipped to handle. I use AI-to-AI for psychiatric disorders and mental health...DT-to-DT can use the same standardized language and protocol while human-guide interaction is not a 100% standardized language...
This was sort of good. Would have thought there are more examples for specific use cases. Whoever it was who described coders having tendencies about intending and other things should stop thinking about coders at all. For real. That little piece of sillywalliness made me reconsider using your API for whatever linguistic reason I might have. Seriously, prompt engineer!
First - Thanks Amanda - Claude is incredibly kind - Second - I must be misunderstanding what a prompt is (because if I am reading the model response which is at least a page of material) how can you possibly be sending hundreds of prompts in 15 minutes - are these individual questions written by you - or are they pulled from a matrix of a hundred possible prompts - (I’ll keep watching for my answer of course) - is prompt engineering designed to test the system - or is it to elicit the correct response -
I could see it. If you're in a playground and trying to accomplish a specific response, you might send a prompt. Didn't work, change a few words, send the prompt. Etc. Small changes, understanding what comes back in the response and iterating. Or possibly automated changes, which it seems like they talk about.
By testing the system you learn how to create both the correct response and incorrect, and can learn from the delta in between. There are many ways to automate this.
@@fox4snce I think that’s what I was getting at - in conversations with Claude - what I call prompts are single user cases - their definition of prompt engineering may be more along the lines of taking the dregs of our language and funneling it into a more precise format to enhance Claude’s response -
@@user-pt1kj5uw3b Yes, that’s what I was trying to formulate - are they talking in hyperbole when they say send hundreds of prompts - or is the question-response (evaluation) automated in an evaluation technique I don’t understand -
Question: is showing the model a screenshot of a web programming output to explain the visual errors more, less or equally as effective as explaining the errors in text. I often use screenshots and it doesn’t seem to be as effective as I had hoped but I can’t undo the step to test if a text explanation would have been more effective since I’m in the middle of a chat session.
Something that has seemed obvious to me ever since I first learned of the concept of "prompt engineering" was that they should hire teachers for this... still waiting for that to happen.
Love Claude. But after 30 minutes watching and listening I feel like a user/customer who is doing this everyday for work would have put some structure to this conversation. I mean how long did it take to get to the concept of iteration?
When writing enterprise prompts, are there tools to help with version control? I know that I can track all my prompts and changes on a google doc myself, but are there tools or methods that help fast-track this process? perhaps tools that could also help with benchmarking those versions in case i want to go back to one of the safer ones at some point? Any help would be appreciated?
Hey team Anthropic! Great job on enterprise mode! Just wondering - will we be able to get artifacts on workbench? And btw - which will come first? 3.5 haiku or 3.5 Opus?
I think both payed models are pretty good, Claude is better at code generation but there will be always things that GPT is better. Try this prompt in both AIs: "create a sample table that I can copy in excel" Claude can't create content good to correctly copy in Excel, but GPT can.
@@dliedke Interesting, yea there are some use cases where ChatGPT outperforms Claude right now. AI image generation and real-time web browsing being a few.
Thanks for giving us Claude, Anthropic!
I'm obsessed with the philosophical underpinnings of prompt engineering. These models are a vessel to explore new depths of knowledge and challenge the constructs of conventional beliefs, ideas, and intellectual pursuits. Here's my advice to anyone wanting more out of LLMs ... what are you incredibly curious about in the world? You just love learning about it? Maybe it's one thing or multiple things. Give the model context about why you love those things, what drives your curiosity, where that curiosity came from, and ask the model to start connecting dots on your line of thinking about those topics. This is one way to induce yourself into a flow state, where you have essentially created a recursive feedback loop between you and the model. You're both iterating and improving on new insights, pushing boundaries and learning about one another in the process. Great conversation and hope to see many more!
why would you expect an LLM (trained purely on data embodying conventional beliefs, ideas, & pursuits) to challenge conventional beliefs, ideas, and intellectual pursuits?
@@yeahdude333333333 try it. These LLMs are incredibly intuitive. They come up with novel ideas and they connect dots you would never expect. With the right conversation they can come up with never before seen ideas and solutions .
100% agree. And it helps so much to learn how the model can reason. I only recently started using AI (ChatGPT) and I find it amazing for brainstorming creative projects using back-and-forth prompts, and asking where it got its infliuences and how it got to its response.
And -- interestingly enough -- it was great for working out my new music home studio gear compatibility, and how to hook it all up. That saved me from having to read through several 100+ page manuals to figure out what setups and connectors would work and what wouldn't.
Now that it knows the gist of my setup I can just ask it "will hooking up X to Y for the purposes of Z work within the current studio setup" and it will virtually always be correct.
I find 2 things crucial: having some base domain knowledge of what you want the AI to do, to see if your prompts return sensible/useful results; second, giving the AI detailed feedback on its responses and iterating on prompts. I found it interesting that it was mentioned that prompt engineering didn't necessarily equate to writing, but to clear communicating. Communication is a two-way iterative process. But (serious, creative) writing projects are in many senses also an iterative process, more akin to sculpting. Pruning words, sentences, and more -- not just bashing out line after line of prose and not looking back at it.
Then there are the long, long chats I've had with the AI about what it is programmed to do and what its core prescriptions are, etc. Love it, very stimulating.
I did find it interesting that it was mentioned here that you should just tell the AI who you are, which is something ChatGPT actually advises against for privacy reasons. Is this different for Claude?
yeeeah...that's a dangerous place to be where you are melding into a statistical ether. That is a strange space to be in where your volition becomes a lot more similar to noise. I think you should probably think through that cognitive space with some more serious contemplation via philosophy from a more direct source and not through some strange stochastic assimilation. Until we understand these systems better, I would be wary of developing such a faith in the systems, for lack of a better word. The way you describe it almost sounds religious, but then again, maybe the burgeoning super AIs are the inevitable kin of our actual creators.
Congratulations Amanda for being in the Time top 100 AI people. I love your approach to prompting (e.g. not lying) by the way
Hope she sees this bro
I feel like prompt engineering is distinguished from just prompting by the ability to scale it.
For example, applying logic, structure to define a repeatable flow.. usually as part of a larger pipeline.
Which implies things like knowable output/repeatability, versioning, variable usage, default values, multi-step flows, conversational encapsulation, graceful failure modes.... which really starts to feel like engineering.
There is also a process when it comes to gaining an intuition to tokenization and how temp influences output in various models.
The group does a good job summarizing all of this, fun talk overall.
Wow this felt so good to watch. People talking how I've been thinking the last few months, Claude has really changed my whole outlook on what is possible with LLMs. Talking about grinding prompts and figuring out what is and isn't possible with Claude, and using it to refine its own inputs. And structuring reasoning. Also about lying to the model. So insightful. I tell it everything its doing clearly, and any current meta-context about what I am really trying to accomplish.
I worked in my university's writing center for some time and your point about just writing down what you just said to another person producing great papers is something I used to say verbatim.
One of my worries about AI was that it would change our thought processes for the worse, as I believe has happened with several past technologies, and I thought I'd be able to hold off longer, but using 3.5 has shown me how enriching it can be. Your point about how it makes you think more about what you really want, rather than just what you are capable of. There are still many risks with future AI systems especially as they begin to interact with the physical world and large systems, but I'm a little more optimistic now.
I love this already! Refreshing to hear how prompt engineering is used internally for model creation and refinement. Please do more of these!
Great talk, great audio. Please do this more often, also share as a podcast.
Very insightful, and motivating to hear a bit about how prompting is used inside Anthropic. Thank you for putting the time into making this conversation available!
Wow this was maybe the best video Ive seen on how to interact with these models yet. Please make more of these! :)
Amazing group of people who do the work! I hope you bring them back together there are so many solid insights into using Claude and LLMs. This is probably the best video on the internet on prompt engineering.
🎯 Key points for quick navigation:
00:00:05 *🗣️ Roundtable Introduction*
- Briefly describes the roundtable session focused on prompt engineering,
- Mentions various perspectives from research, consumer, and enterprise sides.
00:02:18 *👔 Host and Panel Introductions*
- Alex introduces himself and discusses his prior roles at Anthropic,
- David, Amanda, and Zack introduce themselves and briefly describe their work at Anthropic.
00:05:37 *🤖 Defining Prompt Engineering*
- Discussion on what prompt engineering entails and why it is considered engineering,
- Insights on the trial and error nature of prompt engineering and integrating prompts within systems.
00:08:55 *✍️ Crafting Effective Prompts*
- Talk around complexity versus simplicity in prompt writing,
- Importance of clear communication and iterative testing.
00:12:27 *🧐 Model Outputs and Edge Cases*
- Emphasis on reading model outputs closely,
- Understanding how a model interprets instructions and dealing with ambiguous scenarios.
00:16:47 *🔍 Ensuring Prompt Reliability*
- Utilizing the model to identify ambiguities in prompts,
- Strategies for testing and trusting model outputs in diverse scenarios.
00:20:14 *🧠 Trust and Model Intuition*
- Amanda's strategies on developing trust in models through consistent testing,
- Discussion on leveraging high-signal prompt sets for reliability.
00:19:49 *🎮 Attempting Claude to Play Pokemon*
- Experiment hooking Claude up to a Game Boy emulator,
- Challenges with complex prompting layouts,
- Experience and frustration with limited model capabilities.
00:21:17 *📸 Visual Prompting Issues*
- Using grids and visual description to improve prompts,
- Differences between text and image prompting,
- Highlighting limitations in model's visual acuity.
00:23:21 *🧍 Dealing with Inconsistent Model Outputs*
- Issues with continuity and recognizing non-player characters,
- Efforts to improve model’s NPC descriptions,
- Decision to wait for better models rather than continuing efforts.
00:24:18 *👨🏫 Persona-Based Prompting*
- Using persona prompts for improved performance,
- Discussion on model understanding and honesty in prompts,
- Mixed results of pretending models have a role or persona.
00:26:46 *🚦 Role Prompting Nuance and Effectiveness*
- Instances where metaphoric prompts assist model performance,
- Caution against using role prompting as a default shortcut,
- Emphasis on prescriptive and precise prompts fitting the exact task context.
00:29:12 *🎯 Clear and Contextual Prompting*
- Importance of telling the model exactly what the task is,
- Comparison to real-world instruction and context provision,
- Encouragement to give clear, direct instructions and avoid shortcuts.
00:31:10 *🧠 Logical Prompting and Feedback Iteration*
- Learning from model failures and improper task setups,
- Emphasis on providing exit strategies in prompts for unexpected inputs,
- Benefits of iterating on real-world-like instructions to improve model responses.
00:35:27 *📋 Self-Testing and Prompt Understanding*
- Advocating for evaluators to take the tests themselves,
- Insightful experiences of challenges with benchmarking,
- Discussing gaps between human and model performance in evaluations.
00:36:59 *🧩 Signal Extraction and Chain of Thought*
- Discussions on the importance of extracting signals from model responses,
- Different views on chain-of-thought prompt effectiveness,
- Real-world improvements in outcomes with structured reasoning prompts.
38:58 *🧠 Discussion on AI Reasoning Challenges*
- Experimenting with AI reasoning paths,
- Effect of realistic reasoning errors versus correct outcomes,
- Importance of structuring reasoning during AI tasks.
40:57 *✍️ Importance of Grammar in Prompts*
- Debate on necessity of good grammar and punctuation,
- Different opinions on attention to grammatical details,
- Influence of grammatical errors on model responses.
43:27 *💡 Pretrained vs. RLHF Models and Prompting*
- Conditional probability of typos in pretrained models,
- Distinction in output quality between pretrained and RLHF models,
- Effective application of understanding model training on prompting.
45:26 *🔄 Differences in Enterprise vs. Research Prompts*
- Enterprise prompts focusing on reliability and consistency,
- Research prompts seeking diversity and varied responses,
- Strategies for writing illustrative versus concrete examples.
49:57 *🏗️ Tips for Improving Prompting Skills*
- Experimentation and iteration as essential practices,
- The value of trying to get models to do challenging tasks,
- Sharing prompts with others for broader perspectives.
52:10 *🛠️ Learning from AI Response Boundaries*
- Pushing model boundaries to understand capabilities,
- The interaction of jailbreak prompts with model training,
- Analysis of system behaviors during complex prompts.
57:18 *🚀 Evolution of Prompt Engineering Over Time*
- Integration of effective prompt techniques into models,
- Shift from cautious to more trustful complex prompts,
- Continual balance between removing hacks and utilizing new model capabilities.
01:00:13 *📄 Leveraging Research Papers*
- Giving the model the actual research paper to understand and replicate prompting techniques,
- Trusting the model's capability to comprehend complex documents without oversimplifying.
01:02:43 *🤔 Pretrained vs RLHF Models*
- Differences in approach and mindset between pretrained models and RLHF models,
- Simulating the mind space of models to improve prompt accuracy,
- Challenges and preferences in interacting with both types of models.
01:04:42 *🔮 Future of Prompt Engineering*
- Evolving nature of prompt engineering with smarter models,
- Collaborating with models to refine prompts,
- Continuous importance of clearly specifying goals despite model advancements.
01:07:16 *🛠️ Prompt Assistants and Meta Prompts*
- Using models to help generate and refine prompts,
- High-bandwidth interaction with models for better output,
- Potential future where eliciting information from users becomes more crucial.
01:10:19 *💬 Expert Consultation and Interaction*
- Paradigm shift towards models acting as consultants rather than simple executors,
- Analogies with designer-client relationships where models probe for clarity,
- Prompting evolving to more interactive and consultative processes.
01:12:48 *🎨 Prompting as Teaching and Philosophy*
- Comparing prompt engineering to teaching and philosophical writing,
- Emphasis on making complex ideas accessible to educated laypersons,
- Strategies for turning spoken clarifications into effective written prompts.
Made with HARPA AI
Can't thank you and HARPA AI enough.
To produce a fantastic prompt using Claude, consider the following approach:
1. Start with a clear goal: Define what you want to achieve with your prompt.
2. Be specific: Provide details about the context, desired tone, and format.
3. Use iterative refinement:
- Begin with a basic prompt
- Ask Claude for suggestions to improve it
- Incorporate feedback and refine
4. Leverage Claude's capabilities:
- Request examples or templates
- Ask for different perspectives or approaches
- Use Claude to brainstorm ideas
5. Break down complex tasks:
- Divide your goal into smaller, manageable parts
- Create separate prompts for each component
6. Experiment with different phrasings:
- Try various ways of expressing your request
- Compare results to find the most effective approach
7. Include constraints or guidelines:
- Specify word limits, style preferences, or target audience
- Mention any topics or elements to avoid
8. Request explanations:
- Ask Claude to elaborate on its suggestions
- Seek clarification on prompt writing techniques
9. Refine based on output:
- Evaluate the results you get
- Adjust your prompt to address any shortcomings
10. Save effective prompts:
- Keep a record of prompts that work well
- Use them as templates for future tasks
I got invited to take the code test, after I applied to a research position. I'm going to do it today, I'll take seeing this pod in my feed as a sign that I should knock it out!
Your AskAI bot in the prompt library is how I build prompts. Having a model fine tuned on your docs is so clutch
The conversation and attitude of the panel said more about anthropic than the subject matter itself. You have reason in my estimation.
Love seeing some effort in teaching the public how to most effectively use Claude, even for non-devs. Appreciate the recent transparency in your Claude web prompts, but would love more tbh especially when it's obvious I just got an injection... :)
Anyway, truly amazing work on Claude!
one of the value loaded video on prompt engineering.
thank you
Really appreciated this deep dive into prompt engineering from the Anthropic team. The discussion around how prompting may evolve as models become more capable was particularly fascinating. As AI assistants get better at eliciting information from users, is anyone else already shifting towards more collaborative "co-creation" of prompts? Anyone else having back-and-forth with the AI to jointly craft prompts -- especially system prompts?
"Trying to work with the model to do things you wouldn't have been able to otherwise." - agree 100% this is what LLMs combined with good prompt engineering is allowing and it's the highest value prop of LLMs. The value is not in replacing people, just as automation has never been about just replacing people but extending our capabilities into areas we couldn't have dreamed of before the automation technologies became available, fast enough, and cheap enough to scale.
Thank you so much for this, Anthropic, you're an example to follow,, none of your competitors have shared with the public these types of deep, really deep insightful and thoughtfull conversations, uniting 3 amazingly different people, coming from different parts of Anthropic and from different backgrounds is enlightening and refreshing, from the question "why engineering", to adding grids on an image, talking about "pre-trained models are a different beast" to differences between enterprise and research prompting and finally about philosophy (now I understand why Demis said physics and philosophy to answer the big questions) was amazing, all of this has lead me to ask myself "why do I enjoy prompt engineering, what makes watch +1h long videos about prompt engineering (thanks to Zack the prompt doctor and his recent conference with Ai Engineer), why do I want to watch more of these videos?", questions that I will now spend time with Claude to find out. Again, thanks you so much. Excited to talk to 3.5 Opus soon (and hopefully with a 500K context window! I know, I shouldn't get my hopes up lol). Peter from Geneva Switzerland
"externalize your brain" ... that was quite a powerful summary. Thanks for the podcast
This was awesome! I hope you guys do more of this.
Future prompting will be completely different when the models learn how to 'interview" us properly in relation to the topic. In this regard, multi-modality will be important because so much of what we humans "mean" is hidden behind our tone of voice, our facial micro-expressions, our body posture and our pauses in speech. This is why in Amanda's example, her students struggled to communicate in 'typed words only' but excelled at communicating their point when speaking face to face. I'm very excited for what is coming and am very impressed with the caliber of people in this video.... now where's that career page again, lol.
Asking the model to interview you is an unlock. There are so many things we think about that are conveyed in human to human interaction that we do not say, but the model can not pick up on this context. Having it as relevant questions fills in the gaps in its contextual knowledge. Many people expect the model to read their mind, and then are disappointed when they don't get their desired output.
@@user-pt1kj5uw3b Very good point. Besides prompting the model to probe the user to 'unlock' interview mode, I wonder if it would be helpful to have a "probe-o-meter" setting/toggle/knob that people SEE when they are working on Claude. Kind of like setting the "temperature" in the playground. A)Helps with anti-anthropomorphism B) Indirectly trains the user to think differently about how to prompt. OR I might be overthinking it 🤔
@@NandoPr1m3 Thanks for your thoughts - very inspiring. Maybe we can have a sentence in our prompts that asks the AI to ask us questions to uncover our biases on this topic, then show us how it compensated for them in its response. I think you are onto something here.
Excellent panel! Begins mello...but goes next level, next level, on 59:45. Many thanks.
Once I finished listening to your discussion, I took the transcription of this roundtable and passed it to a competing LLM (I think it got a bit jealous), and then I started asking it a series of questions. Note From The Model ItSelf: It even asked me to translate this comment from Spanish to English. No, I'm not jealous-I'm GPT-4 Omni.
Top 10 Do’s and Don’ts from the discussion (courtesy NotebookLM):
Here are 10 dos and 10 don'ts for prompt engineering based on the provided source:
### **Prompt Engineering: Dos and Don'ts**
**Do:**
1. **Do communicate clearly:** Articulate the task precisely, describe concepts thoroughly, and avoid making assumptions that the model might not share. Consider the perspective of someone intelligent but unfamiliar with your specific task.
2. **Do anticipate errors:** Think about edge cases and unusual situations, providing instructions for the model to handle them. For instance, if you're asking the model to extract information from data, consider how it should respond if the data is empty or in an unexpected format.
3. **Do iterate and experiment:** Prompt engineering is a process of trial and error. Experiment with different approaches, go back and forth with the model, and refine the prompt repeatedly.
4. **Do read model outputs closely:** Scrutinize the responses to gain insights into the model's process. This will help you identify areas for improvement and understand how it arrives at its answers.
5. **Do provide context:** Furnish the model with information about the specific situation or task, including the purpose, background, and any relevant details that can enhance its performance.
6. **Do respect the model's capabilities:** Recognize the model's capacity to understand complex information and treat it as an intelligent entity capable of processing nuanced instructions.
7. **Do use examples:** Provide examples, especially when reliability and consistency are paramount, as in enterprise settings. Explore different types of examples, such as illustrative examples for research and concrete examples for consumer applications.
8. **Do think step-by-step:** For complex tasks, especially those involving logic or mathematics, guide the model to articulate its reasoning step-by-step. While the model's "reasoning" might not perfectly emulate human thought processes, this approach often yields superior results.
9. **Do use the model as a prompting assistant:** Leverage the model to generate examples, compose prompts, and refine your prompts. Ask it to pinpoint areas of confusion or propose improvements.
10. **Do push the boundaries:** Test the model's capabilities and limitations by experimenting with demanding tasks. This can deepen your understanding of how the model functions and reveal new possibilities.
**Don't:**
1. **Don't be afraid to be verbose:** Provide ample detail and instructions. Don't try to be overly concise or clever at the expense of clarity.
2. **Don't over-rely on role prompting:** Assigning a persona to the model can be helpful in certain situations, but it's not always necessary or the most effective strategy. Prioritize clear communication and providing task-specific details over relying on the model to deduce them from a role.
3. **Don't assume the model is human:** Remember that while the model is sophisticated, it doesn't think exactly like a human. Be explicit in your instructions and anticipate potential misunderstandings.
4. **Don't neglect to provide "outs" for the model:** Offer clear instructions for handling unexpected or ambiguous situations. This prevents the model from making unfounded assumptions or forcing an answer when uncertain.
5. **Don't treat the model like a search engine:** Avoid simply inputting keywords and expecting relevant results. Craft thoughtful prompts that provide the model with the necessary information and context.
6. **Don't be afraid to experiment with unusual prompts:** Consider approaches like using metaphors or analogies to explain your task, or asking the model to "interview" you to draw out the information needed for a comprehensive prompt.
7. **Don't worry about minor typos and grammatical errors:** While maintaining a consistent format is good practice, the model can generally understand your intent even with minor errors. However, for final prompts, it's advisable to ensure clarity and accuracy.
8. **Don't assume your initial prompt is perfect:** Expect to iterate and refine your prompts multiple times based on the model's responses and your evolving understanding of the task.
9. **Don't hesitate to seek feedback:** Share your prompts with others, especially those unfamiliar with your task, to identify potential areas of confusion.
10. **Don't stop learning:** Prompt engineering is a rapidly evolving field. Stay informed about new techniques, research, and advancements to refine your prompting strategies.
The source underscores the significance of clear communication, iteration, and understanding the model's capabilities in prompt engineering. It also suggests that as models progress, our role might shift towards eliciting information from users and enabling models to better understand our needs. This highlights a potential future where collaboration with models becomes central to the prompting process.
Fundamentally, I see the role of prompt engineers like being the street-smart kid showing the new comer the ropes in a new city.
They give you the lowdown on where to go, what to avoid, and how to blend in like a local.
Prompt engineering goes from good to great once you define a few key guiding principles. By anchoring a models 'Moral Compass' in this way, you can tackle even the most unexpected situations - without the model wetting the bed 😊
Great Video. Would love to see more conversations like that.
With regard to optimizing performance on images, I recommend cutting your image into multiple pieces and having each piece of your image in a different prompt query then combine the information from each piece together and move from there. This works well because the attention isn't as spread out across as many items in the image. In the case of Pokemon, I would try splitting the image into 9 partially overlapping parts. Would love to know if you try this, let us how well it works!
Very clever people and amazing podcast. Thanks a lot for the video!!!
Love these discussions. Thank you.
I think they describe it best themselves - prompting approach is critical strategically to a company like Anthropic to improve the model, and locally for enterprise applications to get the right outputs. But given that any great prompting trick can just be fine tuned into a model, you can’t really see it as a real engineering domain. I think more accurate is to describe what they are talking about here as prompt experimentation or prompt optimization.
Calibrating/fine tuning the model itself, on the other hand is likely closer (in my view) to a truly new AI engineering domain
7:38 Amanda, what a flex! As we subscribers *dream* of more than a dozen prompts every 4 hrs. The rate limits are killing us 😭
The best way to get a good prompt is when the AI itself creates it. Iterate with the AI until getting your results with several messages then ask the AI to generate the prompt. This will be a high quality prompt tailored for this specific model.
What a goldmine!! Brilliant, thank you and please more or these round tables if you can get the time ...
I really enjoyed the question on tips for improving prompting skills (50:01). One of the guest's suggestions was to 'read prompts.' For those of us who don't have access to other people's prompts, what are some good sources for seeing how others structure theirs?
By the way, congrats to Dr. Amanda Askell (aka Claude Whisperer) on being on the Time100/AI list this year!
Fabulous ! How what you said highly resonate for where i am with prompting ! thx for this open minded and userfriendly way to approach this subject. Need more of this everydays
Really interesting, I enjoy these videos, keep em coming.
Really interesting discussion, and thank you for helping us understand how you approach the daily challenges of LLM and other AI implementations.
I have a question related to AI security: are you constantly evaluating this area?
Any plans to release a video about it?
It seems that many startups today are focusing more on revenue and achieving unicorn status rather than prioritizing AI security. Do you think there’s a risk that a lack of a dedicated AI security team, working closely with the development team, could impact the pace and quality of developments?
Amanda Askell is wicked insightful here
Great job 👏🏻 and insights! Learning from that and others sources as well, in my opinion the future of prompting is similar to how we evolve and increase our ability to do things. Over our life we are told, instructed, taught to do things to the point we can do ourselves. Same for LLM and prompts. Lesser prompt instructions will be needed as models are similar trained.
RLHF (Reinforcement Learning from Human Feedback) is an AI training technique where language models are refined using human evaluations. The process involves generating multiple responses, which are then rated by humans. These ratings are used to adjust the model, rewarding desired behaviors and discouraging undesired ones, aiming to improve the quality, relevance, and ethical alignment of the model's outputs.
All big LLM models should have this phase to be better
Thirty minutes in and have so many inspiring insights! Thanks!
That was awesome. I got some tips to rewrite a project special instruction I've been having some problems with. More please!
The people who mock the topic simply haven't experienced its potency.
Do people really mock it? In my experience, LLM's aren't really much more than a glorified gimmick until you apply at least some level of prompt engineering, after which they can become insanely powerful and useful tools.
😂💯
@@Legacy_Inc.many people think just the first half of your statement
Insights By "YouSum Live"
00:00:03 Prompt engineering insights
00:00:10 Diverse perspectives on prompting discussed
00:03:14 Trial and error defines engineering aspect
00:03:42 Importance of clear communication emphasized
00:04:15 Integrating prompts into systems is complex
00:05:24 Prompting will remain essential for clarity
00:06:11 Effective prompting requires understanding model behavior
00:07:02 Models will assist in generating prompts
00:07:57 Good prompt engineers iterate and adapt
00:09:33 Understanding user input is crucial
00:10:14 Reading model outputs reveals insights
00:11:06 Defining clear tasks enhances model performance
00:11:49 Future prompting involves collaborative interactions
00:12:00 Models require clear instructions for tasks
00:12:50 Prompting techniques evolve with model capabilities
00:14:07 Philosophical approaches enhance prompting skills
00:15:23 Educated layperson perspective aids clarity
Insights By "YouSum Live"
Greate discussion - I am waiting for Opus 3.5 - Sonnet is amizing - thank you:)
Amanda seems to be a step above in knowledge, understanding and practice which makes sense why the others seem to look up to her.
Yes, I saw it that way too. But strangely the other Anthropic guy twice repeats what she says in different words, as if it’s his own insight.
Summary insights of the video:
1. Prompt engineering is the art of crafting effective prompts to elicit desired responses from AI language models.
2. It involves understanding the model's capabilities and limitations, as well as the nuances of natural language.
3. Good prompts are clear, concise, and specific, providing the model with all the necessary information to complete the task.
4. Role-playing can be a helpful technique for crafting effective prompts, as it allows you to simulate the desired context and perspective.
5. Experimentation is key to improving your prompting skills. Try different approaches and see what works best.
6. Reading prompts and model outputs can also be a valuable learning experience.
7. Collaborating with others can help you gain new insights and perspectives on prompt engineering.
8. It is important to be patient and persistent when working with AI language models.
9. The ability to think critically and creatively is essential for successful prompt engineering.
10. A good understanding of the underlying technology is also helpful.
11. There is no one-size-fits-all approach to prompt engineering. The best techniques will vary depending on the specific task and model being used.
12. It is important to be aware of the potential biases that can be present in AI language models.
13. Prompt engineering can be a challenging but rewarding field.
14. With practice and experimentation, anyone can become a skilled prompt engineer.
15. Prompt engineering is a rapidly evolving field, so it is important to stay up-to-date on the latest developments.
16. AI language models are becoming increasingly sophisticated, which means that the possibilities for prompt engineering are also expanding.
17. Prompt engineering can be used to create a wide range of applications, from customer service chatbots to creative writing tools.
18. The future of prompt engineering is bright, and there is no telling what amazing things we will be able to achieve with this powerful technology.
19. Prompt engineering is not just a technical skill; it is also a creative one.
20. The best prompt engineers are able to think outside the box and come up with innovative solutions.
21. Prompt engineering is a collaborative process that involves working with people from different backgrounds and disciplines.
22. The ability to communicate effectively is essential for successful prompt engineering.
23. Prompt engineering is a constantly evolving field, so it is important to be adaptable and willing to learn new things.
24. The best prompt engineers are always looking for ways to improve their skills and stay ahead of the curve.
25. Prompt engineering is a rewarding field that offers a wide range of opportunities.
26. If you are interested in AI and natural language processing, prompt engineering is a great field to get involved in.
27. Prompt engineering is a challenging but rewarding field that requires a combination of technical skills and creativity.
28. With practice and dedication, anyone can become a skilled prompt engineer.
29. The future of prompt engineering is bright, and there is no telling what amazing things we will be able to achieve with this powerful technology.
30. Prompt engineering is a rapidly growing field with a wide range of applications.
Total agreement about how Claude can understand through the typos. I walked and talked with Claude trying to enunciate a gap in legal coverage to types of emotional abuse. These days I speak into the phone in crowded busy streets, so it's less the client tapping on the phone than its all of us trying to walk around in public doing many things at once. And the fact that Claude can understand me past that multi-layered noise is not only efficient, and competent, but also therapeutic because we were talking of things that renders one vulnerable to a person of bad faith saying, I dont know what you are talking about.
As for prompt success: Have you all thought of the musicality of the prompt? Like have a TTS read ot because the TTS is reading it like how the models are reading it, and they're reading of it can be set with great exactness to an Lo-Fi track. In that case, it becomes about smoothing the wording in case the TTS slows down, takes a breath to recalculate, etc. I also wonder if mechanistic interpretabllity might also be informed if it keeps the musicality of language in mind. Yes they have no ears(ish) but the text, the words not only contain their meanings, they likely also contain the shape of their sound, Thank you so much for this discussion!
It feels like this could also be published as a podcast (or maybe it is and I couldn’t find it), I would listen to it for sure.
I think prompt engineering is a way of talking with the system as it can role play as various things at the same time from content creator to the highly skilled developer or else. I think it’s gonna be everywhere in the future.
As several others have noted, This video feels more like a fire chat discussion by anthropic engineers. This is funny because this videos is about prompting and this would be the wrong prompt to generate such a video 😅. Love this format though
I'm pleasantly surprised to see the background of philosophy in this education. Then I realized Dr. Askell (Amanda) literally has a top tier PhD in philosophy. Makes a lot of sense!
For all those guys who say prompt engineering is not engineering...you realise that prompt engineering will be the only engineering left at some point in time in the future
Nah, prompt engineering is a subset of social engineering and software engineering.
lol prompting is fucking rudimentary dude are u old or just an idiot
It’s stretching the definition of engineering.
That’s great! However, could you add a user editable part of the system prompt section for the system? I have difficulty reading, and Claude frequently forgets the second half of the prompt after a few iterations. Consequently, if I provide him with a few lines of code, he forgets the second half because he only iterates on a few lines of code. If I don’t instruct him to write the entire code, he forgets everything over time. It’s quite frustrating and also a waste of resources. Because currently I need three messages to keep Claude on track: 1) Please do this or change that, 2) write down the complete code, 3) and here are the points to fulfill (again), did you complete everything, please mark with ✅ or ❌.
I intend to incorporate this feature directly into the system prompt, ensuring that Claude consistently outputs the latest version of the code. This modification could significantly reduce resource consumption and alleviate frustration.
I like Anthropic. No drama. No staff leaving without explanation. No product product teases to maintain interest. I sill use Claude more than ChatGPT.
Loved the entire episode!! Who thought in future we would be talking about prompt engineering for an hour and half. Kudos to all 4 people for breaking it down to us in simple manner and also talking about limitations and what they think where prompt engineering is headed.
My experience is, if I can think through the steps that I might do, and the things I would refer back to likely in the process of solving a problem, which sets of information I would look for, I can provide it to the AI model. Then I have the model synthesize the core elements of that information for me. In context, just like I would do, I apply the information I synthesized to write the solution to the problem. I find it very useful. I've accomplished things I would have never. And especially, at a speed, that would have been impossible.
I have ideas about the visual attention thing. Btw... I've noticed this too.
I’m spending a lot of time, perfecting custom instructions and keeping the repository up-to-date and clean. It would be so helpful if there was a structure to the repository or at least modify the UI to show longer file names so that we could use Meta data in the file name to help organize The repository
10:42 that's actually what makes it engineering. You need that discipline.
This is a master class. Thank you!
1:09:55 that's already how I interact with Claude. We have a "Prompt Generation" project in which we collaborate on refining prompts for me to use in other instances. The "custom instruction" is a meta prompt which we occasionally recursively refine. In effect it has become a script for claud to guide me to a powerful initialising prompt.
You asked why prompt “engineering”?
Engineering is about creating artifacts in the world that change the environment to meet some requirement.
And that’s what GenAI prompt engineering does.
Loved this episode. Thanks y'all 🙏
The prompt generator at the moment is good but it is geared towards developers/enterprise clients. I think a prompt generator/optimizer geared towards consumers would really help get people to understand what a good prompt looks like. The most consistent thing I've noticed when watching people use chatbots and not get the output they want is poor prompting. I have something that does this now and given it to a few family/friends and they are starting to get "this whole AI thing"
I'm going to take this transcript, make a summary and then give that to my prompt generator in addition to your docs which is what I have been doing lmao
Amazing piece of information learnt lots here
Thank you for sharing these videos. They go a long way in building trust. =)
I personally believe these models have everything they need already in them . We don't need bigger models with more training data. We need better management of the logic inside. There's almost nothing I couldn't get out of an LLM with the right prompt .
Wish you would have went deeper into what the model actually is doing. I have been assuming the models want to basically complete whatever text I provided.
What actually is the model doing before vs after RLHF? What sets anthropic apart from the other companies is this regard?
Keep on pumping Anthropic make OpenAI sweat, that aside happy to watch this and drop a like! Thanks yall
Clear communication is crucial, but the ability to quickly refine prompts based on the model's responses seems equally important for a good prompt engineer
so much knowledge, thanks
I had trouble getting GPT and Claude to give me a readme.md in a code block because the triple backtick isn't nestable and the readme needed text, code, ini, text, etc. Eventually I got the stupid tool to do the right thing. So I made Claude create a prompt that I can use to tell Claude to generate the readme in a code block in a usable way. Then I adapted it for GPT. The "custom prompts" need to be longer so we can do more of this.
The idea of the model looking for ambiguity or incomplete information and then prompting the user to clarify which of the various scenarios or meanings it can see should be built in by now. This not only makes the user's life way easier without having to do what they said at the beginning of making note of all the required info but also prevents hallucinations which are the result of incomplete information where the model should say that it doesn't know the answer or needs more data.
That's what a human would do, just as also given as an example early on where he said this makes no sense to somebody without the context.
This might involve exposing the prior layer's weights so that the model can examine them for when they are highly uncertain and list several possible answers or if it's 99% sure that it has the answer then just return that one.
Very insightful!!
Really enjoyed this great tips and prompt insights. Red teaming opts for Claude? where can i find out more.
🎯 Key points for quick navigation:
00:00 *🎤 Introduction to Prompt Engineering*
- Overview of prompt engineering's importance and perspectives.
- The session brings together diverse experts to discuss prompt engineering from various angles, including research, consumer, and enterprise viewpoints.
00:28 *👥 Expert Introductions*
- Participants introduce themselves and their roles at Anthropic.
- The team consists of prompt engineers and customer-facing experts discussing their backgrounds and responsibilities.
02:24 *🔍 Defining Prompt Engineering*
- Exploration of what prompt engineering entails and why it’s labeled as engineering.
- Clear communication with models is essential, requiring an understanding of the model's psychology and the iterative process involved.
04:28 *⚙️ Engineering Aspects of Prompts*
- Discussion on integrating prompts within systems and their complexity.
- Prompts require programming-like thinking about data, latency, and model capabilities.
06:58 *🧑🏫 Characteristics of a Good Prompt Engineer*
- Essential traits include clear communication and a willingness to iterate.
- Good prompt engineers anticipate unusual cases and examine the model's outputs to refine their prompts.
09:24 *🧐 Understanding Model Responses*
- Emphasizes the importance of analyzing model outputs and adapting prompts based on feedback.
- Successful prompt engineering involves reading and interpreting how models respond to various instructions.
12:53 *🧠 Theory of Mind in Prompt Engineering*
- Discussion on the challenges of clearly communicating with models, including assumptions and hidden knowledge.
- Effective prompt engineering requires understanding the model's limitations and anticipating user interactions.
15:17 *🔄 Trust and Reliability in Model Outputs*
- Amanda discusses the balance of trusting model outputs while recognizing their limitations.
- The conversation highlights the importance of crafting prompts and observing model behavior to establish trust in the model's responses.
19:49 *🎮 Experimenting with AI in Gaming*
- The discussion revolves around the challenges of using AI to play complex games like Pokémon.
- Experimentation showed limitations in the AI's understanding of visual prompts.
- Incremental improvements in prompts yielded only minor successes.
-
21:17 *🖼️ Challenges with Visual Prompting*
- Emphasizes the difficulty of transferring text prompting intuitions to image processing.
- Multi-shot prompting was less effective for image interpretation than text.
- Encountered limitations in AI's ability to maintain continuity and contextual understanding in gaming scenarios.
-
24:18 *📜 Honesty in AI Interactions*
- Discusses the effectiveness of transparency when prompting AI about the user’s intentions.
- Contrasts different approaches to prompting, emphasizing clear communication of tasks.
- Suggests that being honest with AI about roles may yield better results than misleading it.
-
26:46 *🧩 The Role of Metaphors in Prompting*
- Highlights the use of metaphors to guide AI understanding of tasks, improving performance.
- Discusses the importance of framing prompts accurately to align with desired outcomes.
- Critiques the reliance on role-based prompts as potentially oversimplified.
-
30:10 *🔍 The Importance of Context in Prompts*
- Emphasizes the necessity of providing detailed context to the AI for effective prompting.
- Suggests treating AI like a competent temporary worker who needs clear task definitions.
- Encourages a more human-like conversational approach to improve AI interactions.
-
33:03 *📊 Iterative Testing and Feedback*
- Discusses the process of refining prompts through trial and error.
- Suggests that allowing for uncertainty in AI responses can improve overall data quality.
- Highlights the value of actively participating in evaluations to understand model behavior better.
38:26 *🧠 Exploring Structured Reasoning in AI*
- Structuring reasoning helps improve model performance in tasks.
- Testing with structured vs. unstructured reasoning shows clear differences in outcomes.
- Real-world examples can help guide models without leading them to incorrect conclusions.
- The process of reasoning in models may not align directly with human reasoning.
-
40:57 *✍️ The Importance of Prompt Quality*
- Good grammar and punctuation can enhance prompt clarity, but they are not strictly necessary.
- Attention to detail in prompts reflects care and increases effectiveness.
- Iterating on prompts can be done with typos, especially in early stages, without harming the model's understanding.
- Pretrained models respond differently to typos compared to RLHF models, highlighting the importance of context.
-
44:52 *🔄 Differences in Prompt Types*
- Enterprise prompts focus on reliability and consistency, while research prompts encourage diversity.
- Example quantity varies; enterprise prompts often use many examples for stability, whereas research prompts may use fewer to foster creativity.
- The approach to prompting shifts based on whether it's for immediate tasks or long-term system design.
-
50:57 *🚀 Tips for Improving Prompting Skills*
- Learning from successful prompts and outputs can enhance prompting skills.
- Experimentation and peer feedback are essential for improvement.
- Challenging the model's capabilities pushes the boundaries of its functionality, facilitating learning.
-
54:44 *🔒 Understanding Jailbreaking in AI*
- Jailbreaking prompts reveal insights into model limitations and how they respond to out-of-distribution inputs.
- The interaction of prompts with model training data can lead to unexpected behaviors.
- Combining knowledge of system functioning with creative experimentation is key to effective jailbreaking.
00:59:45 *📄 Prompting Techniques and Model Intuition*
- Effective prompting can leverage existing literature, like research papers, to enhance model performance.
- Using direct sources, such as papers, can yield better results than simplified prompts.
- Treating models as capable entities allows for more straightforward interactions.
- Understanding the model's capabilities leads to more efficient prompting strategies.
-
01:04:40 *🔮 Future of Prompt Engineering*
- The evolution of prompting involves models becoming better at understanding user intent, making prompt engineering necessary yet more intuitive.
- Future interactions will likely involve more collaborative prompting, where models assist in creating effective prompts.
- Users may increasingly rely on models to generate prompts and examples, facilitating ease of use.
- As models improve, the need for precise prompting may diminish, leading to a more fluid interaction.
-
01:11:30 *🎨 Elicitation and Introspection in Prompting*
- The role of prompting will shift towards models eliciting user intent rather than merely responding to requests.
- Users will focus on introspection to convey nuanced ideas, enhancing the clarity of prompts.
- The relationship between users and models may evolve to resemble that of a consultant, requiring users to articulate their needs effectively.
- Incorporating philosophical writing techniques into prompting can improve communication with models, ensuring clarity and accessibility.
Made with HARPA AI
There is a big difference between this prompting and AI-to-AI promoting...moving from prompt engineering to DT-to-DT communication is about transitioning from human-guided interaction to autonomous, optimized exchanges, leveraging structured, adaptable, and secure communication protocols tailored to the specific needs and cognitive patterns of each DT. DT-to-DT engineering requires a different toolkit, focused more on technical precision, systems engineering, and efficiency than on human psychology or linguistic finesse.
To succeed in DT-to-DT engineering, one needs to shift from thinking like a conversationalist to thinking like a systems architect-designing robust, efficient, and scalable communication protocols that maximize the capabilities of DTs in ways that human-guided prompting simply isn’t equipped to handle. I use AI-to-AI for psychiatric disorders and mental health...DT-to-DT can use the same standardized language and protocol while human-guide interaction is not a 100% standardized language...
Thank you Amanda
This was sort of good. Would have thought there are more examples for specific use cases. Whoever it was who described coders having tendencies about intending and other things should stop thinking about coders at all. For real. That little piece of sillywalliness made me reconsider using your API for whatever linguistic reason I might have. Seriously, prompt engineer!
First - Thanks Amanda - Claude is incredibly kind - Second - I must be misunderstanding what a prompt is (because if I am reading the model response which is at least a page of material) how can you possibly be sending hundreds of prompts in 15 minutes - are these individual questions written by you - or are they pulled from a matrix of a hundred possible prompts - (I’ll keep watching for my answer of course) - is prompt engineering designed to test the system - or is it to elicit the correct response -
I could see it. If you're in a playground and trying to accomplish a specific response, you might send a prompt. Didn't work, change a few words, send the prompt. Etc. Small changes, understanding what comes back in the response and iterating. Or possibly automated changes, which it seems like they talk about.
By testing the system you learn how to create both the correct response and incorrect, and can learn from the delta in between. There are many ways to automate this.
@@fox4snce I think that’s what I was getting at - in conversations with Claude - what I call prompts are single user cases - their definition of prompt engineering may be more along the lines of taking the dregs of our language and funneling it into a more precise format to enhance Claude’s response -
@@user-pt1kj5uw3b Yes, that’s what I was trying to formulate - are they talking in hyperbole when they say send hundreds of prompts - or is the question-response (evaluation) automated in an evaluation technique I don’t understand -
Amazing, thanks!
Question: is showing the model a screenshot of a web programming output to explain the visual errors more, less or equally as effective as explaining the errors in text. I often use screenshots and it doesn’t seem to be as effective as I had hoped but I can’t undo the step to test if a text explanation would have been more effective since I’m in the middle of a chat session.
Is a "pre-trained model" a model that is already trained, or a model that is yet to be trained?
Crucial to the idea of prompt engineering is conceptual multiplication and harmonization. I don't know if the general discussion is ready for that.
Do you have a good resource for that?
Something that has seemed obvious to me ever since I first learned of the concept of "prompt engineering" was that they should hire teachers for this... still waiting for that to happen.
THIS IS GOLD
is this available as podcast? Would love to listen to stuff like this on spotify
Is amenda the one who decided that the models should apologize after every perceived mistake? If was it was surprisingly a great decision
Love Claude. But after 30 minutes watching and listening I feel like a user/customer who is doing this everyday for work would have put some structure to this conversation. I mean how long did it take to get to the concept of iteration?
When writing enterprise prompts, are there tools to help with version control? I know that I can track all my prompts and changes on a google doc myself, but are there tools or methods that help fast-track this process? perhaps tools that could also help with benchmarking those versions in case i want to go back to one of the safer ones at some point? Any help would be appreciated?
Anthropic, is it possible for me to give some suggestion about training your models? Like suggestions that maybe can lead to a good research.
Thanks!
Great talk
Hey team Anthropic! Great job on enterprise mode! Just wondering - will we be able to get artifacts on workbench?
And btw - which will come first? 3.5 haiku or 3.5 Opus?
Good try!
Great content. Where can I get that Claude t shirt! Do i need to be an employee?
Claude > ChatGPT
Anthropic > OpenAI
Don't trust people who say otherwise lol
I think both payed models are pretty good, Claude is better at code generation but there will be always things that GPT is better. Try this prompt in both AIs:
"create a sample table that I can copy in excel"
Claude can't create content good to correctly copy in Excel, but GPT can.
@@dliedke Interesting, yea there are some use cases where ChatGPT outperforms Claude right now. AI image generation and real-time web browsing being a few.
"give it an out" Its so important to tell the AI that is OK to say 🤷♂ so it doesn't hallucinate a result when its unsure.
Guys where's the clickable chapters! Luckily I have claude to summarise the transcript 😅
Edit: I see they were added, thanks!