Prompt Engineering for Generative AI

Prompt Engineering for Generative AI

by James Phoenix, Mike Taylor
Released May 2024
Publisher(s): O'Reilly Media, Inc.
ISBN: 9781098153434

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Large language models (LLMs) and diffusion models such as ChatGPT and Stable Diffusion have unprecedented potential. Because they have been trained on all the public text and images on the internet, they can make useful contributions to a wide variety of tasks. And with the barrier to entry greatly reduced today, practically any developer can harness LLMs and diffusion models to tackle problems previously unsuitable for automation.

With this book, you'll gain a solid foundation in generative AI, including how to apply these models in practice. When first integrating LLMs and diffusion models into their workflows, most developers struggle to coax reliable enough results from them to use in automated systems. Authors James Phoenix and Mike Taylor show you how a set of principles called prompt engineering can enable you to work effectively with AI.

Learn how to empower AI to work for you. This book explains:

  • The structure of the interaction chain of your program's AI model and the fine-grained steps in between
  • How AI model requests arise from transforming the application problem into a document completion problem in the model training domain
  • The influence of LLM and diffusion model architecture—and how to best interact with it
  • How these principles apply in practice in the domains of natural language processing, text and image generation, and code

Publisher resources

View/Submit Errata

Table of contents

  1. Preface
    1. Software Requirements for This Book
    2. For Mac Users
    3. For Windows Users
    4. Additional Tips
    5. Conventions Used in This Book
    6. Using Code Examples
    7. O’Reilly Online Learning
    8. How to Contact Us
    9. Acknowledgments
  2. 1. The Five Principles of Prompting
    1. Overview of the Five Principles of Prompting
    2. 1. Give Direction
    3. 2. Specify Format
    4. 3. Provide Examples
    5. 4. Evaluate Quality
    6. 5. Divide Labor
    7. Summary
  3. 2. Introduction to Large Language Models for Text Generation
    1. What Are Text Generation Models?
      1. Vector Representations: The Numerical Essence of Language
      2. Transformer Architecture: Orchestrating Contextual Relationships
      3. Probabilistic Text Generation: The Decision Mechanism
    2. Historical Underpinnings: The Rise of Transformer Architectures
    3. OpenAI’s Generative Pre-trained Transformers
      1. GPT-3.5-turbo and ChatGPT
    4. GPT-4
    5. Google’s Gemini
    6. Meta’s LLaMA and Open Source
    7. Leveraging Quantization and LoRA
    8. Mistral
    9. Anthropic: Claude
    10. GPT-4V(ision)
    11. Model Comparison
    12. Summary
  4. 3. Standard Practices for Text Generation with ChatGPT
    1. Generating Lists
    2. Hierarchical List Generation
    3. When To Avoid Using Regular Expressions
    4. Generating JSON
      1. YAML
    5. Filtering YAML Payloads
    6. Handling Invalid Payloads in YAML
    7. Diverse Format Generation with ChatGPT
      1. Mock CSV Data
    8. Explain It Like I’m Five
    9. Universal Translation Through LLMs
    10. Ask For Context
    11. Text Style Unbundling
    12. Identifying the Desired Textual Features
    13. Generating New Content with the Extracted Features
    14. Extracting Specific Textual Features with LLMs
    15. Summarization
    16. Summarizing Given Context Window Limitations
    17. Chunking Text
      1. Benefits of chunking text
      2. Scenarios for Chunking Text
      3. Poor Chunking Example
    18. Chunking Strategies
    19. Sentence Detection using SpaCy
    20. Building a simple chunking algorithm in Python
    21. Sliding Window Chunking
    22. Text Chunking Packages
    23. Text Chunking with Tiktoken
    24. Encodings
      1. Understanding the tokenization of strings
    25. Estimating Token Usage for Chat API Calls
    26. Sentiment Analysis
      1. Techniques for Improving Sentiment Analysis
      2. Limitations and Challenges in Sentiment Analysis
    27. Least To Most
      1. Planning the Architecture
      2. Coding Individual Functions
      3. Adding Tests
      4. Benefits of the Least To Most Technique
      5. Challenges with the Least To Most Technique
    28. Role Prompting
    29. Benefits of Role Prompting
    30. Challenges of Role Prompting
    31. When to use Role Prompting
    32. GPT Prompting Tactics
      1. Avoiding Hallucinations with Reference
      2. Give GPTs “Thinking Time”
      3. The Inner Monologue Tactic
      4. Self-Eval LLM Responses
    33. Classification with LLMs
    34. Building A Classification Model
    35. Majority Vote For Classification
    36. Criteria Evaluation
    37. Meta Prompting
    38. Summary
  5. 4. Advanced Techniques for Text Generation with LangChain
    1. Introduction to LangChain
      1. Environment Setup
    2. Chat Models
    3. Streaming Chat Models
    4. Creating Multiple LLM Generations
    5. Langchain Prompt Templates
    6. LangChain Expression Language (LCEL)
    7. Using PromptTemplate with Chat Models
    8. Output Parsers
    9. LangChain Evals
    10. OpenAI Function Calling
    11. Parallel Function Calling
    12. Function Calling in LangChain
    13. Extracting Data with LangChain
    14. Query Planning
    15. Creating Few-Shot Prompt Templates
      1. Fixed Length Few-Shot Examples
      2. Formatting the Examples
      3. Selecting by Length Few-Shot Examples
    16. Limitations with Few-Shot Examples
    17. Saving and Loading LLM Prompts
    18. Data Connection
    19. Document Loaders
    20. Text Splitters
    21. Text Splitting by Length and Token Size
    22. Text Splitting with Recursive Character Splitting
    23. Task Decomposition
    24. Prompt Chaining
      1. Sequential Chain
      2. Itemgetter and Dictionary Key Extraction
      3. Structuring LCEL Chains
      4. Document Chains
      5. Stuff
      6. Refine
      7. Map Reduce
      8. Map re-rank
    25. Summary
  6. 5. Vector Databases with FAISS and Pinecone
    1. Retrieval Augmented Generation (RAG)
    2. Introducing Embeddings
    3. Document Loading
    4. Memory Retrieval with FAISS
    5. RAG with LangChain
    6. Hosted Vector Databases with Pinecone
    7. Self Querying
    8. Alternative Retrieval Mechanisms
    9. Summary
  7. 6. Autonomous Agents with Memory and Tools
    1. Chain-of-Thought
    2. Agents
      1. Reason and Act (reAct)
      2. Reason and Act Implementation
      3. Using Tools
    3. Using LLMs as an API (OpenAI Functions)
    4. Comparing OpenAI Functions and reAct
      1. Use Cases for OpenAI Functions
      2. reAct
      3. Use Cases for reAct
    5. Agent ToolKits
    6. Customizing Standard Agents
    7. Custom Agents in LCEL
    8. Understanding and Using Memory
      1. Long Term Memory:
      2. Short Term Memory
      3. Short-Term Memory in QA Conversation Agents
    9. Memory in LangChain
      1. Preserving the state
      2. Querying the state
      3. ConversationBufferMemory
    10. Other Popular Memory Types in LangChain
      1. Conversation Buffer Window Memory
      2. Conversation Summary Memory
      3. Conversation Summary Buffer Memory
      4. ConversationToken Buffer Memory
    11. OpenAI Functions Agent with Memory
    12. Advanced Agent Frameworks
      1. Plan and Execute Agents
      2. Tree of Thoughts
    13. Callbacks
      1. Global (Constructor) Callbacks
      2. Request-specific Callbacks
      3. The Verbose Argument
      4. When to Use Which?
      5. Token Counting with LangChain
    14. Summary
  8. 7. Introduction to Diffusion Models for Image Generation
    1. OpenAI DALL-E
    2. Midjourney
    3. Stable Diffusion
    4. Google Gemini
    5. Text to Video
    6. Model Comparison
    7. Summary
  9. 8. Standard Practices for Image Generation with Midjourney
    1. Format Modifiers
    2. Art Style Modifiers
    3. Reverse-Engineering Prompts
    4. Quality Boosters
    5. Negative Prompts
    6. Weighted Terms
    7. Prompting with an Image
    8. Inpainting
    9. Outpainting
    10. Consistent Characters
    11. Prompt Re-writing
    12. Meme Unbundling
    13. Meme Mapping
    14. Prompt Analysis
    15. Summary
  10. 9. Advanced Techniques for Image Generation with Stable Diffusion
    1. Running Stable Diffusion
    2. AUTOMATIC1111 Web User Interface
    3. Img2Img
    4. Upscaling Images
    5. Interrogate CLIP
    6. SD Inpainting and Outpainting
    7. ControlNet
    8. Segment Anything Model (SAM)
    9. Dreambooth Fine-Tuning
    10. Stable Diffusion XL Refiner
    11. Summary
  11. 10. Building AI-powered Applications
    1. AI Blog Writing
    2. Topic Research
    3. Expert Interview
    4. Generate Outline
    5. Text Generation
    6. Writing Style
    7. Title Optimization
    8. AI Blog Images
    9. User Interface
    10. Summary
  12. About the Authors

Product information

  • Title: Prompt Engineering for Generative AI
  • Author(s): James Phoenix, Mike Taylor
  • Release date: May 2024
  • Publisher(s): O'Reilly Media, Inc.
  • ISBN: 9781098153434