Prompt Engineering for Generative AI

Book description

Large language models (LLMs) and diffusion models such as ChatGPT and Stable Diffusion have unprecedented potential. Because they have been trained on all the public text and images on the internet, they can make useful contributions to a wide variety of tasks. And with the barrier to entry greatly reduced today, practically any developer can harness LLMs and diffusion models to tackle problems previously unsuitable for automation.

With this book, you'll gain a solid foundation in generative AI, including how to apply these models in practice. When first integrating LLMs and diffusion models into their workflows, most developers struggle to coax reliable enough results from them to use in automated systems. Authors James Phoenix and Mike Taylor show you how a set of principles called prompt engineering can enable you to work effectively with AI.

Learn how to empower AI to work for you. This book explains:

  • The structure of the interaction chain of your program's AI model and the fine-grained steps in between
  • How AI model requests arise from transforming the application problem into a document completion problem in the model training domain
  • The influence of LLM and diffusion model architecture—and how to best interact with it
  • How these principles apply in practice in the domains of natural language processing, text and image generation, and code

Publisher resources

View/Submit Errata

Table of contents

  1. 1. Five Principles of Prompting
    1. What is Prompt Engineering?
    2. 1. Give Direction
    3. 2. Specify Format
    4. 3. Provide Examples
    5. 4. Evaluate Quality
    6. 5. Divide Labor
  2. 2. Intro to Image Generation Models
    1. What are Diffusion models?
    2. OpenAI DALL-E
    3. Midjourney
    4. Stable Diffusion
  3. 3. Standard Practices for Text Generation
    1. Generating Lists
    2. Hierarchical List Generation
    3. Generating JSON
      1. YAML
    4. Filtering YAML Payloads
    5. Handling Invalid Payloads in YAML
    6. Explain It Like I’m Five
    7. Ask For Context
    8. Text Style Unbundling
    9. Identifying the Desired Textual Features
    10. Alternatively Extract Specific Relevant Textual Features
    11. Generating New Content with the Extracted Features
    12. Summarization
    13. Chunking Text
      1. Benefits of chunking text
      2. Poor Chunking Example
    14. Applying Prompt Engineering Principles
    15. Sentence Detection using SpaCy
    16. Building a simple chunking algorithm in Python
    17. Sliding Window Chunking
    18. Text chunking with Tiktoken
    19. Encodings
      1. Understanding the tokenization of strings
    20. Estimating Token Usage for Chat API Calls
    21. Sentinment Analysis
      1. Techniques for Fine-tuning Sentiment Analysis
      2. Limitations and Challenges in Sentiment Analysis
    22. Least To Most
      1. Planning the Architecture
      2. Coding Individual Functions
      3. Adding Tests
      4. Benefits of the Least To Most Technique
    23. Role Prompting
    24. Summary
  4. 4. Standard Practices for Image Generation
    1. Format Modifiers
    2. Art Style Modifiers
    3. Reverse-Engineering Prompts
    4. Quality Boosters
    5. Negative Prompts
    6. Weighted Terms
    7. Inpainting
    8. Outpainting
    9. Meme Unbundling
    10. Meme Mapping
  5. 5. Vector Databases
    1. What are Vector Databases?
    2. What are Embeddings?
    3. Document Loading
    4. Memory Retrieval with FAISS
    5. Hosted Vector Databases with Pinecone
  6. 6. Advanced Techniques for Text Generation
    1. Meta Prompting
    2. GPT Best Practices
      1. Avoiding Hallucinations with Reference
      2. Give GPT’s “Thinking Time”
      3. The Inner Monologue Tactic
      4. Critique Previous LLM Responses
    3. Introduction to LangChain
      1. Environment Setup
      2. Making an OpenAI LLM Request
      3. Generating an OpenAI Chat Model Request
    4. Langchain Prompt Templates
    5. Output Parsers
    6. Creating Few-Shot Prompt Templates
      1. Constructing the Example Set
      2. Formatting the Examples
      3. Creating the FewShotPromptTemplate
      4. Selecting by Length
      5. Saving and Loading LLM Prompts
    7. Data Connection
      1. Document Loaders
    8. Text Splitters
    9. Text Segmentation with Recursive Character Splitting
    10. VectorStores
      1. Building a simple QA retrieval system using FAISS
      2. Similarity Search with Score
      3. Loading and Saving the Vector Store
      4. Filtering and Similarity Search
      5. Maximal Marginal Relevance (MMR)
    11. Retrievers
      1. Vector Databases
      2. Self Querying
      3. Alternative Retrieval Mechanisms
    12. Task Decomposition
    13. Prompt Chaining
      1. LLMChain
      2. Sequential Chain
      3. Document Chains
      4. Stuff
      5. Refine
      6. Map Reduce
      7. Map re-rank
    14. Chain of Thought
    15. Agents
      1. Reason and Act (ReAct)
      2. Tools
      3. Agent ToolKits
      4. Using LLMs as an API (OpenAI Functions)
    16. Memory
      1. Long Term Memory (LTM):
      2. Short Term Memory (STM):
      3. Short-Term Memory in QA Conversation Agents:
      4. Adding Memory to an Agent in LangChain
    17. Memory in LangChain
      1. Preserving the state:
      2. Querying the state:
      3. ConversationBufferMemory
    18. Popular Memory Types in LangChain
      1. Conversation Buffer Window Memory
      2. Conversation Summary Memory
      3. Conversation Summary Buffer Memory
      4. ConversationToken Buffer Memory
    19. Advanced Agent Frameworks
      1. Plan and Execute Agents
      2. Tree of Thoughts
    20. Callbacks
      1. Global (Constructor) Callbacks
      2. Request-specific Callbacks
      3. The Verbose Argument
      4. When to Use Which?
      5. Token Counting with LangChain
    21. Classification with LLMs
    22. Building A Classification Model
    23. Majority Vote For Classification
  7. 7. Advanced Techniques for Image Generation
    1. Stable Diffusion and AUTOMATIC1111
    2. Img2Img
    3. Upscaling Images
    4. Interrogate CLIP
    5. SD Inpainting and Outpainting
    6. ControlNet
    7. Segment Anything Model (SAM)
    8. Textual Inversion
    9. Dreambooth Fine-Tuning

Product information

  • Title: Prompt Engineering for Generative AI
  • Author(s): James Phoenix, Mike Taylor
  • Release date: July 2024
  • Publisher(s): O'Reilly Media, Inc.
  • ISBN: 9781098153373