book

Building LLM Powered Applications

by Valentina Alto

May 2024

Intermediate to advanced

342 pages

8h 45m

English

Packt Publishing

Read now

Unlock full access

Who this book is forWhat this book coversTo get the most out of this bookGet in touchMaking the Most Out of This Book – Get to Know Your Free BenefitsUnlock Your Book’s Exclusive BenefitsHow to unlock these benefits in three easy stepsNeed help?
What are large foundation models and LLMs?AI paradigm shift – an introduction to foundation modelsUnder the hood of an LLMMost popular LLM transformers-based architecturesEarly experimentsIntroducing the transformer architectureTraining and evaluating LLMsTraining an LLMModel evaluationBase models versus customized modelsHow to customize your modelSummaryReferences
How LLMs are changing software developmentThe copilot systemIntroducing AI orchestrators to embed LLMs into applicationsThe main components of AI orchestratorsLangChainHaystackSemantic KernelHow to choose a frameworkSummaryReferences
The most promising LLMs in the marketProprietary modelsGPT-4Gemini 1.5Claude 2Open-source modelsLLaMA-2Falcon LLMMistralBeyond language modelsA decision framework to pick the right LLMConsiderationsCase studySummaryReferences
Technical requirementsWhat is prompt engineering?Principles of prompt engineeringClear instructionsSplit complex tasks into subtasksAsk for justificationGenerate many outputs, then use the model to pick the best oneUse delimitersAdvanced techniquesFew-shot approachChain of thoughtReActSummaryReferences
Technical requirementsA brief note about LangChainGetting started with LangChainModels and promptsData connectionsMemoryChainsAgentsWorking with LLMs via the Hugging Face HubCreate a Hugging Face user access tokenStoring your secrets in an .env fileStart using open-source LLMsSummaryReferences
Technical requirementsGetting started with conversational applicationsCreating a plain vanilla botAdding memoryAdding non-parametric knowledgeAdding external toolsDeveloping the front-end with StreamlitSummaryReferences
Technical requirementsIntroduction to recommendation systemsExisting recommendation systemsK-nearest neighborsMatrix factorizationNeural networksHow LLMs are changing recommendation systemsImplementing an LLM-powered recommendation systemData preprocessingBuilding a QA recommendation chatbot in a cold-start scenarioBuilding a content-based systemDeveloping the front-end with StreamlitSummaryReferences
Technical requirementsWhat is structured data?Getting started with relational databasesIntroduction to relational databasesOverview of the Chinook databaseHow to work with relational databases in PythonImplementing the DBCopilot with LangChainLangChain agents and SQL AgentPrompt engineeringAdding further toolsDeveloping the front-end with StreamlitSummaryReferences
Technical requirementsChoosing the right LLM for codeCode understanding and generationFalcon LLMCodeLlamaStarCoderAct as an algorithmLeveraging Code InterpreterSummaryReferences

Technical requirementsWhy multimodality?Building a multimodal agent with LangChainOption 1: Using an out-of-the-box toolkit for Azure AI ServicesGetting Started with AzureCognitiveServicesToolkitSetting up the toolkitLeveraging a single toolLeveraging multiple toolsBuilding an end-to-end application for invoice analysisOption 2: Combining single tools into one agentYouTube tools and WhisperDALL·E and text generationPutting it all togetherOption 3: Hard-coded approach with a sequential chainComparing the three optionsDeveloping the front-end with StreamlitSummaryReferences
Technical requirementsWhat is fine-tuning?When is fine-tuning necessary?Getting started with fine-tuningObtaining the datasetTokenizing the dataFine-tuning the modelUsing evaluation metricsTraining and savingSummaryReferences
What is Responsible AI and why do we need it?Responsible AI architectureModel levelMetaprompt levelUser interface levelRegulations surrounding Responsible AISummaryReferences
The latest trends in language models and generative AIGPT-4V(ision)DALL-E 3AutoGenSmall language modelsCompanies embracing generative AICoca-ColaNotionMalbekMicrosoftSummaryReferences

Content preview from Building LLM Powered Applications

10 Building Multimodal Applications with LLMs

In this chapter, we are going beyond LLMs, to introduce the concept of multimodality while building agents. We will see the logic behind the combination of foundation models in different AI domains – language, images, and audio – into one single agent that can adapt to a variety of tasks. By the end of this chapter, you will be able to build your own multimodal agent, providing it with the tools and LLMs needed to perform various AI tasks.

Throughout this chapter, we will cover the following topics: