Introduction to Python and Large Language Models / Знакомство с Python и большими языковыми моделями
Год издания: 2024
Автор: Grigorov Dilyan / Григоров Дилян
Издательство: Apress Media LLC
ISBN: 979-8-8688-0540-0
Язык: Английский
Формат: PDF/EPUB
Качество: Издательский макет или текст (eBook)
Интерактивное оглавление: Да
Количество страниц: 395
Описание: Gain a solid foundation for Natural Language Processing (NLP) and Large Language Models (LLMs), emphasizing their significance in today’s computational world. This book is an introductory guide to NLP and LLMs with Python programming.
The book starts with the basics of NLP and LLMs. It covers essential NLP concepts, such as text preprocessing, feature engineering, and sentiment analysis using Python. The book offers insights into Python programming, covering syntax, data types, conditionals, loops, functions, and object-oriented programming. Next, it delves deeper into LLMs, unraveling their complex components.
You’ll learn about LLM elements, including embedding layers, feedforward layers, recurrent layers, and attention mechanisms. You’ll also explore important topics like tokens, token distributions, zero-shot learning, LLM hallucinations, and insights into popular LLM architectures such as GPT-4, BERT, T5, PALM, and others. Additionally, it covers Python libraries like Hugging Face, OpenAI API, and Cohere. The final chapter bridges theory with practical application, offering step-by-step examples of coded applications for tasks like text generation, summarization, language translation, question-answering systems, and chatbots.
In the end, this book will equip you with the knowledge and tools to navigate the dynamic landscape of NLP and LLMs.
What You’ll Learn:
Understand the basics of Python and the features of Python 3.11
Explore the essentials of NLP and how do they lay the foundations for LLMs.
Review LLM components.
Develop basic apps using LLMs and Python.
Who This Book Is For:
Data analysts, AI and Machine Learning Experts, Python developers, and Software Development Professionals interested in learning the foundations of NLP, LLMs, and the processes of building modern LLM applications for various tasks.
Получите прочную основу для работы с системами обработки естественного языка (NLP) и большими языковыми моделями (LLM), подчеркнув их значимость в современном мире вычислений. Эта книга представляет собой вводное руководство по NLP и LLM с программированием на Python.
Книга начинается с основ NLP и LLMS. В ней рассматриваются основные концепции NLP, такие как предварительная обработка текста, разработка функциональных возможностей и анализ настроений с использованием Python. Книга дает представление о программировании на Python, охватывая синтаксис, типы данных, условные выражения, циклы, функции и объектно-ориентированное программирование. Далее автор углубляется в LLMS, разбираясь с их сложными компонентами.
Вы узнаете об элементах LLM, включая слои встраивания, слои прямой связи, рекуррентные слои и механизмы внимания. Вы также изучите такие важные темы, как токены, распределение токенов, обучение с нуля, галлюцинации LLM и ознакомитесь с популярными архитектурами LLM, такими как GPT-4, BERT, T5, PALM и другими. Кроме того, он охватывает такие библиотеки Python, как Hugging Face, OpenAI API и Cohere. Заключительная глава связывает теорию с практическим применением, предлагая пошаговые примеры закодированных приложений для таких задач, как генерация текста, обобщение, языковой перевод, системы ответов на вопросы и чат-боты.
В конечном счете, эта книга снабдит вас знаниями и инструментами для навигации по динамичному ландшафту NLP и LLMS.
Чему вы научитесь:
Разберитесь в основах Python и возможностях Python 3.11
Изучите основы NLP и то, как они закладывают основы для LLM.
Ознакомьтесь с компонентами LLM.
Разрабатывайте базовые приложения с использованием LLM и Python.
Для кого предназначена эта книга:
Аналитики данных, специалисты по искусственному интеллекту и машинному обучению, разработчики на Python и специалисты по разработке программного обеспечения заинтересованы в изучении основ NLP, LLMS и процессов создания современных LLM-приложений для различных задач.
Примеры страниц (скриншоты)
Оглавление
About the Author ............................................................................................................xv
About the Technical Reviewer ...........................................................................................xvii
Acknowledgments ...........................................................................................................xix
Introduction ...................................................................................................................xxi
Chapter 1: Evolution and Significance of Large Language Models ....................................;..... 1
The Evolutionary Steps of Large Language Models .............................................................. 2
Markov, Shannon, and the Language Models ...................................................................... 3
Chomsky and the Language Models .................................................................................. 5
Rule-Based Language Models ........................................................................................... 6
The First Chatbot: ELIZA ................................................................................................. 6
Statistical Language Processing ........................................................................................ 8
Vector Space Models and State Space Models ..................................................................... 12
Neural Language Models – The Rise of LLMs ....................................................................... 13
Attention-Based Language Models ....................................................................................... 20
Large Language Models (LLMs) ............................................................................................. 22
The Era of Multimodal Learning ............................................................................................ 22
Understanding the NLP Basics .................................................................................................... 28
What Exactly Is Natural Language Processing? .................................................................... 28
How Does NLP Function? ...................................................................................................... 29
Elements of NLP .................................................................................................................... 29
NLP Tasks .............................................................................................................................. 31
Text Preprocessing and Feature Engineering ........................................................................ 32
Word Embeddings and Semantic Understanding .................................................................. 43
Sentiment Analysis and Text Classification with Python ....................................................... 47
Exploring Approaches in Natural Language Processing (NLP) ............................................... 52
Delineating NLP Basics from LLM Capabilities ...................................................................... 54
Contrasting Traditional NLP Techniques with LLMs ............................................................... 55
Summary ............................................................................................................................... 57
Chapter 2: What Are Large Language Models? ...................................................................... 59
LM’s Development Stages .......................................................................................................... 59
How Do Large Language Models Work? ..................................................................................... 62
Overall Architecture of Large Language Models ......................................................................... 63
In-Depth Architecture of the LLMs .............................................................................................. 64
Tokenization .......................................................................................................................... 65
Attention ..................................................................................................................................... 65
Attention Mechanisms in LLMs ............................................................................................. 66
Positional Encoding ............................................................................................................... 67
Activation Functions .............................................................................................................. 68
Layer Normalization .............................................................................................................. 69
Architectures ............................................................................................................................... 72
Encoder-Decoder ................................................................................................................... 73
Causal Decoder ..................................................................................................................... 73
Prefix Decoder ....................................................................................................................... 73
Pre-training Objectives .......................................................................................................... 73
Model Adaptation ........................................................................................................................ 74
Pre-training ........................................................................................................................... 74
Alignment Verification and Utilization .................................................................................... 75
Prompting/Utilization ............................................................................................................. 76
Training of LLMs .......................................................................................................................... 77
Benefits and Challenges of LLMs in Various Domains ................................................................ 78
General Purpose .................................................................................................................... 78
Medical Applications ............................................................................................................. 79
Healthcare Communication and Management ...................................................................... 80
Enhanced Natural Language Processing ............................................................................... 80
Education ............................................................................................................................... 81
Content Creation and Augmentation ...................................................................................... 82
Language Translation and Localization ................................................................................. 82
Research and Data Analysis .................................................................................................. 82
Finance .................................................................................................................................. 82
Creative Arts .......................................................................................................................... 83
Ethical and Responsible Use ................................................................................................. 83
Legal and Compliance Assistance ......................................................................................... 83
Financial Analysis and Forecasting ....................................................................................... 84
Disaster Response and Management .................................................................................... 84
Personalized Marketing and Customer Insights .................................................................... 84
Gaming and Interactive Entertainment .................................................................................. 84
Accessibility Enhancements .................................................................................................. 85
Environmental Monitoring and Sustainability ........................................................................ 85
LLMs and Engineering Applications ....................................................................................... 85
Chatbots ................................................................................................................................ 86
LLM Agents ............................................................................................................................ 87
LLM Limitations ........................................................................................................................... 87
Bias ........................................................................................................................................ 88
Hallucinations ........................................................................................................................ 89
Beyond the Hype of the LLMs – Why Are They So Popular? ........................................................ 90
Common Benefits – The Real Reason Why LLMs Are So Popular .......................................... 91
Large Language Models for Business ................................................................................... 95
Summary .................................................................................................................................. 100
Chapter 3: Python for LLMs .................................................................................................. 101
Python at a Glance .................................................................................................................... 101
Python Syntax and Semantics ............................................................................................. 102
Syntax Design Principles ..................................................................................................... 103
Zen of Python ....................................................................................................................... 103
Python Identifiers ................................................................................................................ 104
Python Indentation .............................................................................................................. 105
Python Multiline Statements ............................................................................................... 106
Quotations in Python ........................................................................................................... 106
Comments in Python ........................................................................................................... 107
How to Install Python and Your First Python Program ......................................................... 108
Install Python on macOS ...................................................................................................... 112
Installing Python on Linux – Ubuntu/Debian and Fedora ..................................................... 113
Your First Python Program ........................................................................................................ 114
Variables and Data Types, Numbers, Strings, and Casting ........................................................ 114
Naming a Variable ............................................................................................................... 115
Data Types ................................................................................................................................. 116
Numbers in Python .............................................................................................................. 116
RAW Strings ......................................................................................................................... 120
Booleans and Operators ...................................................................................................... 121
Conditionals and Loops ............................................................................................................. 132
Conditionals ......................................................................................................................... 132
Grouping Statements ........................................................................................................... 133
Nested Blocks ...................................................................................................................... 134
Else and Elif Clauses ........................................................................................................... 135
One-Line if Statements ........................................................................................................ 136
Python Loops (For and While) ................................................................................................... 136
While Loop in Python ........................................................................................................... 137
Else Statement with while Loop .......................................................................................... 138
Creating an Infinite Loop with Python while Loop ............................................................... 138
For Loops in Python ............................................................................................................. 139
Else Statement with for Loop .............................................................................................. 140
Nested Loops in Python ....................................................................................................... 140
Loop Control Statements ..................................................................................................... 142
Python Data Structures: Lists, Sets, Tuples, Dictionaries .......................................................... 142
What Is a Data Structure? ................................................................................................... 143
Built-In Data Structures ....................................................................................................... 144
Additional List Operations ................................................................................................... 147
Dictionaries in Python .......................................................................................................... 148
Tuples in Python .................................................................................................................. 152
Sets in Python ..................................................................................................................... 155
Regular Functions and Lambda Functions ................................................................................ 158
What Is a Function in Python? ............................................................................................. 158
The return Statement .......................................................................................................... 159
Return or Print in a Function ............................................................................................... 160
Methods vs. Functions ........................................................................................................ 160
How to Call a Function in Python ......................................................................................... 160
Function Arguments in Python ............................................................................................ 161
Summary .................................................................................................................................. 164
Chapter 4: Python and Other Programming Approaches ......................................................... 165
Object-Oriented Programming in Python ................................................................................... 165
Why Do We Use Object-Oriented Programming in Python? ................................................. 166
Your First Python Object ...................................................................................................... 168
Object-Oriented Programming (OOP) in Python Is Founded on Four Fundamental
Concepts .............................................................................................................................. 169
Modules and File Handling .................................................................................................. 174
Python File Handling ............................................................................................................ 180
The Powerful Features of Python 3.11 ................................................................................ 186
Understanding the Role of Python 3.11 in AI and NLP – Why Python? ................................ 195
Summary .................................................................................................................................. 197
Chapter 5: Basic Overview of the Components of the LLM Architectures ......................................... 199
Embedding Layers .................................................................................................................... 200
Stage 1: Nodes .................................................................................................................... 202
Stage 2: Returning to the Words .......................................................................................... 203
Stage 3: Implementing the Softmax Layer .......................................................................... 204
Feedforward Layers ............................................................................................................. 205
Recurrent Layers ................................................................................................................. 208
Attention Mechanisms ......................................................................................................... 209
Understanding Tokens and Token Distributions and Predicting the Next Token ........................ 212
Understanding Tokenization in the Context of Large Language Models .............................. 212
The Advantages of Tokenization for LLMs ........................................................................... 213
Limitations and Challenges ................................................................................................. 213
Challenges in Current Tokenization Techniques .................................................................. 213
Tokenization Strategies in Large Language Models ............................................................ 215
Predicting the Next Token .................................................................................................... 218
Zero-Shot and Few-Shot Learning ............................................................................................ 222
Few-Shot Learning .............................................................................................................. 222
Zero-Shot Learning .............................................................................................................. 223
Few-Shot Learning .............................................................................................................. 226
One-Shot Learning .............................................................................................................. 226
Zero-Shot Learning .............................................................................................................. 227
LLM Hallucinations .................................................................................................................... 227
Classification of Hallucinations in Large Language Models (LLMs) ..................................... 228
Implications of AI Hallucination ........................................................................................... 229
Mitigating the Risks of AI Hallucinations: Strategies for Prevention ................................... 230
When Hallucinations Might Be Good? ....................................................................................... 232
Future Implications ................................................................................................................... 233
Examples of LLM Architectures ................................................................................................. 234
GPT-4 ................................................................................................................................... 235
BERT .................................................................................................................................... 240
T5 ........................................................................................................................................ 245
Cohere ................................................................................................................................. 246
PaLM 2 ................................................................................................................................. 247
Pathway Interaction and Collaboration ..................................................................................... 249
Selective Pathway Engagement .......................................................................................... 249
Generating Outputs .............................................................................................................. 250
Jurassic-2 ............................................................................................................................ 250
Claude v1 ............................................................................................................................. 251
Falcon 40B ........................................................................................................................... 252
LLaMA .................................................................................................................................. 254
LaMDA ................................................................................................................................. 255
Guanaco-65B ....................................................................................................................... 257
Orca ..................................................................................................................................... 258
StableLM ............................................................................................................................. 258
Palmyra ............................................................................................................................... 259
GPT4ALL .............................................................................................................................. 260
Summary .................................................................................................................................. 261
Chapter 6: Applications of LLMs in Python ................................................................................. 263
Text Generation and Creative Writing ........................................................................................ 263
The Mechanism Behind Text Generation ............................................................................. 263
The Significance of Text Generation .................................................................................... 264
Key Use Cases of Text Generation ....................................................................................... 264
What Is Creative Writing ...................................................................................................... 265
Utilizing LLMs for Creative Writing Endeavors ..................................................................... 265
Blog Post Generator on a Topic and Length Provided by the User Based on OpenAI ........... 266
Language Translation and Multilingual LLMs ............................................................................ 268
Advantages of Utilizing LLMs for Translation ....................................................................... 268
How LLMs Translate Languages? ........................................................................................ 269
Challenges Associated with LLMs in Translation ................................................................. 269
The Potential Impacts of LLMs on the Translation and Localization Industry ...................... 270
Text Summarization and Document Understanding .................................................................. 273
Article Summarization Application Using User-Provided
URL .............................................. 274
Question-Answering Systems: Knowledge at Your Fingertips .................................................. 279
Enhancing Question-Answering Capabilities Through Large Language Models (LLMs) .......... 279
Utilizing Large Language Models for Advanced Document Analysis ................................... 279
The Journey from Data to Response: A Comprehensive Overview ...................................... 280
Practical Applications and Use Cases of Generative Question Answering ........................... 281
Question Answering Chatbot over Documents with Sources .............................................. 282
Full Code of the App ............................................................................................................ 287
Chatbots and Virtual Assistants ................................................................................................ 290
What Is the Concept Behind Chatbots? ............................................................................... 290
Practical Applications of LLM-Trained Chatbots .................................................................. 290
Guide to Building a Chatbot with LLMs ............................................................................... 291
Customer Support Question Answering Chatbot ................................................................. 293
Step 2: Formulate a Prompt for GPT-3 Utilizing Recommended Techniques ....................... 295
Basic Prompting – The Common Thing Between All Applications Presented ............................ 299
Understanding Prompting .................................................................................................... 299
Fundamental Prompting Techniques ................................................................................... 299
Summary .................................................................................................................................. 301
Chapter 7: Harnessing Python 3.11 and Python Libraries for LLM Development ..............................303
LangChain ................................................................................................................................. 303
LangChain Features ............................................................................................................. 304
What Are the Integrations of LangChain? ............................................................................ 305
How to Build Applications in LangChain? ............................................................................ 305
Use Cases of LangChain ...................................................................................................... 306
Example of a LangChain App – Article Summarizer ............................................................ 307
Hugging Face ............................................................................................................................ 309
History of Hugging Face ...................................................................................................... 310
Key Components of Hugging Face ....................................................................................... 310
OpenAI API ................................................................................................................................. 316
Features of the OpenAI API .................................................................................................. 317
Industry Applications of the OpenAI API .............................................................................. 318
Simple Example of a Connection to the OpenAI API ............................................................ 320
Cohere ....................................................................................................................................... 322
Cohere Models ..................................................................................................................... 323
Pinecone ................................................................................................................................... 327
How Vector Databases Operate ........................................................................................... 327
What Exactly Is a Vector Database? .................................................................................... 328
Pinecone’s Features ............................................................................................................ 328
Practical Applications .......................................................................................................... 329
Lamini.ai ................................................................................................................................... 332
Lamini’s Operational Mechanics ......................................................................................... 332
Lamini’s Features, Functionalities, and Advantages ............................................................ 332
Applications and Use Cases for Lamini ............................................................................... 333
Data Collection, Cleaning, and Preparation of Python Libraries ................................................ 337
Gathering and Preparing Data for Large Language Models ................................................. 337
Data Acquisition ................................................................................................................... 338
What Is Data Preprocessing? .............................................................................................. 338
Preparing Datasets for Training ........................................................................................... 339
Managing Unwanted Data ................................................................................................... 339
Handling Document Length ................................................................................................. 343
Text Produced by Machines ................................................................................................. 344
Removing Duplicate Content ............................................................................................... 344
Data Decontamination ......................................................................................................... 345
Addressing Toxicity and Bias ............................................................................................... 347
Protecting Personally Identifiable Information (PII) ............................................................. 350
Managing Missing Data ....................................................................................................... 350
Enhancing Datasets Through Augmentation ........................................................................ 353
Data Normalization .............................................................................................................. 353
Data Parsing ........................................................................................................................ 356
Tokenization ........................................................................................................................ 358
Stemming and Lemmatization ............................................................................................. 359
Feature Engineering for Large Language Models ..................................................................... 362
Word Embeddings ............................................................................................................... 362
Contextual Embeddings ....................................................................................................... 362
Subword Embeddings ......................................................................................................... 363
Best Practices for Data Processing ........................................................................................... 364
Implementing Strong Data Cleansing Protocols .................................................................. 365
Proactive Bias Management ................................................................................................ 365
Implementing Continuous Quality Control and Feedback Mechanisms .............................. 365
Fostering Interdisciplinary Collaboration ............................................................................. 365
Prioritizing Educational Growth and Skill Development ...................................................... 366
Delving into Key Libraries ......................................................................................................... 366
Summary .................................................................................................................................. 368
Index ................................................................................................................................ 369
