May 2025
Intermediate to advanced
382 pages
10h 33m
Chinese
Content preview from 图解大模型 : 生成式AI 原理与实战
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
词元和嵌入
｜
47
Phi-3
（和
Llama 2
）
<s>
English
and C AP IT AL IZ ATION
�
� � � � � �
show
_ to kens False None elif == >= else : two tabs :" " Three tabs : "
"
1
2
.
0
*
5
0
=
6
0
0
2.1.6
分词器属性
前文简单讨论了一些训练好的分词器，展示了不同分词器之间的多种差异。是什么决定了
它们的分词行为呢？有三大类设计上的选择决定了分词器如何分解文本：分词方法、用于
初始化分词器的参数以及训练分词器的目标数据所在的领域。
1.
分词方法
正如我们所见，分词方法有许多种，其中
BPE
是最流行的一种。每种方法都定义了一种
算法，用于选择合适的词元集来表示数据集。要想大概了解这些方法，可以参考
Hugging
Face
的分词器汇总页面。
2.
用于初始化分词器的参数
选择分词方法后，
LLM
设计者需要设置分词器的一些关键参数，主要包括词表大小、特殊
词元和大小写处理策略。
词表大小
分词器的词表中保留多少个词元？（常见的词表大小的值有
30K
3
、
50K
，但我们越来越
多地看到像
100K
这样更大规模的词表了。）
特殊词元
我们希望模型跟踪哪些特殊词元？我们可以根据需要添加任意数量的特殊词元，特别是
要针对特定用例构建
LLM
时。常见的特殊词元包括：
•
文本开始词元（例如
<s>
）
•
文本结束词元
•
填充词元
•
未知词元
• CLS
词元
•
掩码词元
除此之外，
LLM
设计者还可以添加有助于对所关注问题领域建模的词元 ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.