October 2025
Intermediate
424 pages
10h 19m
Korean
Content preview from 실무로 통하는 LLM 애플리케이션 설계
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
113
3
장
어휘와 토큰화
이와 관련해 타오
Tao
연구진
6
은 어휘 크기에 대한 스케일링 법칙을 제안했습니다. 이들은 모델의
크기와 연산량이 증가할수록 최적의 어휘 크기도 함께 증가해야 한다고 주장합니다. 이 연구가
이루어진 시점을 기준으로, 대부분의 모델이 최적보다 작은 어휘 크기를 사용하며 어휘 크기를
늘리면 성능 향상을 기대할 수 있다고 합니다.
3.2
토크나이저
이제 인간과 모델 사이의 텍스트 처리 인터페이스 역할을 하는 소프트웨어인 토크나이저를 자
세히 알아보겠습니다. 토크나이저는 두 가지 주요 역할을 담당합니다.
1
토크나이저 사전
훈련
단계에서는 대규모 텍스트에 토크나이저를 적용해 어휘를 생성합니다.
2
모델 훈련과 추론 과정에서 입력을 처리할 때는 형식이 없는 원시 텍스트를 토크나이저 알고리즘
으로
처
리하여 유효한 토큰의 시퀀스로 분할합니다. 이러한 과정을 도식화한 것이 바로 [그림
3
-
2
]입니다.
"G25": 2030,
"44" : 2031,
"Gtrue" : 2032,
"G</" : 2033,
"sel" : 2034,
"usion" : 2035,
"Gnet" : 2036,
"ember" : 2037,
"ready" : 2038,
"Goffic" : 2039,
"raw" : 2040,
"pone" : 2041,
"If" : 2042,
"begin" : 2043, ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.