March 2026
Intermediate
504 pages
11h 26m
Korean
Content preview from 파이토치로 배우는 LLM & AI
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
394
2
부
모델 사용
출력된 토큰
ID
는 다음과 같습니다.
토큰 ID: [101, 1996, 14357, 2108, 2339, 3840, 2837, 13462, 2005, 1996, 19763,
2172, 3075, 7903, 5273, 13172, 1027, 2005, 23021, 1012, 102]
첫 번째와 마지막 토큰
ID
는
101
과
102
입니다. 이 토큰은 문장의 시작과 끝을 알리는 특수 토
큰으로 모델이 해당 토큰을 기대하므로 토크나이저가 추가한 것입니다.
이제 토큰
ID
를 다시 문자열로 디코딩해 보죠.
# 특수 토큰을 확인하기 위해 디코딩합니다.
decoded = tokenizer.decode(token_ids)
print("\n특수 토큰이 포함된 디코딩 결과:", decoded)
결과에서 특수 토큰이 포함된 문장을 볼 수 있습니다.
특수 토큰이 포함된 디코딩 결과: [CLS] the ultramarathoner prequalified for the
immunohistochemistry conference in
neuroscience. [SEP]
다양한 실험을 통해 토크나이저가 텍스트를 토큰과 특수 토큰으로 변환하는 방법을 이해해 보
기를 권장합니다. 이를 이해하면 모델 동작을 디버깅하거나 어휘 사전을 바꾸기 위해 미세 튜
닝할 때 종종
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.