October 2025
Intermediate
424 pages
10h 19m
Korean
Content preview from 실무로 통하는 LLM 애플리케이션 설계
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
84
1
부
LLM의 구성 요소
비영어 텍스트 제거
langdetect
,
langid
,
fastText
,
pycld2
와 같은 라이브러리를 사용해 텍스트의 언어를 감지합니다. 예를 들
어
C4
데이터셋은
langdetect
로 판단했을 때 영어일 확률이
0
.
99
보다 높은 텍스트만 유지합니다. 이러한
라이브러리는 상용구와 웹 페이지 부산물들에 영어 확률을 낮게 부여하므로 이들을 제거하는 데도 활용할
수 있습니다.
검색 엔진 최적화(
SEO
) 텍스트/스팸 제거
반복적인 문자 시퀀스가 많은 문서는 제거됩니다. 폐쇄 클래스 단어
closed
class
word
비율이 낮은 문서도 제거됩
니다. 영어의 폐쇄 클래스 단어는 ‘
of
’, ‘
at
’, ‘
the
’, ‘
is
’와 같은 기능어입니다. 페이지가 키워드 스터핑이나 다른
SEO
기법을 사용한다면 폐쇄 클래스 단어 비율이 낮을 것입니다.
음란물/유해 텍스트 제거
‘
List
of
Dirty
,
Naughty
,
Obscene
or
Otherwise
Bad
Words
’
47
와 같은 키워드 목록에 포함된 단어가 있
는 문서들은 제거됩니다.
langdetect
와
langid
같은 도구는 대규모로 작성된 텍스트의 언어를 빠르게 판별하는 데 유용
하지만, 코드 스위칭된 텍스트 (다국어가 혼합된 다중 언어 텍스트)는 어떻게 처리할까요? 직
접 시도해 보죠. ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.