May 2022
Beginner to intermediate
756 pages
21h 30m
Korean
book
HTTP 완벽 가이드
by David Gourlry, Brian Totty, Lee Eung Jun, Sang -Il Jeong, Anshu Agarwal, Seilu Reddy, Marjorie Sayre
Content preview from HTTP 완벽 가이드
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
265
9
장 웹 로봇
The Definitive Guide
의 능력을 이용하는 것으로 알려져 있다(구글
17
과 같은 몇몇 검색엔진은 그들이 크
롤링한 페이지들을 그대로 보관하기 때문에, 콘텐츠가 제거되더라도 일정 시간 동
안에는 여전히 검색되고 접근 가능하다).
동적 게이트웨이 접근
로봇들이 그들이 접근하고 있는 것에 대해 언제나 잘 알고 있는 것은 아니다. 로봇
은 게이트웨이 애플리케이션의 콘텐츠에 대한 URL로 요청을 할 수도 있다. 이 경우
얻은 데이터는 아마도 특수 목적을 위한 것일 테고 처리 비용이 많이 들 것이다. 많
은 웹 사이트 관리자가 게이트웨이에서 얻은 문서를 요청하는 순진한 로봇들을 좋
아하지 않는다.
9.4
로봇 차단하기
로봇 커뮤니티는 로봇에 의한 웹 사이트 접근이 유발할 수 있는 문제를 알고 있었
다. 1994년, 로봇이 그들에게 맞지 않는 장소에 들어오지 않도록 하고 웹 마스터에
게 로봇의 동작을 더 잘 제어할 수 있는 메커니즘을 제공하는 단순하고 자발적인
기법이 제안되었다. 이 표준은 “Robots Exclusion Standard”라고 이름 지어졌지만,
로봇의 접근을 제어하는 정보를 저장하는 파일의 이름을 따서 종종 그냥 robots.txt
라고 불린다.
robots.txt의 아이디어는 단순하다. 어떤 웹 서버는 서버의 문서 루트에 robots ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.