March 2018
Beginner to intermediate
332 pages
6h 1m
Korean
Content preview from 9가지 사례로 익히는 고급 스파크 분석(2판): 현실 세계 빅데이터로 배우는 데이터 과학과 머신러닝
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
34
9가지 사례로 익히는 고급 스파크 분석 (2판)
2.4
스파크 셸과
SparkContext
시작하기
여기에서는
UC
어바인의 머신러닝 데이터 저장소
UC
Irvine
Machine
Learning
Repository
에서 구할 수 있는
표본 데이터를 사용한다. 이 저장소에서 연구와 교육을 위한 흥미로운 (더구나 무료인 ) 데이
터셋들을 구할 수 있다. 우리가 분석할 데이터셋은
2010
년에 독일의 한 병원에서 실시한 레코
드 링크 연구에서 나온 것으로, 환자의 성과 이름, 주소, 생일 등 몇 가지 기준에 따라 묶은 수
백만 건의 환자 기록 묶음이 들어 있다. 각 묶음에는 문자열 유사도에 따라
0
.
0
부터
1
.
0
까지 점
수를 매겼고, 그 후 같은 사람을 표현한 묶음과 그렇지 않은 묶음을 분류해 일일이 표시하였다.
데이터셋을 만드는 데 사용한 필드 자체의 원래 값은 환자의 개인 정보를 보호하기 위해 제거
되었으며 식별번호, 필드별 유사도 점수, 묶음별 실제 분류값 (같은 사람인지 여부)이 레코드
링크 연구 용도로 발행되었다.
다음과 같이 셸
shell
명령어로 저장소로부터 데이터를 가져오자.
$ mkdir linkage
$ cd linkage/
$ curl -L -o donation.zip http://bit.ly/1Aoywaq
$ unzip donation.zip
$ unzip 'block_*.zip'
사용할 수 있는 하둡 클러스터가 있으면,
HDFS
에 블록 데이터용 디렉터리를 생성하고 데이터
파일을 그 디렉터리에 복사한다.
$ hadoop fs -mkdir ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.