Skip to Main Content
Hadoop数据分析
book

Hadoop数据分析

by Benjamin Bengfort, Jenny Kim
April 2018
Intermediate to advanced content levelIntermediate to advanced
229 pages
6h 19m
Chinese
Posts & Telecom Press
Content preview from Hadoop数据分析
44
3
def __init__(self, infile=sys.stdin, separator='\t'):
super(BigramMapper, self).__init__(infile, separator)
self.stopwords = nltk.corpus.stopwords.words("english")
self.punctuation = string.punctuation
def exclude(self, token):
return token in self.punctuation or token in self.stopwords
def normalize(self, token):
return token.lower()
def tokenize(self, value):
for token in nltk.wordpunct_tokenize(value):
token = self.normalize(token)
if not self.exclude(token):
yield token
def map(self):
for value in self:
for bigram in nltk.bigrams(self.tokenize(value)):
self.counter("words") # 计算短语的总数
self.emit(bigram, 1)
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Start your free trial

You might also like

Kudu:构建高性能实时数据分析存储系统

Kudu:构建高性能实时数据分析存储系统

Jean-Marc Spaggiari, Mladen Kovacevic, Brock Noland, Ryan Bosshart
Java并发编程实战

Java并发编程实战

Brian Goetz, Tim Peierls
面向机器学习的自然语言标注

面向机器学习的自然语言标注

James Pustejovsky, Amber Stubbs

Publisher Resources

ISBN: 9787115479648