Learning Jupyter 5 - Second Edition

Spark primes

We can run a series of numbers through a filter to determine whether each number is prime or not. We can use this script:

import pysparkif not 'sc' in globals():    sc = pyspark.SparkContext()    def is_it_prime(number):        #make sure n is a positive integer    number = abs(number)        #simple tests    if number < 2:        return False        #2 is special case    if number == 2:        return True        #all other even numbers are not prime    if not number & 1:        return False        #divisible into it's square root    for x in range(3, int(number**0.5)+1, 2):        if number % x == 0:            return False            #must be a prime    return True# pick a good rangenumbers = sc.parallelize(range(100000))# see how many primes are in that rangeprint(numbers.filter(is_it_prime).count())

The script generates numbers ...