O'Reilly logo

Learning Jupyter by Dan Toomey

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Log file examination

I downloaded one of the access_log files from http://www.monitorware.com/. Like any other web access log, we have one line per entry, like this:

64.242.88.10 - - [07/Mar/2004:16:05:49 -0800] "GET /twiki/bin/edit/Main/Double_bounce_sender?topicparent=Main.ConfigurationVariables HTTP/1.1" 401 12846
  • The first part is the IP address of the caller, followed by timestamp, type of HTTP access, URL referenced, HTTP type, resultant HTTP Response code, and finally, the number of bytes in the response.
  • We can use Spark to load in and parse out some statistics of the log entries, as in this script:
import pyspark
if not 'sc' in globals():
    sc = pyspark.SparkContext()
textFile = sc.textFile("access_log")
print(textFile.count(),"access records") ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required