Principles and Practice of Big Data, 2nd Edition

Principles and Practice of Big Data, 2nd Edition

by Jules J. Berman
Released July 2018
Publisher(s): Academic Press
ISBN: 9780128156100

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Principles and Practice of Big Data: Preparing, Sharing, and Analyzing Complex Information, Second Edition updates and expands on the first edition, bringing a set of techniques and algorithms that are tailored to Big Data projects. The book stresses the point that most data analyses conducted on large, complex data sets can be achieved without the use of specialized suites of software (e.g., Hadoop), and without expensive hardware (e.g., supercomputers). The core of every algorithm described in the book can be implemented in a few lines of code using just about any popular programming language (Python snippets are provided).

Through the use of new multiple examples, this edition demonstrates that if we understand our data, and if we know how to ask the right questions, we can learn a great deal from large and complex data collections. The book will assist students and professionals from all scientific backgrounds who are interested in stepping outside the traditional boundaries of their chosen academic disciplines.

  • Presents new methodologies that are widely applicable to just about any project involving large and complex datasets
  • Offers readers informative new case studies across a range scientific and engineering disciplines
  • Provides insights into semantics, identification, de-identification, vulnerabilities and regulatory/legal issues
  • Utilizes a combination of pseudocode and very short snippets of Python code to show readers how they may develop their own projects without downloading or learning new software

Table of contents

  1. Cover image
  2. Title page
  3. Table of Contents
  4. Copyright
  5. Other Books by Jules J. Berman
  6. Dedication
  7. About the Author
  8. Author's Preface to Second Edition
    1. Abstract
  9. Author's Preface to First Edition
  10. 1: Introduction
    1. Abstract
    2. Section 1.1. Definition of Big Data
    3. Section 1.2. Big Data Versus Small Data
    4. Section 1.3. Whence Comest Big Data?
    5. Section 1.4. The Most Common Purpose of Big Data Is to Produce Small Data
    6. Section 1.5. Big Data Sits at the Center of the Research Universe
  11. 2: Providing Structure to Unstructured Data
    1. Abstract
    2. Section 2.1. Nearly All Data Is Unstructured and Unusable in Its Raw Form
    3. Section 2.2. Concordances
    4. Section 2.3. Term Extraction
    5. Section 2.4. Indexing
    6. Section 2.5. Autocoding
    7. Section 2.6. Case Study: Instantly Finding the Precise Location of Any Atom in the Universe (Some Assembly Required)
    8. Section 2.7. Case Study (Advanced): A Complete Autocoder (in 12 Lines of Python Code)
    9. Section 2.8. Case Study: Concordances as Transformations of Text
    10. Section 2.9. Case Study (Advanced): Burrows Wheeler Transform (BWT)
  12. 3: Identification, Deidentification, and Reidentification
    1. Abstract
    2. Section 3.1. What Are Identifiers?
    3. Section 3.2. Difference Between an Identifier and an Identifier System
    4. Section 3.3. Generating Unique Identifiers
    5. Section 3.4. Really Bad Identifier Methods
    6. Section 3.5. Registering Unique Object Identifiers
    7. Section 3.6. Deidentification and Reidentification
    8. Section 3.7. Case Study: Data Scrubbing
    9. Section 3.8. Case Study (Advanced): Identifiers in Image Headers
    10. Section 3.9. Case Study: One-Way Hashes
  13. 4: Metadata, Semantics, and Triples
    1. Abstract
    2. Section 4.1. Metadata
    3. Section 4.2. eXtensible Markup Language
    4. Section 4.3. Semantics and Triples
    5. Section 4.4. Namespaces
    6. Section 4.5. Case Study: A Syntax for Triples
    7. Section 4.6. Case Study: Dublin Core
  14. 5: Classifications and Ontologies
    1. Abstract
    2. Section 5.1. It's All About Object Relationships
    3. Section 5.2. Classifications, the Simplest of Ontologies
    4. Section 5.3. Ontologies, Classes With Multiple Parents
    5. Section 5.4. Choosing a Class Model
    6. Section 5.5. Class Blending
    7. Section 5.6. Common Pitfalls in Ontology Development
    8. Section 5.7. Case Study: An Upper Level Ontology
    9. Section 5.8. Case Study (Advanced): Paradoxes
    10. Section 5.9. Case Study (Advanced): RDF Schemas and Class Properties
    11. Section 5.10. Case Study (Advanced): Visualizing Class Relationships
  15. 6: Introspection
    1. Abstract
    2. Section 6.1. Knowledge of Self
    3. Section 6.2. Data Objects: The Essential Ingredient of Every Big Data Collection
    4. Section 6.3. How Big Data Uses Introspection
    5. Section 6.4. Case Study: Time Stamping Data
    6. Section 6.5. Case Study: A Visit to the TripleStore
    7. Section 6.6. Case Study (Advanced): Proof That Big Data Must Be Object-Oriented
  16. 7: Standards and Data Integration
    1. Abstract
    2. Section 7.1. Standards
    3. Section 7.2. Specifications Versus Standards
    4. Section 7.3. Versioning
    5. Section 7.4. Compliance Issues
    6. Section 7.5. Case Study: Standardizing the Chocolate Teapot
  17. 8: Immutability and Immortality
    1. Abstract
    2. Section 8.1. The Importance of Data That Cannot Change
    3. Section 8.2. Immutability and Identifiers
    4. Section 8.3. Coping With the Data That Data Creates
    5. Section 8.4. Reconciling Identifiers Across Institutions
    6. Section 8.5. Case Study: The Trusted Timestamp
    7. Section 8.6. Case Study: Blockchains and Distributed Ledgers
    8. Section 8.7. Case Study (Advanced): Zero-Knowledge Reconciliation
  18. 9: Assessing the Adequacy of a Big Data Resource
    1. Abstract
    2. Section 9.1. Looking at the Data
    3. Section 9.2. The Minimal Necessary Properties of Big Data
    4. Section 9.3. Data That Comes With Conditions
    5. Section 9.4. Case Study: Utilities for Viewing and Searching Large Files
    6. Section 9.5. Case Study: Flattened Data
  19. 10: Measurement
    1. Abstract
    2. Section 10.1. Accuracy and Precision
    3. Section 10.2. Data Range
    4. Section 10.3. Counting
    5. Section 10.4. Normalizing and Transforming Your Data
    6. Section 10.5. Reducing Your Data
    7. Section 10.6. Understanding Your Control
    8. Section 10.7. Statistical Significance Without Practical Significance
    9. Section 10.8. Case Study: Gene Counting
    10. Section 10.9. Case Study: Early Biometrics, and the Significance of Narrow Data Ranges
  20. 11: Indispensable Tips for Fast and Simple Big Data Analysis
    1. Abstract
    2. Section 11.1. Speed and Scalability
    3. Section 11.2. Fast Operations, Suitable for Big Data, That Every Computer Supports
    4. Section 11.3. The Dot Product, a Simple and Fast Correlation Method
    5. Section 11.4. Clustering
    6. Section 11.5. Methods for Data Persistence (Without Using a Database)
    7. Section 11.6. Case Study: Climbing a Classification
    8. Section 11.7. Case Study (Advanced): A Database Example
    9. Section 11.8. Case Study (Advanced): NoSQL
  21. 12: Finding the Clues in Large Collections of Data
    1. Abstract
    2. Section 12.1. Denominators
    3. Section 12.2. Word Frequency Distributions
    4. Section 12.3. Outliers and Anomalies
    5. Section 12.4. Back-of-Envelope Analyses
    6. Section 12.5. Case Study: Predicting User Preferences
    7. Section 12.6. Case Study: Multimodality in Population Data
    8. Section 12.7. Case Study: Big and Small Black Holes
  22. 13: Using Random Numbers to Knock Your Big Data Analytic Problems Down to Size
    1. Abstract
    2. Section 13.1. The Remarkable Utility of (Pseudo)Random Numbers
    3. Section 13.2. Repeated Sampling
    4. Section 13.3. Monte Carlo Simulations
    5. Section 13.4. Case Study: Proving the Central Limit Theorem
    6. Section 13.5. Case Study: Frequency of Unlikely String of Occurrences
    7. Section 13.6. Case Study: The Infamous Birthday Problem
    8. Section 13.7. Case Study (Advanced): The Monty Hall Problem
    9. Section 13.8. Case Study (Advanced): A Bayesian Analysis
  23. 14: Special Considerations in Big Data Analysis
    1. Abstract
    2. Section 14.1. Theory in Search of Data
    3. Section 14.2. Data in Search of Theory
    4. Section 14.3. Bigness Biases
    5. Section 14.4. Data Subsets in Big Data: Neither Additive Nor Transitive
    6. Section 14.5. Additional Big Data Pitfalls
    7. Section 14.6. Case Study (Advanced): Curse of Dimensionality
  24. 15: Big Data Failures and How to Avoid (Some of) Them
    1. Abstract
    2. Section 15.1. Failure Is Common
    3. Section 15.2. Failed Standards
    4. Section 15.3. Blaming Complexity
    5. Section 15.4. An Approach to Big Data That May Work for You
    6. Section 15.5. After Failure
    7. Section 15.6. Case Study: Cancer Biomedical Informatics Grid, a Bridge Too Far
    8. Section 15.7. Case Study: The Gaussian Copula Function
  25. 16: Data Reanalysis: Much More Important Than Analysis
    1. Abstract
    2. Section 16.1. First Analysis (Nearly) Always Wrong
    3. Section 16.2. Why Reanalysis Is More Important Than Analysis
    4. Section 16.3. Case Study: Reanalysis of Old JADE Collider Data
    5. Section 16.4. Case Study: Vindication Through Reanalysis
    6. Section 16.5. Case Study: Finding New Planets From Old Data
  26. 17: Repurposing Big Data
    1. Abstract
    2. Section 17.1. What Is Data Repurposing?
    3. Section 17.2. Dark Data, Abandoned Data, and Legacy Data
    4. Section 17.3. Case Study: From Postal Code to Demographic Keystone
    5. Section 17.4. Case Study: Scientific Inferencing From a Database of Genetic Sequences
    6. Section 17.5. Case Study: Linking Global Warming to High-Intensity Hurricanes
    7. Section 17.6. Case Study: Inferring Climate Trends With Geologic Data
    8. Section 17.7. Case Study: Lunar Orbiter Image Recovery Project
  27. 18: Data Sharing and Data Security
    1. Abstract
    2. Section 18.1. What Is Data Sharing, and Why Don't We Do More of It?
    3. Section 18.2. Common Complaints
    4. Section 18.3. Data Security and Cryptographic Protocols
    5. Section 18.4. Case Study: Life on Mars
    6. Section 18.5. Case Study: Personal Identifiers
  28. 19: Legalities
    1. Abstract
    2. Section 19.1. Responsibility for the Accuracy and Legitimacy of Data
    3. Section 19.2. Rights to Create, Use, and Share the Resource
    4. Section 19.3. Copyright and Patent Infringements Incurred by Using Standards
    5. Section 19.4. Protections for Individuals
    6. Section 19.5. Consent
    7. Section 19.6. Unconsented Data
    8. Section 19.7. Privacy Policies
    9. Section 19.8. Case Study: Timely Access to Big Data
    10. Section 19.9. Case Study: The Havasupai Story
  29. 20: Societal Issues
    1. Abstract
    2. Section 20.1. How Big Data Is Perceived by the Public
    3. Section 20.2. Reducing Costs and Increasing Productivity With Big Data
    4. Section 20.3. Public Mistrust
    5. Section 20.4. Saving Us From Ourselves
    6. Section 20.5. Who Is Big Data?
    7. Section 20.6. Hubris and Hyperbole
    8. Section 20.7. Case Study: The Citizen Scientists
    9. Section 20.8. Case Study: 1984, by George Orwell
  30. Index

Product information

  • Title: Principles and Practice of Big Data, 2nd Edition
  • Author(s): Jules J. Berman
  • Release date: July 2018
  • Publisher(s): Academic Press
  • ISBN: 9780128156100