What is bad data? Some people consider it a technical phenomenon, like missing values or malformed records, but bad data includes a lot more. In this handbook, data expert Q. Ethan McCallum has gathered 19 colleagues from every corner of the data arena to reveal how they’ve recovered from nasty data problems.
Table of Contents
- Bad Data Handbook
- About the Authors
- 1. Setting the Pace: What Is Bad Data?
- 2. Is It Just Me, or Does This Data Smell Funny?
- 3. Data Intended for Human Consumption, Not Machine Consumption
- 4. Bad Data Lurking in Plain Text
- 5. (Re)Organizing the Web’s Data
- 6. Detecting Liars and the Confused in Contradictory Online Reviews
- 7. Will the Bad Data Please Stand Up?
- 8. Blood, Sweat, and Urine
- 9. When Data and Reality Don’t Match
- 10. Subtle Sources of Bias and Error
- 11. Don’t Let the Perfect Be the Enemy of the Good: Is Bad Data Really Bad?
12. When Databases Attack: A Guide for When to Stick to Files
- Consider Files as Your Datastore
- File Concepts
- A Web Framework Backed by Files
- 13. Crouching Table, Hidden Network
14. Myths of Cloud Computing
- Introduction to the Cloud
- What Is “The Cloud”?
- The Cloud and Big Data
- Introducing Fred
- At First Everything Is Great
- They Put 100% of Their Infrastructure in the Cloud
- As Things Grow, They Scale Easily at First
- Then Things Start Having Trouble
- They Need to Improve Performance
- Higher IO Becomes Critical
- A Major Regional Outage Causes Massive Downtime
- Higher IO Comes with a Cost
- Data Sizes Increase
- Geo Redundancy Becomes a Priority
- Horizontal Scale Isn’t as Easy as They Hoped
- Costs Increase Dramatically
- Fred’s Follies
- Myth 1: Cloud Is a Great Solution for All Infrastructure Components
- Myth 2: Cloud Will Save Us Money
- Myth 3: Cloud IO Performance Can Be Improved to Acceptable Levels Through Software RAID
- Myth 4: Cloud Computing Makes Horizontal Scaling Easy
- Conclusion and Recommendations
15. The Dark Side of Data Science
- Avoid These Pitfalls
- Know Nothing About Thy Data
- Thou Shalt Provide Your Data Scientists with a Single Tool for All Tasks
- Thou Shalt Analyze for Analysis’ Sake Only
- Thou Shalt Compartmentalize Learnings
- Thou Shalt Expect Omnipotence from Data Scientists
- Final Thoughts
- 16. How to Feed and Care for Your Machine-Learning Experts
17. Data Traceability
- Personal Experience
- Immutability: Borrowing an Idea from Functional Programming
- An Example
- 18. Social Media: Erasable Ink?
- 19. Data Quality Analysis Demystified: Knowing When Your Data Is Good Enough
- About the Author
- Title: Bad Data Handbook
- Release date: November 2012
- Publisher(s): O'Reilly Media, Inc.
- ISBN: 9781449321888