Logging and Log Management

Logging and Log Management

by Kevin Schmidt, Chris Phillips, Anton Chuvakin
Released December 2012
Publisher(s): Syngress
ISBN: 9781597496360

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Logging and Log Management: The Authoritative Guide to Understanding the Concepts Surrounding Logging and Log Management introduces information technology professionals to the basic concepts of logging and log management. It provides tools and techniques to analyze log data and detect malicious activity. The book consists of 22 chapters that cover the basics of log data; log data sources; log storage technologies; a case study on how syslog-ng is deployed in a real environment for log collection; covert logging; planning and preparing for the analysis log data; simple analysis techniques; and tools and techniques for reviewing logs for potential problems. The book also discusses statistical analysis; log data mining; visualizing log data; logging laws and logging mistakes; open source and commercial toolsets for log data collection and analysis; log management procedures; and attacks against logging systems. In addition, the book addresses logging for programmers; logging and compliance with regulations and policies; planning for log analysis system deployment; cloud logging; and the future of log standards, logging, and log analysis. This book was written for anyone interested in learning more about logging and log management. These include systems administrators, junior security engineers, application developers, and managers.
  • Comprehensive coverage of log management including analysis, visualization, reporting and more
  • Includes information on different uses for logs -- from system operations to regulatory compliance
  • Features case Studies on syslog-ng and actual real-world situations where logs came in handy in incident response
  • Provides practical guidance in the areas of report, log analysis system selection, planning a log analysis system and log data normalization and correlation

Table of contents

  1. Cover image
  2. Title page
  3. Table of Contents
  4. Copyright
  5. Acknowledgments
    1. Dr. Anton A. Chuvakin
    2. Kevin J. Schmidt
    3. Christopher Phillips
  6. About the Authors
  7. About the Technical Editor
  8. Foreword
  9. Preface
    1. Intended Audience
    2. Prerequisites
    3. Organization of the Book
    4. Chapter 5: Case Study: syslog-ng
    5. Chapter 6: Covert logging
    6. Chapter 7: Analysis Goals, Planning and Preparation: What Are We Looking for?
    7. Chapter 8: Simple Analysis Techniques
    8. Chapter 9: Filtering, Matching and Correlation
    9. Chapter 10: Statistical Analysis
    10. Chapter 11: Log Data Mining
    11. Chapter 12: Reporting and Summarization
    12. Chapter 13: Visualizing Log Data
    13. Chapter 14: Logging Laws and Logging Mistakes
    14. Chapter 15: Tools for Log Analysis and Collection
    15. Chapter 16: Log Management Procedures: Escalation, Response
    16. Chapter 17: Attacks Against Logging Systems
    17. Chapter 18: Logging for Programmers
    18. Chapter 19: Logs and Compliance
    19. Chapter 20: Planning Your Own Log Analysis System
    20. Chapter 21: Cloud Logging
    21. Chapter 22: Log Standard and Future Trends
  10. Chapter 1. Logs, Trees, Forest: The Big Picture
    1. Introduction
    2. Log Data Basics
    3. A Look at Things to Come
    4. Logs Are Underrated
    5. Logs Can Be Useful
    6. People, Process, Technology
    7. Security Information and Event Management (SIEM)
    8. Summary
    9. References
  11. Chapter 2. What is a Log?
    1. Introduction
    2. Logs? What logs?
    3. Criteria of Good Logging
    4. Summary
    5. References
  12. Chapter 3. Log Data Sources
    1. Introduction
    2. Logging Sources
    3. Log Source Classification
    4. Summary
  13. Chapter 4. Log Storage Technologies
    1. Introduction
    2. Log Retention Policy
    3. Log Storage Formats
    4. Database Storage of Log Data
    5. Hadoop Log Storage
    6. The Cloud and Hadoop
    7. Log Data Retrieval and Archiving
    8. Summary
    9. References
  14. Chapter 5. syslog-ng Case Study
    1. Introduction
    2. Obtaining syslog-ng
    3. What Is syslog-ngsyslog-ng?
    4. Example Deployment
    5. Troubleshooting syslog-ng
    6. Summary
    7. References
  15. Chapter 6. Covert Logging
    1. Introduction
    2. Complete Stealthy Log Setup
    3. Logging in Honeypots
    4. Covert Channels for Logging Brief
    5. Summary
    6. References
  16. Chapter 7. Analysis Goals, Planning, and Preparation: What Are We Looking for?
    1. Introduction
    2. Goals
    3. Planning
    4. Preparation
    5. Summary
  17. Chapter 8. Simple Analysis Techniques
    1. Introduction
    2. Line by Line: Road to Despair
    3. Simple Log Viewers
    4. Limitations of Manual Log Review
    5. Responding to the Results of Analysis
    6. Examples
    7. Summary
    8. References
  18. Chapter 9. Filtering, Normalization, and Correlation
    1. Introduction
    2. Filtering
    3. Normalization
    4. Correlation
    5. Common Patterns to Look For
    6. The Future
    7. Summary
    8. Reference
  19. Chapter 10. Statistical Analysis
    1. Introduction
    2. Frequency
    3. Baseline
    4. Machine Learning
    5. Combining Statistical Analysis with Rules-based Correlation
    6. Summary
    7. References
  20. Chapter 11. Log Data Mining
    1. Introduction
    2. Data Mining Intro
    3. Log Mining Intro
    4. Log Mining Requirements
    5. What We Mine For?
    6. Deeper into Interesting
    7. Summary
    8. References
  21. Chapter 12. Reporting and Summarization
    1. Introduction
    2. Defining the Best Reports
    3. Network Activity Reports
    4. Resource Access Reports
    5. Malware Activity Reports
    6. Critical Errors and Failures Reports
    7. Summary
  22. Chapter 13. Visualizing Log Data
    1. Introduction
    2. Visual Correlation
    3. Real-time Visualization
    4. Treemaps
    5. Log Data Constellations
    6. Traditional Log Data Graphing
    7. Summary
    8. References
  23. Chapter 14. Logging Laws and Logging Mistakes
    1. Introduction
    2. Logging Laws
    3. Logging Mistakes
    4. Summary
    5. Reference
  24. Chapter 15. Tools for Log Analysis and Collection
    1. Introduction
    2. Outsource, Build, or Buy
    3. Basic Tools for Log Analysis
    4. Utilities for Centralizing Log Information
    5. Log Analysis Tools—Beyond the Basics
    6. Commercial Vendors
    7. Summary
    8. References
  25. Chapter 16. Log Management Procedures: Log Review, Response, and Escalation
    1. Introduction
    2. Assumptions, Requirements, and Precautions
    3. Common Roles and Responsibilities
    4. PCI and Log Data
    5. Logging Policy
    6. Review, Response, and Escalation Procedures and Workflows
    7. Validation of Log Review
    8. Logbook—Evidence of Exception of Investigations
    9. PCI Compliance Evidence Package
    10. Management Reporting
    11. Periodic Operational Tasks
    12. Additional Resources
    13. Summary
    14. References
  26. Chapter 17. Attacks Against Logging Systems
    1. Introduction
    2. Attacks
    3. Summary
    4. References
  27. Chapter 18. Logging for Programmers
    1. Introduction
    2. Roles and Responsibilities
    3. Logging for Programmers
    4. Security Considerations
    5. Performance Considerations
    6. Summary
    7. References
  28. Chapter 19. Logs and Compliance
    1. Introduction
    2. PCI DSS
    3. ISO2700x Series
    4. HIPAA
    5. FISMA
    6. Summary
  29. Chapter 20. Planning Your Own Log Analysis System
    1. Introduction
    2. Planning
    3. Software Selection
    4. Policy Definition
    5. Architecture
    6. Scaling
    7. Summary
  30. Chapter 21. Cloud Logging
    1. Introduction
    2. Cloud Computing
    3. Cloud Logging
    4. Regulatory, Compliance, and Security Issues
    5. Big Data in the Cloud
    6. SIEM in the Cloud
    7. Pros and Cons of Cloud Logging
    8. Cloud Logging Provider Inventory
    9. Additional Resources
    10. Summary
    11. References
  31. Chapter 22. Log Standards and Future Trends
    1. Introduction
    2. Extrapolations of Today to the Future
    3. Log Future and Standards
    4. Desired Future
    5. Summary
  32. Index

Product information

  • Title: Logging and Log Management
  • Author(s): Kevin Schmidt, Chris Phillips, Anton Chuvakin
  • Release date: December 2012
  • Publisher(s): Syngress
  • ISBN: 9781597496360