book

Big Data Analytics with Applications in Insider Threat Detection

by Bhavani Thuraisingham, Pallabi Parveen, Mohammad Mehedy Masud, Latifur Khan

November 2017

Intermediate to advanced

578 pages

22h 28m

English

Auerbach Publications

Read now

Unlock full access

1.1 Overview1.2 Supporting Technologies1.3 Stream Data Analytics1.4 Applications of Stream Data Analytics for Insider Threat Detection1.5 Experimental BDMA and BDSP Systems1.6 Next Steps in BDMA and BDSP1.7 Organization of This Book1.8 Next Steps
Introduction to Part I
2.1 Overview2.2 Security Policies2.2.1 Access Control Policies2.2.1.1 Authorization-Based Access Control Policies2.2.1.2 Role-Based Access Control2.2.1.3 Usage Control2.2.1.4 Attribute-Based Access Control2.2.2 Administration Policies2.2.3 Identification and Authentication2.2.4 Auditing: A Database System2.2.5 Views for Security2.3 Policy Enforcement and Related Issues2.3.1 SQL Extensions for Security2.3.2 Query Modification2.3.3 Discretionary Security and Database Functions2.4 Data Privacy2.5 Summary and DirectionsReferences
3.1 Introduction3.2 Overview of Data Mining Tasks and Techniques3.3 Artificial Neural Networks3.4 Support Vector Machines3.5 Markov Model3.6 Association Rule Mining (ARM)3.7 Multiclass Problem3.8 Image Mining3.8.1 Overview3.8.2 Feature Selection3.8.3 Automatic Image Annotation3.8.4 Image Classification3.9 SummaryReferences
4.1 Overview4.2 Data Mining for Cyber Security4.2.1 Cyber Security Threats4.2.1.1 Cyber Terrorism, Insider Threats, and External Attacks4.2.1.2 Malicious Intrusions4.2.1.3 Credit Card Fraud and Identity Theft4.2.1.4 Attacks on Critical Infrastructures4.2.2 Data Mining for Cyber Security4.3 Data Mining Tools4.4 Summary and DirectionsReferences
5.1 Introduction5.2 Cloud Computing5.2.1 Overview5.2.2 Preliminaries5.2.2.1 Cloud Deployment Models5.2.2.2 Service Models5.2.3 Virtualization5.2.4 Cloud Storage and Data Management5.2.5 Cloud Computing Tools5.2.5.1 Apache Hadoop5.2.5.2 MapReduce5.2.5.3 CouchDB5.2.5.4 HBase5.2.5.5 MongoDB5.2.5.6 Hive5.2.5.7 Apache Cassandra5.3 Semantic Web5.3.1 XML5.3.2 RDF5.3.3 SPARQL5.3.4 OWL5.3.5 Description Logics5.3.6 Inferencing5.3.7 SWRL5.4 Semantic Web and Security5.4.1 XML Security5.4.2 RDF Security5.4.3 Security and Ontologies5.4.4 Secure Query and Rules Processing5.5 Cloud Computing Frameworks Based on Semantic Web Technologies5.5.1 RDF Integration5.5.2 Provenance Integration5.6 Summary and DirectionsReferences
6.1 Introduction6.2 Insider Threat Detection6.3 The Challenges, Related Work, and Our Approach6.4 Data Mining for Insider Threat Detection6.4.1 Our Solution Architecture6.4.2 Feature Extraction and Compact Representation6.4.2.1 Vector Representation of the Content6.4.2.2 Subspace Clustering6.4.3 RDF Repository Architecture6.4.4 Data Storage6.4.4.1 File Organization6.4.5 Answering Queries Using Hadoop MapReduce6.4.6 Data Mining Applications6.5 Comprehensive Framework6.6 Summary and DirectionsReferences
7.1 Introduction7.2 Infrastructure Tools to Host BDMA Systems7.3 BDMA Systems and Tools7.3.1 Apache Hive7.3.2 Google BigQuery7.3.3 NoSQL Database7.3.4 Google BigTable7.3.5 Apache HBase7.3.6 MongoDB7.3.7 Apache Cassandra7.3.8 Apache CouchDB7.3.9 Oracle NoSQL Database7.3.10 Weka7.3.11 Apache Mahout7.4 Cloud Platforms7.4.1 Amazon Web Services’ DynamoDB7.4.2 Microsoft Azure’s Cosmos DB7.4.3 IBM’s Cloud-Based Big Data Solutions7.4.4 Google’s Cloud-Based Big Data Solutions7.5 Summary and DirectionsReferences
Introduction to Part II
8.1 Introduction8.2 Challenges8.3 Infinite Length and Concept Drift8.4 Concept Evolution8.5 Limited Labeled Data8.6 Experiments8.7 Our Contributions8.8 Summary and DirectionsReferences
9.1 Introduction9.2 Approach to Data Stream Classification9.3 Single-Model Classification9.4 Ensemble Classification and Baseline Approach9.5 Novel Class Detection9.5.1 Novelty Detection9.5.2 Outlier Detection9.5.3 Baseline Approach9.6 Data Stream Classification with Limited Labeled Data9.6.1 Semisupervised Clustering9.6.2 Baseline Approach9.7 Summary and DirectionsReferences
10.1 Introduction10.2 Ensemble Development10.2.1 Multiple Partitions of Multiple Chunks10.2.1.1 An Ensemble Built on MPC10.2.1.2 MPC Ensemble Updating Algorithm10.2.2 Error Reduction Using MPC Training10.2.2.1 Time Complexity of MPC10.3 Experiments10.3.1 Datasets and Experimental Setup10.3.1.1 Real (Botnet) Dataset10.3.1.2 Baseline Methods10.3.2 Performance Study10.4 Summary and DirectionsReferences
11.1 Introduction11.2 ECSMiner11.2.1 Overview11.2.2 High Level Algorithm11.2.3 Nearest Neighborhood Rule11.2.4 Novel Class and Its Properties11.2.5 Base Learners11.2.6 Creating Decision Boundary during Training11.3 Classification with Novel Class Detection11.3.1 High-Level Algorithm11.3.2 Classification11.3.3 Novel Class Detection11.3.4 Analysis and Discussion11.3.4.1 Justification of the Novel Class Detection Algorithm11.3.4.2 Deviation between Approximate and Exact q -NSC Computation11.3.4.3 Time and Space Complexity11.4 Experiments11.4.1 Datasets11.4.1.1 Synthetic Data with only Concept Drift (SynC)11.4.1.2 Synthetic Data with Concept Drift and Novel Class (SynCN)11.4.1.3 Real Data—KDDCup 99 Network Intrusion Detection (KDD)11.4.1.4 Real Data—Forest Covers Dataset from UCI Repository (Forest)11.4.2 Experimental Set-Up11.4.3 Baseline Approach11.4.4 Performance Study11.4.4.1 Evaluation Approach11.4.4.2 Results11.5 Summary and DirectionsReferences
12.1 Introduction12.2 Description of ReaSC12.3 Training with Limited Labeled Data12.3.1 Problem Description12.3.2 Unsupervised K -Means Clustering12.3.3 K -Means Clustering with Cluster-Impurity Minimization12.3.4 Optimizing the Objective Function with Expectation Maximization (E-M)12.3.5 Storing the Classification Model12.4 Ensemble Classification12.4.1 Classification Overview12.4.2 Ensemble Refinement12.4.3 Ensemble Update12.4.4 Time Complexity12.5 Experiments12.5.1 Dataset12.5.2 Experimental Setup12.5.3 Comparison with Baseline Methods12.5.4 Running Times, Scalability, and Memory Requirement12.5.5 Sensitivity to Parameters12.6 Summary and DirectionsReferences
13.1 Introduction13.2 Discussion of the Approaches13.2.1 MPC Ensemble Approach13.2.2 Classification and Novel Class Detection in Data Streams (ECSMiner)13.2.3 Classification with Scarcely Labeled Data (ReaSC)13.3 Extensions13.4 Summary and DirectionsReferences
Introduction to Part III
14.1 Introduction14.2 Sequence Stream Data14.3 Big Data Issues14.4 Contributions14.5 Summary and DirectionsReferences
15.1 Introduction15.2 Insider Threat Detection15.3 Stream Mining15.4 Big Data Techniques for Scalability15.5 Summary and DirectionsReferences
16.1 Introduction16.2 Ensemble Learning16.3 Ensemble for Unsupervised Learning16.4 Ensemble for Supervised Learning16.5 Summary and DirectionsReferences
17.1 Introduction17.2 Supervised Learning17.3 Unsupervised Learning17.3.1 GBAD-MDL17.3.2 GBAD-P17.3.3 GBAD-MPS17.4 Summary and DirectionsReferences
18.1 Introduction18.2 Dataset18.3 Experimental Setup18.3.1 Supervised Learning18.3.2 Unsupervised Learning18.4 Results18.4.1 Supervised Learning18.4.2 Unsupervised Learning18.5 Summary and DirectionsReferences
19.1 Introduction19.2 Classifying Sequence Data19.3 Unsupervised Stream-Based Sequence Learning (USSL)19.3.1 Construct the LZW Dictionary by Selecting the Patterns in the Data Stream19.3.2 Constructing the Quantized Dictionary19.4 Anomaly Detection19.5 Complexity Analysis19.6 Summary and DirectionsReferences
20.1 Introduction20.2 Dataset20.3 Concept Drift in the Training Set20.4 Results20.4.1 Choice of Ensemble Size20.5 Summary and DirectionsReferences
21.1 Introduction21.2 Hadoop Mapreduce Platform21.3 Scalable LZW and QD Construction Using Mapreduce Job21.3.1 2MRJ Approach21.3.2 1MRJ Approach21.4 Experimental Setup and Results21.4.1 Hadoop Cluster21.4.2 Big Dataset for Insider Threat Detection21.4.3 Results for Big Data Set Related to Insider Threat Detection21.4.3.1 On OD Dataset21.4.3.2 On DBD Dataset21.5 Summary and DirectionsReferences
22.1 Introduction22.2 Discussion22.3 Future Work22.3.1 Incorporate User Feedback22.3.2 Collusion Attack22.3.3 Additional Experiments22.3.4 Anomaly Detection in Social Network and Author Attribution22.3.5 Stream Mining as a Big Data Mining Problem22.4 Summary and DirectionsReferences
Introduction to Part IV
23.1 Introduction23.2 Our Approach23.3 Related Work23.4 Architecture23.5 Mapreduce Framework23.5.1 Overview23.5.2 Input Files Selection23.5.3 Cost Estimation for Query Processing23.5.4 Query Plan Generation23.5.5 Breaking Ties by Summary Statistics23.5.6 MapReduce Join Execution23.6 Results23.6.1 Experimental Setup23.6.2 Evaluation23.7 Security Extensions23.7.1 Access Control Model23.7.2 Access Token Assignment23.7.3 Conflicts23.8 Summary and DirectionsReferences
24.1 Introduction24.2 Our Premise24.3 Modules of Inxite24.3.1 Overview24.3.2 Information Engine24.3.2.1 Entity Extraction24.3.2.2 Information Integration24.3.3 Person of Interest Analysis24.3.3.1 InXite Person of Interest Profile Generation and Analysis24.3.3.2 InXite POI Threat Analysis24.3.3.3 InXite Psychosocial Analysis24.3.3.4 Other features24.3.4 InXite Threat Detection and Prediction24.3.5 Application of SNOD24.3.5.1 SNOD++24.3.5.2 Benefits of SNOD++24.3.6 Expert Systems Support24.3.7 Cloud-Design of Inxite to Handle Big Data24.3.8 Implementation24.4 Other Applications24.5 Related Work24.6 Summary and DirectionsReferences
25.1 Introduction25.2 Design Philosophy25.3 System Design25.3.1 Design of CAISS25.3.2 Design of CAISS++25.3.2.1 Limitations of CAISS25.3.3 Formal Policy Analysis25.3.4 Implementation Approach25.4 Related Work25.4.1 Our Related Research25.4.2 Overall Related Research25.4.3 Commercial Developments25.5 Extensions for Big Data-Based Social Media Applications25.6 Summary and DirectionsReferences
26.1 Introduction26.2 Integrating Blackbook with Amazon s326.3 Experiments26.4 Summary and DirectionsReferences
27.1 Introduction27.2 Malware Detection27.2.1 Malware Detection as a Data Stream Classification Problem27.2.2 Cloud Computing for Malware Detection27.2.3 Our Contributions27.3 Related Work27.4 Design and Implementation of the System27.4.1 Ensemble Construction and Updating27.4.2 Error Reduction Analysis27.4.3 Empirical Error Reduction and Time Complexity27.4.4 Hadoop/MapReduce Framework27.5 Malicious Code Detection27.5.1 Overview27.5.2 Nondistributed Feature Extraction and Selection27.5.3 Distributed Feature Extraction and Selection27.6 Experiments27.6.1 Datasets27.6.2 Baseline Methods27.7 Discussion27.8 Summary and DirectionsReferences
28.1 Introduction28.2 Architecture for the Inference Controller28.3 Semantic Web Technologies and Provenance28.3.1 Semantic Web-Based Models28.3.2 Graphical Models and Rewriting28.4 Inference Control through Query Modification28.4.1 Our Approach28.4.2 Domains and Provenance28.4.3 Inference Controller with Two Users28.4.4 SPARQL Query Modification28.5 Implementing the Inference Controller28.5.1 Our Approach28.5.2 Implementation of a Medical Domain28.5.3 Generating and Populating the Knowledge Base28.5.4 Background Generator Module28.6 Big Data Management and Inference Control28.7 Summary and DirectionsReferences
Introduction to Part V
29.1 Introduction29.2 Trust, Privacy, and Confidentiality29.2.1 Current Successes and Potential Failures29.2.2 Motivation for a Framework29.3 CPT Framework29.3.1 The Role of the Server29.3.2 CPT Process29.3.3 Advanced CPT29.3.4 Trust, Privacy, and Confidentiality Inference Engines29.4 Our Approach to Confidentiality Management29.5 Privacy for Social Media Systems29.6 Trust for Social Networks29.7 Integrated System29.8 CPT within the Context of Big Data and Social Networks29.9 Summary and DirectionsReferences
30.1 Overview30.2 Integrity Management and Data Provenance for Big Data Systems30.2.1 Need for Integrity30.2.2 Aspects of Integrity30.2.3 Inferencing, Data Quality, and Data Provenance30.2.4 Integrity Management, Cloud Services and Big Data30.2.5 Integrity for Big Data30.3 Design of Our Framework30.4 The Global Big Data Security and Privacy Controller30.5 Summary and DirectionsReferences
31.1 Introduction31.2 Use Cases31.3 Layered Framework for Secure IoT31.4 Protecting the Data31.5 Scalable Analytics for IoT Security Applications31.6 Summary and DirectionsReferences
32.1 Introduction32.2 Our Approach32.2.1 Challenges32.2.2 Behavioral Feature Extraction and Analysis32.2.2.1 Graph-Based Behavior Analysis32.2.2.2 Sequence-Based Behavior Analysis32.2.2.3 Evolving Data Stream Classification32.2.3 Reverse Engineering Methods32.2.4 Risk-Based Framework32.2.5 Application to Smartphones32.2.5.1 Data Gathering32.2.5.2 Malware Detection32.2.5.3 Data Reverse Engineering of Smartphone Applications32.3 Our Experimental Activities32.3.1 Covert Channel Attack in Mobile Apps32.3.2 Detecting Location Spoofing in Mobile Apps32.3.3 Large Scale, Automated Detection of SSL/TLS Man-in-the-Middle Vulnerabilities in Android Apps32.4 Infrastructure Development32.4.1 Virtual Laboratory Development32.4.1.1 Laboratory Setup32.4.1.2 Programming Projects to Support the Virtual Lab32.4.1.3 An Intelligent Fuzzier for the Automatic Android GUI Application Testing32.4.1.4 Problem Statement32.4.1.5 Understanding the Interface32.4.1.6 Generating Input Events32.4.1.7 Mitigating Data Leakage in Mobile Apps Using a Transactional Approach32.4.1.8 Technical Challenges32.4.1.9 Experimental System32.4.1.10 Policy Engine32.4.2 Curriculum Development32.4.2.1 Extensions to Existing Courses32.4.2.2 New Capstone Course on Secure Mobile Computing32.5 Summary and DirectionsReferences
33.1 Introduction33.2 Motivation33.2.1 The Problem33.2.2 Air Quality Data33.2.3 Need for Such a Case Study33.3 Methodologies33.4 The Framework Design33.4.1 Storing and Retrieving Multiple Types of Scientific Data33.4.1.1 The Problem and Challenges33.4.1.2 Current Systems and Their Limitations33.4.1.3 The Future System33.4.2 Privacy and Security Aware Data Management for Scientific Data33.4.2.1 The Problem and Challenges33.4.2.2 Current Systems and Their Limitations33.4.2.3 The Future System33.4.3 Offline Scalable Statistical Analytics33.4.3.1 The Problem and Challenges33.4.3.2 Current Systems and Their Limitations33.4.3.3 The Future System33.4.3.4 Mixed Continuous and Discrete Domains33.4.4 Real-Time Stream Analytics33.4.4.1 The Problem and Challenges33.4.5 Current Systems and Their Limitations33.4.5.1 The Future System33.5 Summary and DirectionsReferences
34.1 Introduction34.2 Current Research and Infrastructure Activities in BDMA and BDSP34.2.1 Big Data Analytics for Insider Threat Detection34.2.2 Secure Data Provenance34.2.3 Secure Cloud Computing34.2.4 Binary Code Analysis34.2.5 Cyber-Physical Systems Security34.2.6 Trusted Execution Environment34.2.7 Infrastructure Development34.3 Education and Infrastructure Program in BDMA34.3.1 Curriculum Development34.3.2 Experimental Program34.3.2.1 Geospatial Data Processing on GDELT34.3.2.2 Coding for Political Event Data34.3.2.3 Timely Health Indicator34.4 Security and Privacy for Big Data34.4.1 Our Approach34.4.2 Curriculum Development34.4.2.1 Extensions to Existing Courses34.4.2.2 New Capstone Course on BDSP34.4.3 Experimental Program34.4.3.1 Laboratory Setup34.4.3.2 Programming Projects to Support the Lab34.5 Summary and DirectionsReferences
35.1 Introduction35.2 Issues in BDSP35.2.1 Introduction35.2.2 Big Data Management and Analytics35.2.3 Security and Privacy35.2.4 Big Data Analytics for Security Applications35.2.5 Community Building35.3 Summary of Workshop Presentations35.3.1 Keynote Presentations35.3.1.1 Toward Privacy Aware Big Data Analytics35.3.1.2 Formal Methods for Preserving Privacy While Loading Big Data35.3.1.3 Authenticity of Digital Images in Social Media35.3.1.4 Business Intelligence Meets Big Data: An Overview of Security and Privacy35.3.1.5 Toward Risk-Aware Policy-Based Framework for BDSP35.3.1.6 Big Data Analytics: Privacy Protection Using Semantic Web Technologies35.3.1.7 Securing Big Data in the Cloud: Toward a More Focused and Data-Driven Approach35.3.1.8 Privacy in a World of Mobile Devices35.3.1.9 Access Control and Privacy Policy Challenges in Big Data35.3.1.10 Timely Health Indicators Using Remote Sensing and Innovation for the Validity of the Environment35.3.1.11 Additional Presentations35.3.1.12 Final Thoughts on the Presentations35.4 Summary of the Workshop Discussions35.4.1 Introduction35.4.2 Philosophy for BDSP35.4.3 Examples of Privacy-Enhancing Techniques35.4.4 Multiobjective Optimization Framework for Data Privacy35.4.5 Research Challenges and Multidisciplinary Approaches35.4.6 BDMA for Cyber Security35.5 Summary and DirectionsReferences
36.1 About This Chapter36.2 Summary of This Book36.3 Directions for BDMA and BDSP36.4 Where Do We Go from Here?

Content preview from Big Data Analytics with Applications in Insider Threat Detection

Cloud Computing and Semantic Web Technologies

5.1 Introduction

Chapters 2 through 4 have discussed concepts in data security and privacy, data mining, and data mining for cyber security. These three supporting technologies are part of the foundational technologies for the concepts discussed in this book. For example, Section II describes stream data analytics for large datasets. In particular, we discuss an innovative technique called “novel class detection” where we integrate data mining with stream data management technologies. Section III describes our approach to applying the techniques for stream mining discussing Section II for insider threat detection. We utilize the cloud platform for managing and analyzing large datasets. We will ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Start your free trial

Practical Enterprise Data Lake Insights: Handle Data-Driven Challenges in an Enterprise Big Data Lake

Publisher Resources

ISBN: 9781351645768

Big Data Analytics with Applications in Insider Threat Detection

by Bhavani Thuraisingham, Pallabi Parveen, Mohammad Mehedy Masud, Latifur Khan

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

You might also like

Practical Enterprise Data Lake Insights: Handle Data-Driven Challenges in an Enterprise Big Data Lake

Big Data Analytics for Intelligent Healthcare Management

The Enterprise Big Data Lake

Artificial Intelligence Trends for Data Analytics Using Machine Learning and Deep Learning Approaches

Publisher Resources

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,and much more.

You might also like

Practical Enterprise Data Lake Insights: Handle Data-Driven Challenges in an Enterprise Big Data Lake

Big Data Analytics for Intelligent Healthcare Management

The Enterprise Big Data Lake

Artificial Intelligence Trends for Data Analytics Using Machine Learning and Deep Learning Approaches

Publisher Resources

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.