Basic Concepts in
This chapter describes basic concepts in data mining, typical tasks for data
mining, and basic data structures as targets of data mining.
5.1 What is Data Mining?
First, the fundamental concepts of data mining [Han et al. 2006, Tan et al.
2006], which can be used as principal techniques for constructing hypotheses
in the analysis of social big data, will be brieﬂ y described. Data mining is, in a
nutshell, to discover frequent patterns and meaningful structures appearing
in a large amount of data used by applications. Principal techniques, such
as multivariate analysis, for validating hypotheses in social big data will
be explained in a separate chapter.
One of the basic techniques for data mining is association rule mining,
also known as association analysis. It is to discover frequent co-occurrences
between structured data used in business applications, which are usually
managed by database management systems (DBMS) such as relational
database systems. An algorithm called Apriori is used in many cases for
that purpose. For example, association rule mining discovers combinations
of items co-occurring frequently in a group of items (i.e., contents of the
shopping carts) that customers have purchased at the same time in retail
stores such as supermarkets. Association rules are made from frequent
combinations of items discovered by the algorithm. Based on association
rules, a lot of application systems recommend a set of items by revising
arrangements of them. Association rule mining is extended and applied
to the history of product purchases and the history of click streams on the
Web pages in order to discover the frequent patterns of series data. Mining
historical data is called historical data mining in particular.
Basic Concepts in Data Mining 87
On the other hand, a classiﬁ er is learned based on data whose classes
(i.e., categories) are known in advance. Then, if there is new data, classes
to which they should belong are determined by using the learned classiﬁ er.
This task called classiﬁ cation is one of the basic data mining techniques.
Naïve Bayes and decision trees are used as typical classiﬁ ers. Classiﬁ cation
is used by such a variety of applications as determination of promising
customers, detection of spam e-mails and determination of categories of new
specimens in science or medicine. Determination of continuous values such
as temperatures and stock prices is also called prediction of future values.
Prediction requires methods such as regression analysis as a basic approach
or multivariate analysis as a more advanced approach. Indeed, these
analytical approaches have been developed more or less independently
from data mining. However, they are considered a kind of extensions of
data mining and will be described as one of the key technologies for social
big data mining separately in this book. Based on a combination of two or
more existing classiﬁ ers, ensemble learning creates a more accurate classiﬁ er
than each of the original ones.
It may be possible to deﬁ ne the degrees of similarity between data even
if the categories of the data are not known in advance. The opposite concept
of similarity is dissimilarity or distance. Based on the deﬁ ned similarity,
grouping data into the same group which are similar to each other in a
collection of data is called cluster analysis or clustering, which is also one
of the basic technologies of data mining. Unlike classiﬁ cation, clustering
doesn’t demand that the names and characteristics of clusters are known
in advance. Techniques such as a hierarchical agglomerative method and
a nonhierarchical k-means method are often used for clustering. Promising
applications of clustering include discovery of groups of similar customers
A data mining task which can detect exceptional values or values
different from standard values is called outlier detection. There are methods
for outlier detection based on statistical models, data distances, and data
densities. There are alternative ways to ﬁ nd outliers using clustering and
classiﬁ cation. Outlier detection has been used for applications, such as
detection of credit card frauds or network intrusions.
5.2 Technical Issues and Related Technologies
Here the relationships between data mining and its peripheral technologies
will be summarized in order to better understand the features of data
mining. Since there are various technologies related to data mining such as
databases, information retrieval, and Web search (i.e., search engine), the
relationships between data mining and such technologies will be described