This book is about turning data into knowledge. Data is cheap (at least relatively); knowledge is harder to come by.
I will present three related pieces:
The study of random events. Most people have an intuitive understanding of degrees of probability, which is why you can use words like “probably” and “unlikely” without special training, but we will talk about how to make quantitative claims about those degrees.
A tool that is well-suited to quantitative analysis. Computers are commonly used to process statistics. Also, computational experiments are useful for exploring concepts in probability and statistics.
The thesis of this book is that if you know how to program, you can use that skill to help you understand probability and statistics. These topics are often presented from a mathematical perspective, and that approach works well for some people. But some important ideas in this area are hard to work with mathematically and relatively easy to approach computationally.
If you Google this question, you will find plenty of discussion. Some people claim it’s true, others say it’s a myth, and some people say it’s the other way around: first babies come early.
In many of these discussions, people provide data to support their claims. I found many examples like these:
“My two friends that have given birth recently to their first babies, BOTH went almost 2 weeks overdue before going into labor or being induced.”
“My first one came 2 weeks late and now I think the second one is going to come out two weeks early!!”
“I don’t think that can be true because my sister was my mother’s first and she was early, as with many of my cousins.”
Reports like these are called anecdotal evidence because they are based on data that is unpublished and usually personal. In casual conversation, there is nothing wrong with anecdotes, so I don’t mean to pick on the people I quoted.
But we might want evidence that is more persuasive and an answer that is more reliable. By those standards, anecdotal evidence usually fails, because:
- Small number of observations
If the gestation period is longer for first babies, the difference is probably small compared to the natural variation. In that case, we might have to compare a large number of pregnancies to be sure that a difference exists.
- Selection bias
- Confirmation bias
Anecdotes are often personal stories, and often misremembered, misrepresented, repeated inaccurately, etc.
So how can we do better?
To address the limitations of anecdotes, we will use the tools of statistics, which include:
- Data collection
- Descriptive statistics
- Exploratory data analysis
- Hypothesis testing
By performing these steps with care to avoid pitfalls, we can reach conclusions that are more justifiable and more likely to be correct.
Since 1973, the U.S. Centers for Disease Control and Prevention (CDC) have conducted the National Survey of Family Growth (NSFG), which is intended to gather “information on family life, marriage and divorce, pregnancy, infertility, use of contraception, and men’s and women’s health. The survey results are used ... to plan health services and health education programs, and to do statistical studies of families, fertility, and health.”
We will use data collected by this survey to investigate whether first babies tend to come late, and other questions. In order to use this data effectively, we have to understand the design of the study.
The NSFG is a cross-sectional study, which means that it captures a snapshot of a group at a point in time. The most common alternative is a longitudinal study, which observes a group repeatedly over a period of time.
The people who participate in a survey are called respondents; a group of respondents is called a cohort. In general, cross-sectional studies are meant to be representative, which means that every member of the target population has an equal chance of participating. Of course, that ideal is hard to achieve in practice, but people who conduct surveys come as close as they can.
The NSFG is not representative; instead, it is deliberately oversampled. The designers of the study recruited three groups—Hispanics, African-Americans, and teenagers—at rates higher than their representation in the U.S. population. The reason for oversampling is to make sure that the number of respondents in each of these groups is large enough to draw valid statistical inferences.
Of course, the drawback of oversampling is that it is not as easy to draw conclusions about the general population based on statistics from the survey. We will come back to this point later.
Download the files named
2002FemPreg.dat.gz. The first is the respondent file, which contains one line for each of the 7,643 female respondents. The second file contains one line for each pregnancy reported by a respondent.
Online documentation of the survey is at http://www.icpsr.umich.edu/nsfg6. Browse the sections in the left navigation bar to get a sense of what data is included. You can also read the questionnaires at http://cdc.gov/nchs/data/nsfg/nsfg_2002_questionnaires.htm.
The web page for this book provides code to process the data files from the NSFG. Download http://thinkstats.com/survey.py and run it in the same directory you put the data files in. It should read the data files and print the number of lines in each:
Number of respondents 7643 Number of pregnancies 13593
Browse the code to get a sense of what it does. The next section explains how it works.
“Oeuf” means egg, “chapeau” means hat. It’s like those French have a different word for everything.
Each line in the respondents file contains information about one respondent. This information is called a record. The variables that make up a record are called fields. A collection of records is called a table.
If you read
survey.py, you will
see class definitions for
which is an object that represents a record, and
Table, which represents a table.
There are two subclasses of
Pregnancy—which contain records from the
respondent and pregnancy tables. For the time being, these classes are
empty; in particular, there is no init method to initialize their
attributes. Instead, we will use
Table.MakeRecord to convert a line of text
There are also two subclasses of
Pregnancies. The init method in each class
specifies the default name of the data file and the type of record to
Table object has an
records, which is a
GetFields method returns a list of tuples that
specify the fields from the record that will be stored as attributes in
Record object. (You might want
to read that last sentence twice.)
For example, here is
def GetFields(self): return [ ('caseid', 1, 12, int), ('prglength', 275, 276, int), ('outcome', 277, 277, int), ('birthord', 278, 279, int), ('finalwgt', 423, 440, float), ]
The first tuple says that the field
caseid is in columns 1 through 12 and it’s an
integer. Each tuple contains the following information:
The name of the attribute where the field will be stored. Most of the time, I use the name from the NSFG codebook, converted to all lowercase.
The index of the starting column for this field. For example, the start index for
caseidis 1. You can look up these indices in the NSFG codebook at http://nsfg.icpsr.umich.edu/cocoon/WebDocs/NSFG/public/index.htm.
The index of the ending column for this field; for example, the end index for
caseidis 12. Unlike in Python, the end index is inclusive.
- conversion function
A function that takes a string and converts it to an appropriate type. You can use built-in functions, like
float, or user-defined functions. If the conversion fails, the attribute gets the string value
’NA’. If you don’t want to convert a field, you can provide an identity function or use
For pregnancy records, we extract the following variables:
The integer ID of the respondent.
The integer duration of the pregnancy in weeks.
An integer code for the outcome of the pregnancy. The code 1 indicates a live birth.
The integer birth order of each live birth; for example, the code for a first child is 1. For outcomes other than live birth, this field is blank.
The statistical weight associated with the respondent. It is a floating-point value that indicates the number of people in the U.S. population this respondent represents. Members of oversampled groups have lower weights.
If you read the casebook carefully, you will see that most of these variables are recodes, which means that they are not part of the raw data collected by the survey, but they are calculated using the raw data.
prglength for live
births is equal to the raw variable
wksgest (weeks of gestation) if it is
available; otherwise, it is estimated using
mosgest * 4.33 (months of gestation times the
average number of weeks in a month).
Recodes are often based on logic that checks the consistency and accuracy of the data. In general it is a good idea to use recodes unless there is a compelling reason to process the raw data yourself.
You might also notice that
Pregnancies has a method called
Recode that does some additional checking and
In this exercise you will write a program to explore the data in the Pregnancies table.
import survey table = survey.Pregnancies() table.ReadRecords() print 'Number of pregnancies', len(table.records)
The result should be 13,593 pregnancies.
Write a loop that iterates
tableand counts the number of live births. Find the documentation of
outcomeand confirm that your result is consistent with the summary in the documentation.
Modify the loop to partition the live birth records into two groups, one for first babies and one for the others. Again, read the documentation of
birthordto see if your results are consistent.
When you are working with a new dataset, these kinds of checks are useful for finding errors and inconsistencies in the data, detecting bugs in your program, and checking your understanding of the way the fields are encoded.
You can download a solution to this exercise from http://thinkstats.com/first.py.
A difference like that is called an apparent effect; that is, there might be something going on, but we are not yet sure. There are several questions we still want to ask:
Is it possible that the apparent effect is due to selection bias or some other error in the experimental setup? If so, then we might conclude that the effect is an artifact; that is, something we created (by accident) rather than found.
Answering these questions will take most of the rest of this book.
The best way to learn about statistics is to work on a project you are interested in. Is there a question like, “Do first babies arrive late,” that you would like to investigate?
Think about questions you find personally interesting, items of conventional wisdom, controversial topics, or questions that have political consequences, and see if you can formulate a question that lends itself to statistical inquiry.
Look for data to help you address the question. Governments are good sources because data from public research is often freely available.Another way to find data is Wolfram Alpha, which is a curated collection of good-quality datasets at http://wolframalpha.com. Results from Wolfram Alpha are subject to copyright restrictions; you might want to check the terms before you commit yourself.
Google and other search engines can also help you find data, but it can be harder to evaluate the quality of resources on the web.
If it seems like someone has answered your question, look closely to see whether the answer is justified. There might be flaws in the data or the analysis that make the conclusion unreliable. In that case, you could perform a different analysis of the same data, or look for a better source of data.
- anecdotal evidence
- apparent effect
- cross-sectional study
- longitudinal study
A group we are interested in studying, often a group of people, but the term is also used for animals, vegetables, and minerals.
- raw data
- statistically significant
- summary statistic: