Data Analysis: What Can Be Learned From the Past 50 Years

PREFACE

“These prolegomena are not for the use of apprentices, but of future teachers, and indeed are not to help them to organize the presentation of an already existing science, but to discover the science itself for the first time.” (Immanuel Kant, transl. Gary Hatfield.)

“Diese Prolegomena sind nicht zum Gebrauch vor Lehrlinge, sondern vor künftige Lehrer, und sollen auch diesen nicht etwa dienen, um den Vortrag einer schon vorhandnen Wissenschaft anzuordnen, sondern um diese Wissenschaft selbst allererst zu erfinden.” (Immanuel Kant, 1783)

How do you learn data analysis?

First, how do you teach data analysis? This is a task going beyond “organizing the presentation of an already existing science”. At several academic institutions, we tried to teach it in class, as part of an Applied Statistics requirement. Sometimes I was actively involved, but mostly I played the part of an interested observer. In all cases we ultimately failed. Why?

I believe the main reason was: it is easy to teach data analytic techniques, but it is difficult to teach their use in actual applied situations. We could not force the students to immerse themselves into the underlying subject matter. For the students, acquiring the necessary background information simply was too demanding an effort, in particular since not every subject matter appeals to every person. What worked, at least sort of, was the brutal approach we used in the applied part of the Ph.D. qualifying exam at Harvard. We handed the students ...