O'Reilly Media | Open Sources

Buy this Book

Print Book $24.95

Read it Now!

Print Book £17.50

Reprint Licensing

Tell a friend

Open Sources Voices from the Open Source Revolution

Edited by Chris�DiBona,�Sam�Ockman,�Mark�Stone
Price: $24.95 USD
£17.50 GBP

Cover | Table of Contents

Chapter 1: Acknowledgments: Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

No book like this happens without the help and counsel of a number of people, so I thank my Mom, Dad, Trish, Denise, Neil, and Mickey. I'd also like to thank the folks at the Coffeenet, who have been there for me. And of course my thanks to the contributors who have wasted valuable coding time to work on this book; I appreciate it!

To the people at VA Research Linux Systems, I couldn't have hoped for a better collection of smart, dedicated people; thanks for putting up with me during the writing of this book.

This book would not have happened without the continual support and dedication of Mark Stone. He is the true hero behind this book's creation. He has said that "A book could be written about how this book was written," which I'm sure he means in the nicest way possible. I'm sure one day he will look back and laugh at this time— right, Mark? Mark?

In the initial brainstorming period for the book, a number of people contributed ideas and support that eventually led to Open Sources. These people include Paul Crowley, Paul Russell, Corey Saltiel, Edward Avis, Jeff Licquia, Jeff Knox, Becky Wood, and the guy whose site (http://slashdot.org) acted as the catalyst, Rob Malda. Thanks to all; I hope this book is everything you had wished it to be.

Finally, I could not have completed this work without the morale of Christine Hillmer, who selflessly unpacked our apartment while I toiled away at my keyboard. You are all that I ever could have wished for and I am reminded each day how lucky I am.

—Chris DiBona

—

I'd like to thank the following people for the many ways that they have supported me: Joe McGuckin, Peter Hendrickson, Jo Schuster, Ruth Ockman, Allison Huynh, and Nina Woodard. I'd also like to thank my favorite hackers: David S. Miller and H. Peter Anvin.

—Sam Ockman

—

I'd like to thank Sam and Chris for bringing this crazy idea to me in the first place. I'm sure they had no idea what they were getting into, but I'm equally sure it has been worth it. I'd also like to thank each of the contributors; each is a creative spirit with more ideas than time to execute them, but each understood the importance of this project and the need to make time for it.
Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!
Chapter 1: Introduction: Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Chris DiBona, Sam Ockman, and Mark Stone

Linux creator Linus Torvalds reports that the name "Linus" was chosen for him because of his parents' admiration for Nobel laureate Linus Pauling. Pauling was the rarest of men: a scientist who won the Nobel Prize not once, but twice. We find a cautionary tale for the Open Source community in the story of Pauling's foundational work that made possible the discovery of the structure of DNA.

The actual discovery was made Francis Crick and James Watson, and is famously chronicled in Watson's book The Double Helix. Watson's book is a remarkably frank account of the way science is actually done. He recounts not just the brilliance and insight, but the politics, the competition, and the luck. The quest for the secret of DNA became a fierce competition between, among others, Watson and Crick's lab in Cambridge, and Pauling's lab at Cal Tech.

Watson describes with obvious unease the way in which Pauling came to know that Watson and Crick had solved the mystery, and created a model of DNA's helical structure. The story here centers on Max Delbruk, a mutual friend who traveled between Cambridge and Cal Tech. While sympathetic to Watson and Crick's desire to keep the discovery secret until all results could be confirmed, Delbruk's allegiance ultimately was to science itself. In this passage, Watson describes how he learned that Pauling had heard the news:

Linus Pauling first heard about the double helix from Max Delbruk. At the bottom of the letter that broke the news of the complementary chains, I had asked that he not tell Linus. I was still slightly afraid something would go wrong and did not want Pauling to think about hydrogen-bonded base pairs until we had a few more days to digest our position. My request, however, was ignored. Delbruk wanted to tell everyone in his lab and knew that within hours the gossip would travel from his lab in biology to their friends working under Linus. Also, Pauling made him promise to let him know the minute he heard from me. Then there was the even more important consideration that Delbruk hated any form of secrecy in scientific matters and did not want to keep Pauling in suspense any longer.
Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!
Prologue: Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Linux creator Linus Torvalds reports that the name "Linus" was chosen for him because of his parents' admiration for Nobel laureate Linus Pauling. Pauling was the rarest of men: a scientist who won the Nobel Prize not once, but twice. We find a cautionary tale for the Open Source community in the story of Pauling's foundational work that made possible the discovery of the structure of DNA.

The actual discovery was made Francis Crick and James Watson, and is famously chronicled in Watson's book The Double Helix. Watson's book is a remarkably frank account of the way science is actually done. He recounts not just the brilliance and insight, but the politics, the competition, and the luck. The quest for the secret of DNA became a fierce competition between, among others, Watson and Crick's lab in Cambridge, and Pauling's lab at Cal Tech.

Watson describes with obvious unease the way in which Pauling came to know that Watson and Crick had solved the mystery, and created a model of DNA's helical structure. The story here centers on Max Delbruk, a mutual friend who traveled between Cambridge and Cal Tech. While sympathetic to Watson and Crick's desire to keep the discovery secret until all results could be confirmed, Delbruk's allegiance ultimately was to science itself. In this passage, Watson describes how he learned that Pauling had heard the news:

Linus Pauling first heard about the double helix from Max Delbruk. At the bottom of the letter that broke the news of the complementary chains, I had asked that he not tell Linus. I was still slightly afraid something would go wrong and did not want Pauling to think about hydrogen-bonded base pairs until we had a few more days to digest our position. My request, however, was ignored. Delbruk wanted to tell everyone in his lab and knew that within hours the gossip would travel from his lab in biology to their friends working under Linus. Also, Pauling made him promise to let him know the minute he heard from me. Then there was the even more important consideration that Delbruk hated any form of secrecy in scientific matters and did not want to keep Pauling in suspense any longer.
Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!
What Is Free Software and How Does It Relate to Open Source?: Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

In 1984, Richard Stallman, a researcher at the MIT AI Lab, started the GNU project. The GNU project's goal was, simply put, to make it so that no one would ever have to pay for software. Stallman launched the GNU project because essentially he feels that the knowledge that constitutes a running program—what the computer industry calls the source code—should be free. If it were not, Stallman reasons, a very few, very powerful people would dominate computing.

Where proprietary commercial software vendors saw an industry guarding trade secrets that must be tightly protected, Stallman saw scientific knowledge that must be shared and distributed. The basic tenet of the GNU project and the Free Software Foundation (the umbrella organization for the GNU project) is that source code is fundamental to the furthering of computer science and freely available source code is truly necessary for innovation to continue.

Stallman worried how the world would react to free software. Scientific knowledge is often in the public domain; it is one function of academic publishing to put it there. With software, however, it was clear that just letting the source code go into the public domain would tempt businesses to co-opt the code for their own profitability. Stallman's answer to this threat was the GNU General Public License, known as the GPL (see Appendix B).

The GPL basically says that you may copy and distribute the software licensed under the GPL at will, provided you do not inhibit others from doing the same, either by charging them for the software itself or by restricting them through further licensing. The GPL also requires works derived from work licensed under the GPL to be licensed under the GPL as well.

When Stallman and others in this book talk about free software, they are really talking about free speech. English handles the distinction here poorly, but it is the distinction between gratis and liberty, as in "Free as in speech, not as in beer." This radical message (the freedom part, not the beer part) led many software companies to reject free software outright. After all, they are in the business of making money, not adding to our body of knowledge. For Stallman, this rift between the computer industry and computer science was acceptable, maybe even desirable.
Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!
What Is Open Source Software?: Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

In the spring of 1997, a group of leaders in the free software community assembled in California. This group included Eric Raymond, Tim O'Reilly, and VA Research president Larry Augustin, among others. Their concern was to find a way to promote the ideas surrounding free software to people who had formerly shunned the concept. They were concerned that the Free Software Foundation's anti-business message was keeping the world at large from really appreciating the power of free software.

At Eric Raymond's insistence, the group agreed that what they lacked in large part was a marketing campaign, a campaign devised to win mind share, and not just market share. Out of this discussion came a new term to describe the software they were promoting: Open Source. A series of guidelines were crafted to describe software that qualified as Open Source.

Bruce Perens had laid much of the groundwork for the Open Source Definition. One of the GNU project's states goals was to create a freely available operating system that could serve as the platform for running GNU software. In a classic case of software bootstrapping, Linux had become that platform, and Linux had been created with the help of GNU tools. Perens had headed the Debian project, which managed a distribution of Linux that included within the distribution only software that adhered to the spirit of GNU. Perens had laid this out explicitly in a document called the "Debian Social Contract." The Open Source definition is a direct descendant of the "Debian Social Contract," and thus Open Source is very much in the spirit of GNU.

The Open Source Definition allows greater liberties with licensing than the GPL does. In particular, the Open Source Definition allows greater promiscuity when mixing proprietary and open-source software.

Consequently, an Open Source license could conceivably allow the use and redistribution of open-source software without compensation or even credit. As an example you can take great swaths of the Netscape browser source code and distribute it with another, possibly proprietary, program without even notifying Netscape. Why would Netscape wish this? For a number of reasons, but the most compelling is that it gets greater market share for their client code, which works very well with their commercial offerings. In this way, giving away source code is a very good way to build a platform. This is also one of the reasons why the people at Netscape did not use the GPL.
Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!
The Dark Side of the Force: Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Though he may not have realized it at the time, Watson stood at the threshold of a new era in biological science. At the time of the discovery of the double helix, science in biology and chemistry was essentially a craft, a practical art. It was practiced by a few men working in small groups, primarily under the auspices of academic research. The seeds of change had already been planted, however. With the advent of several medical breakthroughs, notably the polio vaccine and the discovery of penicillin, biological science was about to become an industry.

Today organic chemistry, molecular biology, and basic medical research are not practiced as a craft by a small body of practitioners, but pursued as an industry. While research continues in academia, the vast majority of researchers, and the vast majority of research dollars, belong to the pharmaceutical industry. This alliance between science and industry is an uneasy one at best. While pharmaceutical companies can fund research at a level undreamed of in academic institutions, they also fund research with a vested interest. Consider: would a pharmaceutical company rather put major funding into research for a cure for an illness that is therapy-based or medication-based?

Computer science, too, must exist in an uneasy alliance with industry. Once new ideas came primarily from academic computer scientists; now the computer industry drives innovation forward. While the rank and file of Open Source programmers are still the many computer science undergrads and graduate students around the world, more and more Open Source programmers are working in industry rather than academic settings.

Industry has produced some marvelous innovations: Ethernet, the mouse, and the Graphical User Interface (GUI) all came out of Xerox PARC. But there is an ominous side to the computer industry as well. No one outside of Redmond really thinks that it is a good idea for Microsoft to dictate, to the extent they do, what a computer desktop should look like or have on it.

Industry can have a negative impact on innovation. The Graphical Image Manipulation Program (GIMP) languished incomplete for a year at beta release 0.9. Its creators, two students at Berkeley, had left school to take jobs in industry, and left their innovation behind.
Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!
Use the Source, Luke: Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Open Source was not an idea decreed from the top. The Open Source movement is a genuine grass roots revolution. While evangelists like Eric Raymond and Bruce Perens have had great success changing the language around free software, that change would have been impossible if the conditions were not right. We have reached the stage where an entire generation of students who learned computer science under the influence of GNU is now at work in industry, and have quietly been bringing free software in through the back doors of industry for years. They do so not from altruistic motives, but rather to bring better code to their work.

The revolutionaries are in place. They are the network engineers, system administrators, and programmers who have thrived on open-source software throughout their education, and want to use open-source software to thrive professionally as well. Free software has become a vital part of many companies, often unwittingly, but in some cases quite deliberately. Open Source has come of age: there is such a thing as an Open Source business model.

Bob Young's company, Red Hat Software, Inc., thrives on giving away its core product: Red Hat Linux. One good way to deliver free software is to package it as a full-featured distribution with a nice manual. Young is primarily selling convenience, as most do not want to have to bother with downloading all the pieces that make up a full-featured Linux system.

But he is not the only one doing this. So why does Red Hat dominate the U.S. market? Why does SuSE Linux dominate Europe? Open-source software is a commodity market. In any commodity market, customers value a brand they can trust. Red Hat's strength comes from brand management: consistent marketing and community outreach that makes the community recommend them when their friends ask them which distribution to use. The same is true for SuSE, and the two companies own their respective markets mostly because they were first to take brand management seriously.

Supporting the community is essential. Red Hat, SuSE, and other companies in the Linux space understand that to just make money off of Linux without giving anything back would cause two problems. First, people would consider such a company a freeloader and would recommend a competitor instead. Second, a company must be able to differentiate itself from competitors. Companies like CheapBytes and Linux Central merely provide low-cost distribution, selling CDs for as little as a dollar. For Red Hat to be perceived as offering greater value than these budget distributors, Red Hat must give something back. In a wonderful irony of the Open Source model, Red Hat can afford to charge $49.95 for their distribution only because they support the development of new code and return that code to the community at large as Open Source.
Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!
Innovation Through the Scientific Method: Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

The most fascinating development in the Open Source movement today is not the success of companies like Red Hat or Sendmail Inc. What's intriguing is to see major corporations within the computer industry, companies like IBM and Oracle, turn their attention to Open Source as a business opportunity. What are they looking for in Open Source?

Innovation.

Science is ultimately an Open Source enterprise. The scientific method rests on a process of discovery, and a process of justification. For scientific results to be justified, they must be replicable. Replication is not possible unless the source is shared: the hypothesis, the test conditions, and the results. The process of discovery can follow many paths, and at times scientific discoveries do occur in isolation. But ultimately the process of discovery must be served by sharing information: enabling other scientists to go forward where one cannot; pollinating the ideas of others so that something new may grow that otherwise would not have been born.

Where scientists talk of replication, Open Source programmers talk of debugging. Where scientists talk of discovering, Open Source programmers talk of creating. Ultimately, the Open Source movement is an extension of the scientific method, because at the heart of the computer industry lies computer science. Consider the words of Grace Hopper, inventor of the compiler, who said, in the early 60s:

To me programming is more than an important practical art. It is also a gigantic undertaking in the foundations of knowledge.

Computer science, though, differs fundamentally from all other sciences. Computer science has only one means of enabling peers to replicate results: share the source code. To demonstrate the validity of a program to someone, you must provide them with the means to compile and run the program.

Replication makes scientific results robust. One scientist cannot expect to account for all possible test conditions, nor necessarily have the test environment to fully test every aspect of a hypothesis. By sharing hypotheses and results with a community of peers, the scientist enables many eyes to see what one pair of eyes might miss. In the Open Source development model, this same principle is expressed as "Given enough eyes, all bugs are shallow." By sharing source code, Open Source developers make software more robust. Programs get used and tested in a wider variety of contexts than one programmer could generate, and bugs get uncovered that otherwise would not be found. Because source code is provided, bugs can often be removed, not just discovered, by someone who otherwise would be outside the development process.
Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!
Perils to Open Source: Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Most software ventures, like most scientific enterprises, fail. As Bob Young points out, making successful open-source software is not so very different from making successful proprietary software. In both cases real success is rare, and the best innovators are those who learn from mistakes.

The rampant creativity that leads to innovation in both science and software comes at a cost. Maintaining control of an active Open Source project can be difficult. This fear of losing control prevents some individuals and many companies from active participation. Specifically, one concern when embarking or joining an open-source software project is that a large competitor or group of people will come in and create what is called a fork in the code base. Much like a fork in the road, a code base can at times diverge into two separate, incompatible, roads and never the twain shall meet. This is not an idle problem; look, for instance, at the multiple forks that the BSD-based operating systems have taken, leading to NetBSD, OpenBSD, FreeBSD, and many others. What is to keep this from happening to Linux?

One thing that keeps this happening is the open methods used in the development of the Linux kernel. Linus Torvalds, Alan Cox, and the rest run a tight ship and are the central authority for adding and accessing the kernel. The Linux kernel project has been called a benign dictatorship, with Linus as its dictator, and so far this model has produced a nicely written tight kernel without too much extraneous cruft in it.

What's ironic is that while Linux has experienced little actual forking, there exist large patches that convert the Linux kernel into a hard real-time kernel, suitable for tight critical device control, and additionally there exist versions of Linux that can run under dramatically weird architectures. These patches could be considered forks, as they are based on a set kernel and grow from there out, but because they occupy special niche application areas for Linux, they do not have a fracturing effect on the Linux community as a whole.
Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!
Motivating the Open Source Hacker: Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

The Lucid experience shows that programmers often have loyalty to a project that goes beyond direct compensation for working on the project. Why do people write free software? Why do they give away freely what they could charge hundreds of dollars an hour for? What do they get out of it?

The motivation is not just altruism. The contributors here may not be loaded down with Microsoft stock options, but each has achieved a reputation that should assure him opportunities that pay the rent and feeds the kids. From the outside this can seem like a paradox; you can't eat free software after all. The answer lies, in part, in thinking beyond conventional notions of work and compensation. We are witnessing a new economic model take shape, not just a new culture.

Eric Raymond has become a kind of self-appointed participant anthropologist to the Open Source community, and in his writings he touches on the reasons why the people develop software only to give it away.

Keep in mind that these people have been, for the most part, coding for years, and don't see programming itself as burdensome, or as work. A very complex project like Apache or the Linux kernel brings the satisfaction of the ultimate in intellectual exercise. Much like the rush a runner feels while running a race, a true programmer will feel this same rush after writing a perfect routine or tight piece of code. It is difficult to describe the joy felt after completing or debugging a hideously tricky piece of recursive code that has been a source of trouble for days.

The point is that many programmers code because it is what they love to do, and in fact it is how they define their intellect. Without coding, a programmer feels less of a person, much like an athlete deprived of an opportunity to compete. Discipline can be a problem with programmers as much as with athletes; many programmers really don't enjoy maintaining a piece of code after having mastered it.

Still, other programmers don't take this "macho" view of their craft, and take a more scholarly view. Many programmers consider themselves, rightly, to be scientists. Scientists aren't supposed to hoard profits from their inventions, they are supposed to publish and share their inventions for all to benefit from. A scientist isn't supposed to let profits come at the expense of the pursuit of knowledge.
Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!
The Venture and Investment Future of Linux: Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

A hacker's motivations may be intellectual, but the result need not be a lifestyle of sacrifice. More and more, enterprises and individual Open Source programmers are coming together in a new spirit of pragmatism and opportunity.

In Silicon Valley, where we work and live, there is a history of investment and venture capital that powers the economy of the region. It can be traced back to the years when the transistor was first being exploited for commercial gain and the microprocessor became a force in the industry, replacing cumbersome logic boards with thousands of chips and individual transistors on them.

At any given time there is a hot technology that venture capital concentrates on. While they don't ignore other companies and opportunities, they realize that to achieve their benchmarks of economic performance, they don't just need successful companies, they need hot companies that can do an Initial Public Offering (IPO) within three years of investment or be sold for hundreds of millions to companies like Oracle or Cisco.

In 1998, the great wave of the Internet is ebbing; the rash of Internet IPOs that began with Netscape's spectacular debut has begun to decline. In a fitting act of symbolism, America Online's purchase of Netscape really does signal the end of an era. Internet stocks are now looked at more closely by the investment community, and generally Internet companies are held to the same standards as other companies: they must have some plausible expectation of profitability ahead.

So where will venture go? Our guess is that Linux and open-source software related companies are and will be the hot investment through the end of the millennium. With any luck, you will see a great rash of Linux and Open Source IPOs starting with Red Hat software in late 1999. The money out there to be invested is nothing less than staggering, and companies like Scriptics, Sendmail, and Vix.com are well poised to take advantage of the favorable market conditions and build their dream companies.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!
Science and the New Renaissance: Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

The Open Source development model of today has its roots in the academic computer science of a decade or more ago. What makes Open Source dramatically more successful today, however, is the rapid dissemination of information made possible by the Internet. When Watson and Crick discovered the double helix, they could reasonably expect the information to travel from Cambridge to Cal Tech in a matter of days, or weeks at most. Today the transmission of such information is effectively instantaneous. Open Source has been born into a digital renaissance made possible by the Internet, just as modern science was made possible during the Renaissance by the invention of the printing press.

The Middle Ages lacked an affordable information infrastructure. Written works had to be copied by hand at great expense, and hence the information had to have an immediate value attached to it. Trade records, banking transactions, diplomatic correspondence; this information was concise enough and carried enough immediate value to be transmitted. The speculative writings of alchemists, priests, and philosophers—the men who would later be called scientists—took a much lower priority, and hence the information was disseminated much more slowly. The printing press changed all this by dramatically lowering the barriers to entry in the information infrastructure. Scholars who had previously worked in isolation could, for the first time, establish a sense of community with other scholars all over Europe. But this exercise in community building required an absolute commitment to the open sharing of information.

What was born out of this community was the notion of academic freedom, and ultimately the process we now call the Scientific Method. None of this would have been possible without the need to form community, and the open sharing of information has, for centuries, been the cement that has held the scientific community together.

Imagine for a moment if Newton had withheld his laws of motion, and instead gone into business as a defense contractor to artillerists following the 30 Years War. "No, I won't tell you how I know about parabolic trajectories, but I'll calibrate your guns for a fee." The very idea, of course, sounds absurd. Not only did science not evolve this way, but it could not have evolved this way. If that had been the mindset of those scientifically inclined, their very secrecy would have kept science from developing and evolving at all.
Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!
Chapter 2: A Brief History of Hackerdom: Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Eric S. Raymond

In the beginning, there were Real Programmers.

That's not what they called themselves. They didn't call themselves "hackers," either, or anything in particular; the sobriquet "Real Programmer" wasn't coined until after 1980. But from 1945 onward, the technology of computing attracted many of the world's brightest and most creative minds. From Eckert and Mauchly's ENIAC onward there was a more or less continuous and self-conscious technical culture of enthusiast programmers, people who built and played with software for fun.

The Real Programmers typically came out of engineering or physics backgrounds. They wore white socks and polyester shirts and ties and thick glasses and coded in machine language and assembler and FORTRAN and half a dozen ancient languages now forgotten. These were the hacker culture's precursors, the largely unsung protagonists of its prehistory.

From the end of World War II to the early 1970s, in the great days of batch computing and the "big iron" mainframes, the Real Programmers were the dominant technical culture in computing. A few pieces of revered hacker folklore date from this era, including the well-known story of Mel (included in the Jargon File), various lists of Murphy's Laws, and the mock-German "Blinkenlights" poster that still graces many computer rooms.

Some people who grew up in the "Real Programmer"' culture remained active into the 1990s. Seymour Cray, designer of the Cray line of supercomputers, is said to have once toggled an entire operating system of his own design into a computer of his own design. In octal. Without an error. And it worked. Real Programmer macho supremo.

On a lighter note, Stan Kelly-Bootle, author of The Devil's DP Dictionary (McGraw-Hill, 1981) and folklorist extraordinaire, programmed on the Manchester Mark I, the first fully-operational stored-program digital computer, in 1948. Nowadays he writes technical humor columns for computing magazines which often take the form of a vigorous and knowing dialogue with today's hacker culture.
Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!
Prologue: The Real Programmers: Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

In the beginning, there were Real Programmers.

That's not what they called themselves. They didn't call themselves "hackers," either, or anything in particular; the sobriquet "Real Programmer" wasn't coined until after 1980. But from 1945 onward, the technology of computing attracted many of the world's brightest and most creative minds. From Eckert and Mauchly's ENIAC onward there was a more or less continuous and self-conscious technical culture of enthusiast programmers, people who built and played with software for fun.

The Real Programmers typically came out of engineering or physics backgrounds. They wore white socks and polyester shirts and ties and thick glasses and coded in machine language and assembler and FORTRAN and half a dozen ancient languages now forgotten. These were the hacker culture's precursors, the largely unsung protagonists of its prehistory.

From the end of World War II to the early 1970s, in the great days of batch computing and the "big iron" mainframes, the Real Programmers were the dominant technical culture in computing. A few pieces of revered hacker folklore date from this era, including the well-known story of Mel (included in the Jargon File), various lists of Murphy's Laws, and the mock-German "Blinkenlights" poster that still graces many computer rooms.

Some people who grew up in the "Real Programmer"' culture remained active into the 1990s. Seymour Cray, designer of the Cray line of supercomputers, is said to have once toggled an entire operating system of his own design into a computer of his own design. In octal. Without an error. And it worked. Real Programmer macho supremo.

On a lighter note, Stan Kelly-Bootle, author of The Devil's DP Dictionary (McGraw-Hill, 1981) and folklorist extraordinaire, programmed on the Manchester Mark I, the first fully-operational stored-program digital computer, in 1948. Nowadays he writes technical humor columns for computing magazines which often take the form of a vigorous and knowing dialogue with today's hacker culture.
Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!
The Early Hackers: Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

The beginnings of the hacker culture as we know it today can be conveniently dated to 1961, the year MIT acquired the first PDP-1. The Signals and Power committee of MIT's Tech Model Railroad Club adopted the machine as their favorite tech-toy and invented programming tools, slang, and an entire surrounding culture that is still recognizably with us today. These early years have been examined in the first part of Steven Levy's book Hackers (Anchor/Doubleday, 1984).

MIT's computer culture seems to have been the first to adopt the term "hacker." The TMRC's hackers became the nucleus of MIT's Artificial Intelligence Laboratory, the world's leading center of AI research into the early 1980s. And their influence was spread far wider after 1969, the first year of the ARPAnet.

The ARPAnet was the first transcontinental, high-speed computer network. It was built by the Defense Department as an experiment in digital communications, but grew to link together hundreds of universities and defense contractors and research laboratories. It enabled researchers everywhere to exchange information with unprecedented speed and flexibility, giving a huge boost to collaborative work and tremendously increasing both the pace and intensity of technological advance.

But the ARPAnet did something else as well. Its electronic highways brought together hackers all over the U.S. in a critical mass; instead of remaining in isolated small groups each developing their own ephemeral local cultures, they discovered (or re-invented) themselves as a networked tribe.

The first intentional artifacts of hackerdom—the first slang lists, the first satires, the first self-conscious discussions of the hacker ethic—all propagated on the ARPAnet in its early years. (The first version of the Jargon File, as a major example, dated from 1973.) Hackerdom grew up at the universities connected to the Net, especially (though not exclusively) in their computer science departments.

Culturally, MIT's AI Lab was first among equals from the late 1960s. But Stanford University's Artificial Intelligence Laboratory (SAIL) and (later) Carnegie-Mellon University (CMU) became nearly as important. All were thriving centers of computer science and AI research. All attracted bright people who contributed great things to hackerdom, on both the technical and folkloric levels.
Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!
The Rise of Unix: Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Meanwhile, however, off in the wilds of New Jersey, something else had been going on since 1969 that would eventually overshadow the PDP-10 tradition. The year of ARPAnet's birth was also the year that a Bell Labs hacker named Ken Thompson invented Unix.

Thompson had been involved with the development work on a time-sharing OS called Multics, which shared common ancestry with ITS. Multics was a test-bed for some important ideas about how the complexity of an operating system could be hidden inside it, invisible to the user and even to most programmers. The idea was to make using Multics from the outside (and programming for it!) much simpler, so that more real work could get done.

Bell Labs pulled out of the project when Multics displayed signs of bloating into an unusable white elephant (the system was later marketed commercially by Honeywell but never became a success). Ken Thompson missed the Multics environment, and began to play at implementing a mixture of its ideas and some of his own on a scavenged DEC PDP-7.

Another hacker named Dennis Ritchie invented a new language called "C" for use under Thompson's embryonic Unix. Like Unix, C was designed to be pleasant, unconstraining, and flexible. Interest in these tools spread at Bell Labs, and they got a boost in 1971 when Thompson and Ritchie won a bid to produce what we'd now call an office-automation system for internal use there. But Thompson and Ritchie had their eye on a bigger prize.

Traditionally, operating systems had been written in tight assembler to extract the absolute highest efficiency possible out of their host machines. Thompson and Ritchie were among the first to realize that hardware and compiler technology had become good enough that an entire operating system could be written in C, and by 1974 the whole environment had been successfully ported to several machines of different types.

This had never been done before, and the implications were enormous. If Unix could present the same face, the same capabilities, on machines of many different types, it could serve as a common software environment for all of them. No longer would users have to pay for complete new designs of software every time a machine went obsolete. Hackers could carry around software toolkits between different machines, rather than having to re-invent the equivalents of fire and the wheel every time.
Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!
The End of Elder Days: Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

So matters stood in 1980: three cultures, overlapping at the edges but organized around very different technologies. The ARPAnet/PDP-10 culture, wedded to LISP and MACRO and TOPS-10 and ITS. The Unix and C crowd with their PDP-11s and VAXen and pokey telephone connections. And an anarchic horde of early microcomputer enthusiasts bent on taking computer power to the people.

Among these, the ITS culture could still claim pride of place. But storm clouds were gathering over the Lab. The PDP-10 technology ITS depended on was aging, and the Lab itself was split into factions by the first attempts to commercialize AI technology. Some of the Lab's (and SAIL's and CMU's) best were lured away to high-paying jobs at startup companies.

The death blow came in 1983, when DEC cancelled its follow-on to the PDP-10 in order to concentrate on the PDP-11 and VAX lines. ITS no longer had a future. Because it wasn't portable, it was more effort than anyone could afford to move ITS to new hardware. The Berkeley variant of Unix running on a VAX became the hacking system par excellence, and anyone with an eye on the future could see that microcomputers were growing in power so rapidly that they were likely to sweep all before them.

It was around this time that Levy wrote Hackers. One of his prime informants was Richard M. Stallman (inventor of Emacs), a leading figure at the Lab and its most fanatical holdout against the commercialization of Lab technology.

Stallman (who is usually known by his initials and login name, RMS) went on to form the Free Software Foundation and dedicate himself to producing high-quality free software. Levy eulogized him as "the last true hacker," a description that happily proved incorrect.

Stallman's grandest scheme neatly epitomized the transition hackerdom underwent in the early 80s—in 1982 he began the construction of an entire clone of Unix, written in C and available for free. Thus, the spirit and tradition of ITS was preserved as an important part of the newer, Unix- and VAX-centered hacker culture.
Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!
The Proprietary Unix Era: Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

By 1984, when AT&T divested and Unix became a commercial product for the first time, the most important fault line in hackerdom was between a relatively cohesive "network nation" centered around the Internet and Usenet (and mostly using minicomputer- or workstation-class machines running Unix), and a vast disconnected hinterland of microcomputer enthusiasts.

The workstation-class machines built by Sun and others opened up new worlds for hackers. They were built to do high-performance graphics and pass around shared data over a network. During the 1980s, hackerdom was preoccupied by the software and tool-building challenges of getting the most use out of these features. Berkeley Unix developed built-in support for the ARPAnet protocols, which offered a solution to the networking problem and encouraged further growth of the Internet.

There were several attempts to tame workstation graphics. The one that prevailed was the X Window System. A critical factor in its success was that the X developers were willing to give the sources away for free in accordance with the hacker ethic, and were able to distribute them over the Internet. X's victory over proprietary graphics systems (including one offered by Sun itself) was an important harbinger of changes which, a few years later, would profoundly affect Unix itself.

There was a bit of factional spleen still vented occasionally in the ITS/Unix rivalry (mostly from the ex-ITSers' side). But the last ITS machine shut down for good in 1990; the zealots no longer had a place to stand and mostly assimilated to the Unix culture with various degrees of grumbling.

Within networked hackerdom itself, the big rivalry of the 1980s was between fans of Berkeley Unix and the AT&T versions. Occasionally you can still find copies of a poster from that period, showing a cartoony X-wing fighter out of the Star Wars movies streaking away from an exploding Death Star patterned on the AT&T logo. Berkeley hackers liked to see themselves as rebels against soulless corporate empires. AT&T Unix never caught up with BSD/Sun in the marketplace, but it won the standards wars. By 1990, AT&T and BSD versions were becoming harder to tell apart, having adopted many of each others' innovations.
Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!
The Early Free Unixes: Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Into the gap left by the HURD's failure had stepped a Helsinki University student named Linus Torvalds. In 1991 he began developing a free Unix kernel for 386 machines using the Free Software Foundation's toolkit. His initial, rapid success attracted many Internet hackers to help him develop Linux, a full-featured Unix with entirely free and redistributable sources.

Linux was not without competitors. In 1991, contemporaneously with Linus Torvalds's early experiments, William and Lynne Jolitz were experimentally porting the BSD Unix sources to the 386. Most observers comparing BSD technology with Linus's crude early efforts expected that BSD ports would become the most important free Unixes on the PC.

The most important feature of Linux, however, was not technical but sociological. Until the Linux development, everyone believed that any software as complex as an operating system had to be developed in a carefully coordinated way by a relatively small, tightly-knit group of people. This model was and still is typical of both commercial software and the great freeware cathedrals built by the Free Software Foundation in the 1980s; also of the freeBSD/netBSD/OpenBSD projects that spun off from the Jolitzes' original 386BSD port.

Linux evolved in a completely different way. From nearly the beginning, it was rather casually hacked on by huge numbers of volunteers coordinating only through the Internet. Quality was maintained not by rigid standards or autocracy but by the naively simple strategy of releasing every week and getting feedback from hundreds of users within days, creating a sort of rapid Darwinian selection on the mutations introduced by developers. To the amazement of almost everyone, this worked quite well.

By late 1993, Linux could compete on stability and reliability with many commercial Unixes, and hosted vastly more software. It was even beginning to attract ports of commercial applications software. One indirect effect of this development was to kill off most of the smaller commercial Unix vendors—without developers and hackers to sell to, they folded. One of the few survivors, BSDI (Berkeley Systems Design, Incorporated), flourished by offering full sources with its BSD-based Unix and cultivating close ties with the hacker community.
Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!
The Great Web Explosion: Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

The early growth of Linux synergized with another phenomenon: the public discovery of the Internet. The early 1990s also saw the beginnings of a flourishing Internet-provider industry, selling connectivity to the public for a few dollars a month. Following the invention of the World Wide Web, the Internet's already-rapid growth accelerated to a breakneck pace.

By 1994, the year Berkeley's Unix development group formally shut down, several different free Unix versions (Linux and the descendants of 386BSD) served as the major focal points of hacking activity. Linux was being distributed commercially on CD-ROM and selling like hotcakes. By the end of 1995, major computer companies were beginning to take out glossy advertisements celebrating the Internet-friendliness of their software and hardware!

In the late 1990s the central activities of hackerdom became Linux development and the mainstreaming of the Internet. The World Wide Web has at last made the Internet into a mass medium, and many of the hackers of the 1980s and early 1990s launched Internet Service Providers selling or giving access to the masses.

The mainstreaming of the Internet has even brought the hacker culture the beginnings of mainstream respectability and political clout. In 1994 and 1995, hacker activism scuppered the Clipper proposal, which would have put strong encryption under government control. In 1996 hackers mobilized a broad coalition to defeat the misnamed "Communications Decency Act" (CDA) and prevent censorship of the Internet.

With the CDA victory, we pass out of history into current events. We also pass into a period in which your historian became an actor rather than just an observer. This narrative will continue in "The Revenge of the Hackers."

Benjamin Franklin Bache, in a Philadelphia Aurora editorial, 1794

All governments are more or less combinations against the people . . . and as rulers have no more virtue than the ruled . . . the power of government can only be kept within its constituted bounds by the display of a power equal to itself, the collected sentiment of the people.
Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!
Chapter 3: Twenty Years of Berkeley Unix: From AT&T-Owned to Freely Redistributable: Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Marshall Kirk McKusick

Ken Thompson and Dennis Ritchie presented the first Unix paper at the Symposium on Operating Systems Principles at Purdue University in November 1973. Professor Bob Fabry, of the University of California at Berkeley, was in attendance and immediately became interested in obtaining a copy of the system to experiment with at Berkeley.

At the time, Berkeley had only large mainframe computer systems doing batch processing, so the first order of business was to get a PDP-11/45 suitable for running with the then-current Version 4 of Unix. The Computer Science Department at Berkeley, together with the Mathematics Department and the Statistics Department, were able to jointly purchase a PDP-11/45. In January 1974, a Version 4 tape was delivered and Unix was installed by graduate student Keith Standiford.

Although Ken Thompson at Purdue was not involved in the installation at Berkeley as he had been for most systems up to that time, his expertise was soon needed to determine the cause of several strange system crashes. Because Berkeley had only a 300-baud acoustic-coupled modem without auto answer capability, Thompson would call Standiford in the machine room and have him insert the phone into the modem; in this way Thompson was able to remotely debug crash dumps from New Jersey.

Many of the crashes were caused by the disk controller's inability to reliably do overlapped seeks, contrary to the documentation. Berkeley's 11/45 was among the first systems that Thompson had encountered that had two disks on the same controller! Thompson's remote debugging was the first example of the cooperation that sprang up between Berkeley and Bell Labs. The willingness of the researchers at the Labs to share their work with Berkeley was instrumental in the rapid improvement of the software available at Berkeley.

Though Unix was soon reliably up and running, the coalition of Computer Science, Mathematics, and Statistics began to run into problems; Math and Statistics wanted to run DEC's RSTS system. After much debate, a compromise was reached in which each department would get an eight-hour shift; Unix would run for eight hours followed by sixteen hours of RSTS. To promote fairness, the time slices were rotated each day. Thus, Unix ran 8 a.m. to 4 p.m. one day, 4 p.m. to midnight the next day, and midnight to 8 a.m. the third day. Despite the bizarre schedule, students taking the Operating Systems course preferred to do their projects on Unix rather than on the batch machine.
Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!
Early History: Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Ken Thompson and Dennis Ritchie presented the first Unix paper at the Symposium on Operating Systems Principles at Purdue University in November 1973. Professor Bob Fabry, of the University of California at Berkeley, was in attendance and immediately became interested in obtaining a copy of the system to experiment with at Berkeley.

At the time, Berkeley had only large mainframe computer systems doing batch processing, so the first order of business was to get a PDP-11/45 suitable for running with the then-current Version 4 of Unix. The Computer Science Department at Berkeley, together with the Mathematics Department and the Statistics Department, were able to jointly purchase a PDP-11/45. In January 1974, a Version 4 tape was delivered and Unix was installed by graduate student Keith Standiford.

Although Ken Thompson at Purdue was not involved in the installation at Berkeley as he had been for most systems up to that time, his expertise was soon needed to determine the cause of several strange system crashes. Because Berkeley had only a 300-baud acoustic-coupled modem without auto answer capability, Thompson would call Standiford in the machine room and have him insert the phone into the modem; in this way Thompson was able to remotely debug crash dumps from New Jersey.

Many of the crashes were caused by the disk controller's inability to reliably do overlapped seeks, contrary to the documentation. Berkeley's 11/45 was among the first systems that Thompson had encountered that had two disks on the same controller! Thompson's remote debugging was the first example of the cooperation that sprang up between Berkeley and Bell Labs. The willingness of the researchers at the Labs to share their work with Berkeley was instrumental in the rapid improvement of the software available at Berkeley.

Though Unix was soon reliably up and running, the coalition of Computer Science, Mathematics, and Statistics began to run into problems; Math and Statistics wanted to run DEC's RSTS system. After much debate, a compromise was reached in which each department would get an eight-hour shift; Unix would run for eight hours followed by sixteen hours of RSTS. To promote fairness, the time slices were rotated each day. Thus, Unix ran 8 a.m. to 4 p.m. one day, 4 p.m. to midnight the next day, and midnight to 8 a.m. the third day. Despite the bizarre schedule, students taking the Operating Systems course preferred to do their projects on Unix rather than on the batch machine.
Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!
Early Distributions: Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Meanwhile, interest in the error recovery work in the Pascal compiler brought in requests for copies of the system. Early in 1977, Joy put together the "Berkeley Software Distribution." This first distribution included the Pascal system, and, in an obscure subdirectory of the Pascal source, the editor ex. Over the next year, Joy, acting in the capacity of distribution secretary, sent out about thirty free copies of the system.

With the arrival of some ADM-3a terminals offering screen-addressable cursors, Joy was finally able to write vi, bringing screen-based editing to Berkeley. He soon found himself in a quandary. As is frequently the case in universities strapped for money, old equipment is never replaced all at once. Rather than support code for optimizing the updating of several different terminals, he decided to consolidate the screen management by using a small interpreter to redraw the screen. This interpreter was driven by a description of the terminal characteristics, an effort that eventually became termcap.

By mid-1978, the software distribution clearly needed to be updated. The Pascal system had been made markedly more robust through feedback from its expanding user community, and had been split into two passes so that it could be run on PDP-11/34s. The result of the update was the "Second Berkeley Software Distribution," a name that was quickly shortened to 2BSD. Along with the enhanced Pascal system, vi and termcap for several terminals was included. Once again Bill Joy single-handedly put together distributions, answered the phone, and incorporated user feedback into the system. Over the next year nearly seventy-five tapes were shipped. Though Joy moved on to other projects the following year, the 2BSD distribution continued to expand. The final version of this distribution, 2.11BSD, is a complete system used on hundreds of PDP-11's still running in various corners of the world.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!
VAX Unix: Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Early in 1978, Professor Richard Fateman began looking for a machine with a larger address space on which he could continue his work on Macsyma (originally started on a PDP-10). The newly announced VAX-11/780 fulfilled the requirements and was available within budget. Fateman and thirteen other faculty members put together an NSF proposal that they combined with some departmental funds to purchase a VAX.

Initially the VAX ran DEC's operating system VMS, but the department had gotten used to the Unix environment and wanted to continue using it. So, shortly after the arrival of the VAX, Fateman obtained a copy of the 32/V port of Unix to the VAX by John Reiser and Tom London of Bell Labs.

Although 32/V provided a Version 7 Unix environment on the VAX, it did not take advantage of the virtual memory capability of the VAX hardware. Like its predecessors on the PDP-11, it was entirely a swap-based system. For the Macsyma group at Berkeley, the lack of virtual memory meant that the process address space was limited by the size of the physical memory, initially 1 megabyte on the new VAX.

To alleviate this problem, Fateman approached Professor Domenico Ferrari, a member of the systems faculty at Berkeley, to investigate the possibility of having his group write a virtual memory system for Unix. Ozalp Babaoglu, one of Ferrari's students, set about to find some way of implementing a working set paging system on the VAX; his task was complicated because the VAX lacked reference bits.

As Babaoglu neared the completion of his first cut at an implementation, he approached Bill Joy for some help in understanding the intricacies of the Unix kernel. Intrigued by Babaoglu's approach, Joy joined in helping to integrate the code into 32/V and then with the ensuing debugging.

Unfortunately, Berkeley had only a single VAX for both system development and general production use. Thus, for several weeks over the Christmas break, the tolerant user community alternately found themselves logging into 32/V and "Virtual VAX/Unix." Often their work on the latter system would come to an abrupt halt, followed several minutes later by a 32/V login prompt. By January, 1979, most of the bugs had been worked out, and 32/V had been relegated to history.
Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!
DARPA Support: Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Meanwhile, in the offices of the planners for the Defense Advanced Research Projects Agency (DARPA), discussions were being held that would have a major influence on the work at Berkeley. One of DARPA's early successes had been to set up a nationwide computer network to link together all their major research centers. At that time, they were finding that many of the computers at these centers were reaching the end of their useful lifetime and had to be replaced. The heaviest cost of replacement was the porting of the research software to the new machines. In addition, many sites were unable to share their software because of the diversity of hardware and operating systems.

Choosing a single hardware vendor was impractical because of the widely varying computing needs of the research groups and the undesirability of depending on a single manufacturer. Thus, the planners at DARP