book

Programming Pig

by Alan Gates

October 2011

Intermediate to advanced

220 pages

6h 25m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Data AddictionWho Should Read This BookConventions Used in This BookCode Examples in This BookUsing Code ExamplesSafari® Books OnlineHow to Contact UsAcknowledgments
What Is Pig?Pig on HadoopMapReduce’s hello worldPig Latin, a Parallel Dataflow LanguageComparing query and dataflow languagesHow Pig differs from MapReduceWhat Is Pig Useful For?Pig PhilosophyPig’s History
Downloading and Installing PigDownloading the Pig Package from ApacheDownloading Pig from ClouderaDownloading Pig Artifacts from MavenDownloading the SourceRunning PigRunning Pig Locally on Your MachineRunning Pig on Your Hadoop ClusterRunning Pig in the CloudCommand-Line and Configuration OptionsReturn Codes
Entering Pig Latin Scripts in GruntHDFS Commands in GruntControlling Pig from Grunt
TypesScalar TypesComplex TypesMapTupleBagNullsSchemasCasts
Preliminary MattersCase SensitivityCommentsInput and OutputLoadStoreDumpRelational OperationsforeachExpressions in foreachUDFs in foreachNaming fields in foreachFilterGroupOrder byDistinctJoinLimitSampleParallelUser Defined FunctionsRegistering UDFsRegistering Python UDFsdefine and UDFsCalling Static Java Functions
Advanced Relational OperationsAdvanced Features of foreachflattenNested foreachUsing Different Join ImplementationsJoining small to large dataJoining skewed dataJoining sorted datacogroupunioncrossIntegrating Pig with Legacy Code and MapReducestreammapreduceNonlinear Data FlowsControlling ExecutionsetSetting the PartitionerPig Latin PreprocessorParameter SubstitutionMacrosIncluding Other Pig Latin Scripts
Development ToolsSyntax Highlighting and CheckingdescribeexplainillustratePig StatisticsMapReduce Job StatusDebugging TipsTesting Your Scripts with PigUnit

Writing Your Scripts to Perform WellFilter Early and OftenProject Early and OftenSet Up Your Joins ProperlyUse Multiquery When PossibleChoose the Right Data TypeSelect the Right Level of ParallelismWriting Your UDF to PerformTune Pig and Hadoop for Your JobUsing Compression in Intermediate ResultsData Layout OptimizationBad Record Handling
CompileBindBinding Multiple Sets of VariablesRunRunning Multiple BindingsUtility Methods
Writing an Evaluation Function in JavaWhere Your UDF Will RunEvaluation Function BasicsInteracting with Pig valuesInput and Output SchemasError Handling and Progress ReportingConstructors and Passing Data from Frontend to BackendLoading the distributed cacheUDFContextOverloading UDFsMemory Issues in Eval FuncsAlgebraic InterfaceAccumulator InterfacePython UDFsWriting Filter Functions
Load FunctionsFrontend Planning FunctionsDetermining InputFormatDetermining the locationGetting the casting functionsPassing Information from the Frontend to the BackendBackend Data ReadingGetting ready to readReading recordsAdditional Load Function InterfacesLoading metadataUsing partitionsCasting bytearraysPushing down projectionsStore FunctionsStore Function Frontend PlanningDetermining OutputFormatSetting the output locationChecking the schemaStore Functions and UDFContextWriting DataPreparing to writeWriting recordsFailure CleanupStoring Metadata
Pig and HiveCascadingNoSQL DatabasesHBaseCassandraMetadata in Hadoop
Built-in UDFsBuilt-in Load and Store FunctionsBuilt-in Evaluation and Filter FunctionsBuilt-in math UDFsBuilt-in aggregate UDFsBuilt-in chararray and bytearray UDFsBuilt-in complex type UDFsBuilt-in filter functionsMiscellaneous built-in UDFPiggybank
MapReduceMap PhaseCombiner PhaseShuffle PhaseReduce PhaseOutput PhaseDistributed CacheHandling FailureHadoop Distributed File System

Content preview from Programming Pig

About the Author

Alan is an original member of the engineering team that took Pig from a Yahoo! Labs research project to a successful Apache open source project. In this role he oversaw the implementation of the language, including programming interfaces and the overall design. He has presented Pig at numerous conferences and user groups, universities, and companies. Alan is a member of the Apache Software Foundation and a co-founder of Hortonworks. He has a BS in Mathematics from Oregon State University and a MA in Theology from Fuller Theological Seminary.

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Start your free trial

Publisher Resources

ISBN: 9781449317881Errata Page

Programming Pig

by Alan Gates

About the Author

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

You might also like

Programming Pig, 2nd Edition

Pig Design Patterns

Learning Apache Pig

Apache Hadoop™ YARN: Moving beyond MapReduce and Batch Processing with Apache Hadoop™ 2

Publisher Resources

About the Author

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,and much more.

You might also like

Programming Pig, 2nd Edition

Pig Design Patterns

Learning Apache Pig

Apache Hadoop™ YARN: Moving beyond MapReduce and Batch Processing with Apache Hadoop™ 2

Publisher Resources

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.