book

Pro Spring Batch

Name: Pro Spring Batch
Author: Michael T. Minella
ISBN: 9781430234524

by Michael T. Minella

July 2011

Intermediate to advanced

502 pages

13h 51m

English

Apress

Read now

Unlock full access

Copyright
About the Author
About the Technical Reviewer
Acknowledgments
1. Batch and Spring
1.1. A History of Batch Processing1.2. Batch Challenges1.3. Why Do Batch Processing in Java?1.4. Other Uses for Spring Batch1.5. The Spring Batch Framework1.5.1. Defining Jobs with Spring1.5.2. Managing Jobs1.5.3. Local and Remote Parallelization1.5.4. Standardizing I/O1.5.5. The Spring Batch Admin Project1.5.6. And All the Features of Spring1.6. How This Book Works1.7. Summary
2. Spring Batch 101
2.1. The Architecture of Batch2.1.1. Examining Jobs and Steps2.1.2. Job Execution2.1.3. Parallelization2.1.3.1. Multithreaded Steps2.1.3.2. Parallel Steps2.1.3.3. Remote Chunking2.1.3.4. Partitioning2.1.4. Batch Administration2.1.5. Documentation2.2. Project Setup2.2.1. Obtaining Spring Batch2.2.1.1. Using the SpringSource Tool Suite2.2.1.2. Downloading the Zip Distribution2.2.1.3. Checking Out from Git2.2.1.4. Configuring Maven2.3. It's the Law: Hello, World!2.4. Running Your Job2.5. Exploring the JobRepository2.5.1. Job Repository Configuration2.5.2. The Job Repository Tables2.5.2.1. BATCH_JOB_INSTANCE2.5.2.2. BATCH_JOB_PARAMS2.5.2.3. BATCH_JOB_EXECUTION and BATCH_STEP_EXECUTION2.5.2.4. Job and Step Execution Context Tables2.6. Summary
3. Sample Job
3.1. Understanding Agile Development3.1.1. Capturing Requirements with User Stories3.1.2. Capturing Design with Test-Driven Development3.1.3. Using a Source-Control System3.1.4. Working with a True Development Environment3.2. Understanding the Requirements of the Statement Job3.3. Designing a Batch Job3.3.1. Job Description3.3.1.1. Importing Customer Transaction Data3.3.1.2. Retrieving Stock Closing Prices3.3.1.3. Importing Stock Prices into Database3.3.1.4. Calculating Transaction Fee Tiers3.3.1.5. Calculating Transaction Fees3.3.1.6. Generating Customer Monthly Statements3.3.2. Understanding the Data Model3.4. Summary
4. Understanding Jobs and Steps
4.1. Introducing a Job4.1.1. Tracing a Job's Lifecycle4.2. Configuring a Job4.2.1. Basic Job Configuration4.2.2. Job Inheritance4.2.3. Job Parameters4.2.3.1. Validating Job Parameters4.2.3.2. Incrementing Job Parameters4.2.4. Working with Job Listeners4.2.5. ExecutionContext4.2.6. Manipulating the ExecutionContext4.2.6.1. ExecutionContext Persistence4.3. Working with Steps4.3.1. Chunk vs. Item Processing4.3.2. Step Configuration4.3.2.1. Basic Step4.3.3. Understanding the Other Types of Tasklets4.3.3.1. CallableTaskletAdapter4.3.3.2. MethodInvokingTaskletAdapter4.3.3.3. SystemCommandTasklet4.3.3.4. Tasklet Step4.3.3.5. Step Inheritance4.3.3.6. Chunk-Size Configuration4.3.3.7. Step Listeners4.3.4. Step Flow4.3.4.1. Conditional Logic4.3.4.2. Ending a Job4.3.4.3. Externalizing Flows4.3.4.4. Parallelization of Flows4.3.5. Item Error Handling4.3.5.1. Item Retry4.3.5.2. Item Skip4.4. Summary
5. Job Repository and Metadata
5.1. Configuring the Job Repository5.1.1. Using an In-Memory Job Repository5.1.2. Database5.1.2.1. Database Schema Configuration5.1.2.2. Transaction Configuration5.2. Using Job Metadata5.2.1. The JobExplorer5.2.2. The JobOperator5.3. Summary
6. Running a Job
6.1. Starting a Job6.1.1. Job Execution6.1.2. Spring Batch Job Runners6.1.2.1. CommandLineJobRunner6.1.2.2. JobRegistryBackgroundJobRunner6.1.3. Third-Party Integration6.1.3.1. Scheduling with Quartz6.1.3.2. Running in a Container6.1.3.3. Launching with Spring Batch Admin6.2. Stopping a Job6.2.1. The Natural End6.2.2. Programmatic Ending6.2.2.1. Using the <stop> Tag6.2.2.2. Stopping with StepExecution6.2.3. External Stoppage6.2.3.1. Stopping via Spring Batch Admin6.2.3.2. Stopping Using CommandLineJobRunner6.2.4. Error Handling6.2.4.1. Stopping the Job6.3. Controlling Restart6.3.1. Preventing a Job from Being Rerun6.3.2. Configuring the Number of Restarts6.3.3. Rerunning a Complete Step6.4. Summary

7. Readers
7.1. The ItemReader Interface7.2. File Input7.2.1. Flat Files7.2.1.1. Fixed-Width Files7.2.1.2. Delimited Files7.2.1.3. Custom Record Parsing7.2.1.4. Multiple Record Formats7.2.1.5. Multiline Records7.2.1.6. Multiple Sources7.2.2. XML7.3. Database Input7.3.1. JDBC7.3.1.1. JDBC Cursor Processing7.3.1.2. JDBC Paged Processing7.3.2. Hibernate7.3.2.1. Cursor processing with Hibernate7.3.2.2. Paged Database Access with Hibernate7.3.3. JPA7.4. Existing Services7.5. Custom Input7.6. Error Handling7.6.1. Skipping Records7.6.2. Logging Invalid Records7.6.3. Dealing with No Input7.7. Summary
8. Item Processors
8.1. Introduction to ItemProcessors8.2. Using Spring Batch's ItemProcessors8.2.1. ValidatingItemProcessor8.2.1.1. Input Validation8.2.1.2. Subclassing the ValidatingItemProcessor8.2.2. ItemProcessorAdapter8.2.3. CompositeItemProcessor8.3. Writing Your Own ItemProcessor8.3.1. Filtering Items8.4. Summary
9. Item Writers
9.1. Introduction to ItemWriters9.2. File-Based ItemWriters9.2.1. FlatFileItemWriter9.2.1.1. Formatted Text Files9.2.1.2. Delimited Files9.2.1.3. File Creation Options9.2.2. StaxEventItemWriter9.3. Database-Based ItemWriters9.3.1. JdbcBatchItemWriter9.3.2. HibernateItemWriter9.3.3. JpaItemWriter9.4. Alternative Output Destination ItemWriters9.4.1. ItemWriterAdapter9.4.2. PropertyExtractingDelegatingItemWriter9.4.3. JmsItemWriter9.4.4. SimpleMailMessageItemWriter9.5. Multipart ItemWriters9.5.1. MultiResourceItemWriter9.5.1.1. Header and Footer XML Fragments9.5.1.2. Header and Footer Records in a Flat File9.5.2. CompositeItemWriter9.5.3. ClassifierCompositeItemWriter9.5.3.1. The ItemStream Interface9.6. Summary
10. Sample Application
10.1. Reviewing the Statement Job10.2. Setting Up a New Project10.3. Importing Customer and Transaction Data10.3.1. Creating the Customer Transaction Reader10.3.2. Looking Ip Ids10.3.3. Writing the Customer and Transaction Data10.4. Downloading Current Stock Prices10.4.1. Reading the Tickers10.4.2. Writing the Stock File10.5. Importing Current Stock Prices10.5.1. Reading the Stock Price File10.5.2. Writing Stock Prices to the Database10.6. Calculating Pricing Tiers10.6.1. Reading How Many Transactions the Customer Had10.6.2. Calculating the Pricing Tier10.6.3. Updating the Database with the Calculated Tier10.7. Calculating Transaction Fees10.7.1. Reading the Transactions10.7.1.1. Calculating Transaction Prices10.7.2. Saving Transaction Fees to the Database10.8. Generating Monthly Statement10.8.1. Reading the Statement Data10.8.2. Writing the Statements10.9. Summary
11. Scaling and Tuning
11.1. Profiling Your Batch Process11.1.1. A Tour of VisualVM11.1.2. Profiling Spring Batch Applications11.1.2.1. CPU Profiling11.1.2.2. Memory Profiling11.2. Scaling a Job11.2.1. Multithreaded Steps11.2.2. Parallel Steps11.2.2.1. Preloading Data for Processing11.2.2.2. Loading Orders Into the Database11.2.2.3. Configuring the Parallel Steps11.2.2.4. Building the Picklists11.2.3. Remote Chunking11.2.4. Partitioning11.3. Summary
12. Testing Batch Processes
12.1. Unit Tests with JUnit and Mockito12.1.1. JUnit12.1.1.1. JUnit Lifecycle12.1.2. Mock Objects12.1.3. Mockito12.2. Integration Tests with Spring Classes12.2.1. General Integration Testing with Spring12.2.1.1. Configuring the Testing Environment12.2.1.2. Writing an Integration Test12.2.2. Testing Spring Batch12.2.2.1. Testing Step-Scoped Beans12.2.2.2. Testing a Step12.2.2.3. Testing a Job12.3. Summary

Overview

Since its release, Spring Framework has transformed virtually every aspect of Java development including web applications, security, aspect-oriented programming, persistence, and messaging. Spring Batch, one of its newer additions, now brings the same familiar Spring idioms to batch processing. Spring Batch addresses the needs of any batch process, from the complex calculations performed in the biggest financial institutions to simple data migrations that occur with many software development projects.

Pro Spring Batch is intended to answer three questions:

What? What is batch processing? What does it entail? What makes it different from the other applications we are developing? What are the challenges inherent in the development of a batch process?
Why? Why do batch processing? Why can't we just process things as we get them? Why do we do batch processing differently than the web applications that we currently work on?
How? How to implement a robust, scalable, distributed batch processing system using open-source frameworks

Pro Spring Batch gives concrete examples of how each piece of functionality is used and why it would be used in a real-world application. This includes providing tips that the "school of hard knocks" has taught author Michael Minella during his experience with Spring Batch. Pro Spring Batch includes examples of I/O options that are not mentioned in the official user's guide, as well as performance tips on things like how to limit the impact of maintaining the state of your jobs.

The author also walks you through, from end to end, the design and implementation of a batch process based upon a theoretical real-world example. This includes basic project setup, implementation, testing, tuning and scaling for large volumes.

What you'll learn

Batch concepts and how they relate to the Spring Batch framework
How to use declarative I/O using the Spring Batch readers/writers
Data integrity techniques used by Spring Batch, including transactions and job state/restartability
How to scale batch jobs via distributed batch processing
How to handle testing batch processes (Unit and functional)

Who this book is for

Java developers with Spring experience.
Java Architects designing batch solutions

More specifically, this book is intended for those who have a solid foundation in the core Java platform. Batch processing covers a wide spectrum of topics, not all of which are covered in detail in this book. Concepts in Java which the reader should be comfortable with include file I/O, JDBC, and transactions. Given that Spring Batch is a framework built upon the open-source IoC container Spring, which will not be covered in this book, it is expected that the reader will be familiar with its concepts and conventions. With that in mind, the reader is not expected to have any prior exposure to the Spring Batch framework. All concepts related to it will be explained in detail, with working examples.

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781430234524Purchase book

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills