book

Pro Spring Batch

Name: Pro Spring Batch
Author: Michael T. Minella
ISBN: 9781430234524

by Michael T. Minella

July 2011

Intermediate to advanced

502 pages

13h 51m

English

Apress

Read now

Unlock full access

Copyright
About the Author
About the Technical Reviewer
Acknowledgments
1. Batch and Spring
1.1. A History of Batch Processing1.2. Batch Challenges1.3. Why Do Batch Processing in Java?1.4. Other Uses for Spring Batch1.5. The Spring Batch Framework1.5.1. Defining Jobs with Spring1.5.2. Managing Jobs1.5.3. Local and Remote Parallelization1.5.4. Standardizing I/O1.5.5. The Spring Batch Admin Project1.5.6. And All the Features of Spring1.6. How This Book Works1.7. Summary
2. Spring Batch 101
2.1. The Architecture of Batch2.1.1. Examining Jobs and Steps2.1.2. Job Execution2.1.3. Parallelization2.1.3.1. Multithreaded Steps2.1.3.2. Parallel Steps2.1.3.3. Remote Chunking2.1.3.4. Partitioning2.1.4. Batch Administration2.1.5. Documentation2.2. Project Setup2.2.1. Obtaining Spring Batch2.2.1.1. Using the SpringSource Tool Suite2.2.1.2. Downloading the Zip Distribution2.2.1.3. Checking Out from Git2.2.1.4. Configuring Maven2.3. It's the Law: Hello, World!2.4. Running Your Job2.5. Exploring the JobRepository2.5.1. Job Repository Configuration2.5.2. The Job Repository Tables2.5.2.1. BATCH_JOB_INSTANCE2.5.2.2. BATCH_JOB_PARAMS2.5.2.3. BATCH_JOB_EXECUTION and BATCH_STEP_EXECUTION2.5.2.4. Job and Step Execution Context Tables2.6. Summary
3. Sample Job
3.1. Understanding Agile Development3.1.1. Capturing Requirements with User Stories3.1.2. Capturing Design with Test-Driven Development3.1.3. Using a Source-Control System3.1.4. Working with a True Development Environment3.2. Understanding the Requirements of the Statement Job3.3. Designing a Batch Job3.3.1. Job Description3.3.1.1. Importing Customer Transaction Data3.3.1.2. Retrieving Stock Closing Prices3.3.1.3. Importing Stock Prices into Database3.3.1.4. Calculating Transaction Fee Tiers3.3.1.5. Calculating Transaction Fees3.3.1.6. Generating Customer Monthly Statements3.3.2. Understanding the Data Model3.4. Summary
4. Understanding Jobs and Steps
4.1. Introducing a Job4.1.1. Tracing a Job's Lifecycle4.2. Configuring a Job4.2.1. Basic Job Configuration4.2.2. Job Inheritance4.2.3. Job Parameters4.2.3.1. Validating Job Parameters4.2.3.2. Incrementing Job Parameters4.2.4. Working with Job Listeners4.2.5. ExecutionContext4.2.6. Manipulating the ExecutionContext4.2.6.1. ExecutionContext Persistence4.3. Working with Steps4.3.1. Chunk vs. Item Processing4.3.2. Step Configuration4.3.2.1. Basic Step4.3.3. Understanding the Other Types of Tasklets4.3.3.1. CallableTaskletAdapter4.3.3.2. MethodInvokingTaskletAdapter4.3.3.3. SystemCommandTasklet4.3.3.4. Tasklet Step4.3.3.5. Step Inheritance4.3.3.6. Chunk-Size Configuration4.3.3.7. Step Listeners4.3.4. Step Flow4.3.4.1. Conditional Logic4.3.4.2. Ending a Job4.3.4.3. Externalizing Flows4.3.4.4. Parallelization of Flows4.3.5. Item Error Handling4.3.5.1. Item Retry4.3.5.2. Item Skip4.4. Summary
5. Job Repository and Metadata
5.1. Configuring the Job Repository5.1.1. Using an In-Memory Job Repository5.1.2. Database5.1.2.1. Database Schema Configuration5.1.2.2. Transaction Configuration5.2. Using Job Metadata5.2.1. The JobExplorer5.2.2. The JobOperator5.3. Summary
6. Running a Job
6.1. Starting a Job6.1.1. Job Execution6.1.2. Spring Batch Job Runners6.1.2.1. CommandLineJobRunner6.1.2.2. JobRegistryBackgroundJobRunner6.1.3. Third-Party Integration6.1.3.1. Scheduling with Quartz6.1.3.2. Running in a Container6.1.3.3. Launching with Spring Batch Admin6.2. Stopping a Job6.2.1. The Natural End6.2.2. Programmatic Ending6.2.2.1. Using the <stop> Tag6.2.2.2. Stopping with StepExecution6.2.3. External Stoppage6.2.3.1. Stopping via Spring Batch Admin6.2.3.2. Stopping Using CommandLineJobRunner6.2.4. Error Handling6.2.4.1. Stopping the Job6.3. Controlling Restart6.3.1. Preventing a Job from Being Rerun6.3.2. Configuring the Number of Restarts6.3.3. Rerunning a Complete Step6.4. Summary

7. Readers
7.1. The ItemReader Interface7.2. File Input7.2.1. Flat Files7.2.1.1. Fixed-Width Files7.2.1.2. Delimited Files7.2.1.3. Custom Record Parsing7.2.1.4. Multiple Record Formats7.2.1.5. Multiline Records7.2.1.6. Multiple Sources7.2.2. XML7.3. Database Input7.3.1. JDBC7.3.1.1. JDBC Cursor Processing7.3.1.2. JDBC Paged Processing7.3.2. Hibernate7.3.2.1. Cursor processing with Hibernate7.3.2.2. Paged Database Access with Hibernate7.3.3. JPA7.4. Existing Services7.5. Custom Input7.6. Error Handling7.6.1. Skipping Records7.6.2. Logging Invalid Records7.6.3. Dealing with No Input7.7. Summary
8. Item Processors
8.1. Introduction to ItemProcessors8.2. Using Spring Batch's ItemProcessors8.2.1. ValidatingItemProcessor8.2.1.1. Input Validation8.2.1.2. Subclassing the ValidatingItemProcessor8.2.2. ItemProcessorAdapter8.2.3. CompositeItemProcessor8.3. Writing Your Own ItemProcessor8.3.1. Filtering Items8.4. Summary
9. Item Writers
9.1. Introduction to ItemWriters9.2. File-Based ItemWriters9.2.1. FlatFileItemWriter9.2.1.1. Formatted Text Files9.2.1.2. Delimited Files9.2.1.3. File Creation Options9.2.2. StaxEventItemWriter9.3. Database-Based ItemWriters9.3.1. JdbcBatchItemWriter9.3.2. HibernateItemWriter9.3.3. JpaItemWriter9.4. Alternative Output Destination ItemWriters9.4.1. ItemWriterAdapter9.4.2. PropertyExtractingDelegatingItemWriter9.4.3. JmsItemWriter9.4.4. SimpleMailMessageItemWriter9.5. Multipart ItemWriters9.5.1. MultiResourceItemWriter9.5.1.1. Header and Footer XML Fragments9.5.1.2. Header and Footer Records in a Flat File9.5.2. CompositeItemWriter9.5.3. ClassifierCompositeItemWriter9.5.3.1. The ItemStream Interface9.6. Summary
10. Sample Application
10.1. Reviewing the Statement Job10.2. Setting Up a New Project10.3. Importing Customer and Transaction Data10.3.1. Creating the Customer Transaction Reader10.3.2. Looking Ip Ids10.3.3. Writing the Customer and Transaction Data10.4. Downloading Current Stock Prices10.4.1. Reading the Tickers10.4.2. Writing the Stock File10.5. Importing Current Stock Prices10.5.1. Reading the Stock Price File10.5.2. Writing Stock Prices to the Database10.6. Calculating Pricing Tiers10.6.1. Reading How Many Transactions the Customer Had10.6.2. Calculating the Pricing Tier10.6.3. Updating the Database with the Calculated Tier10.7. Calculating Transaction Fees10.7.1. Reading the Transactions10.7.1.1. Calculating Transaction Prices10.7.2. Saving Transaction Fees to the Database10.8. Generating Monthly Statement10.8.1. Reading the Statement Data10.8.2. Writing the Statements10.9. Summary
11. Scaling and Tuning
11.1. Profiling Your Batch Process11.1.1. A Tour of VisualVM11.1.2. Profiling Spring Batch Applications11.1.2.1. CPU Profiling11.1.2.2. Memory Profiling11.2. Scaling a Job11.2.1. Multithreaded Steps11.2.2. Parallel Steps11.2.2.1. Preloading Data for Processing11.2.2.2. Loading Orders Into the Database11.2.2.3. Configuring the Parallel Steps11.2.2.4. Building the Picklists11.2.3. Remote Chunking11.2.4. Partitioning11.3. Summary
12. Testing Batch Processes
12.1. Unit Tests with JUnit and Mockito12.1.1. JUnit12.1.1.1. JUnit Lifecycle12.1.2. Mock Objects12.1.3. Mockito12.2. Integration Tests with Spring Classes12.2.1. General Integration Testing with Spring12.2.1.1. Configuring the Testing Environment12.2.1.2. Writing an Integration Test12.2.2. Testing Spring Batch12.2.2.1. Testing Step-Scoped Beans12.2.2.2. Testing a Step12.2.2.3. Testing a Job12.3. Summary

Content preview from Pro Spring Batch

Chapter 11. Scaling and Tuning

The IRS processed over 236 million tax returns in 2010. Atlanta's Hartsfield-Jackson airport handled nearly 90 million passengers in 2010. Facebook has more than 45 million status updates a day. Apple sold more than 1.7 million iPhone 4s in its first three days of availability. The amount of data the world generates every day is staggering. It used to be that as the data increased, so did the processors to process it. If your app wasn't fast enough, you could wait a year and buy a new server, and all was fine.

But that isn't the case anymore. CPUs aren't getting faster at nearly the rate they used to. Instead, by adding cores instead of transistors to a single core, CPUs are getting better at parallel processing instead ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781430234524Purchase book

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Pro Spring Batch

by Michael T. Minella

Chapter 11. Scaling and Tuning

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.