Batch Processing

With all the talk about service-oriented architectures some might find it odd that batch processing makes the list of best practices. Unfortunately, SOA is really a long way off for most of us and the processing of large batches of data, both inbound and outbound on a periodic basis, is still a big part of the equation. Because of this, we should spend some time thinking about what the architecture of the area looks like to ensure we have a system that processes reliably and if the processing fails, notifies us reliably.

The maturity levels associated with batch processing are briefly described in the following table with additional details provided on the succeeding pages.

Maturity Level	Hallmarks
0	Name: Ad-hoc Processing approaches, error handling, restart, and status reporting (if any) are developed on a one-off basis Direct database access is used to perform functions No file format, naming convention, or other standards are in place Processing is “straight through,” that is, input files post directly to the database and output files are created directly from the database
1	Name: Standardizing — Input Standard (or at least preferred) input file formats are defined Standard input processing “phasing” is defined Input file processing (limited) code reuse has started
2	Name: Standardizing — Output Basic infrastructure for statusing and error handling is in place Improved input processing phasing and code reuse Standard output processing phasing is defined Output file processing code reuse has started
3	Name: Architected Use of ETL (concepts, if not tools) Architecture and method for addition of new file formats is defined A scheduling infrastructure is in place An error handling infrastructure is in place Database access is isolated through the use of APIs (or comparable)
4	Name: Interactive Interactive mapping of file formats and other setup is performed Performance monitoring infrastructure is in place
5	Name: Dynamic Drag and drop creation of processing functions is performed (standard file formats are no longer required)

As can be seen by the number of levels and the hallmarks, batch processing is not a trivial area, yet few loyalty system architects spend much time here. Admittedly, my background is all about the creation of “general purpose” tools so levels 4 and 5 may be beyond the needs of an internal deployment, but the value of the facilities provided by being at level 3 should be apparent, and for larger programs, should likely be the appropriate target.

Continued ...