Chapter 15. Scalable scripting for large data sets: pipeline and database techniques
Matthew Reynolds
An online retailer needed to learn which of their Domain Name System (DNS) records were getting the most queries after an advertising campaign. They wrote a script to get this information from their DNS server logs, but as the logs grew, the script slowed to a crawl. Worse, when they tried to run the script on multiple servers remotely the script failed with an OutOfMemoryException.
A web search for the terms PowerShell and OutOfMemoryException returns many thousands of hits. People are clearly struggling to manage large data sets.
The typical problem is the use of a fragile pattern that works in the lab but doesn’t scale with real-world ...
Get PowerShell Deep Dives now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
