One of the ways to only load the data corresponding to a mini-batch is to dynamically create mini-batches by processing images randomly from their location. The number of images to be processed in a mini-batch would be equal to the mini-batch size we specify. Of course there would be some bottleneck in the training process because of the dynamic mini-batch creation during training time but that bottleneck is negligible. Specially packages such as keras have efficient dynamic batch creation mechanism. We would be leveraging flow_from_directory functionality in keras to dynamically create mini-batches during training to reduce the memory requirements of the training process. We will still continue ...
Dynamic mini batch creation during training
Get Intelligent Projects Using Python now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.