matlab-choose-big-data-solution

Use tabularTextDatastore combined with tall arrays to process large CSV files in chunks automatically. Create the datastore, wrap it with tall, apply operations like groupsummary, then call gather to collect results into memory.

Tall arrays treat all files as one continuous dataset for aggregate analysis, while transform processes each file or row group independently. Use tall for whole-dataset statistics and transform when file boundaries matter, such as per-file statistics.

Replace readtable with tabularTextDatastore and wrap it in tall. Note that TextType must be set at creation time, TrimNonNumeric is unsupported, and type overrides use TextscanFormats specifiers like %q or %f instead of setvartype.

Yes, readall on a transformed parquetDatastore supports UseParallel=true when Parallel Computing Toolbox is installed. Tall arrays built on parquetDatastore also distribute computation automatically across an open parallel pool.

Build a custom datastore by subclassing matlab.io.Datastore when your format lacks a built-in datastore, such as large XML, JSON, or proprietary binary files. First check whether fileDatastore with a custom ReadFcn is sufficient before writing a full class.

readtable loads the entire file into memory, so large JSON, XML, HTML, or Word files cause OOM errors. Built-in datastores do not support these formats, so the solution is a custom datastore that reads the file in chunks.