Billets mentionnant : filtering

Using Doctrine Collections to improve CSV filtering

I’m happy to announce the release of League CSV – Doctrine Collection Bridge version 1.0.0. The new package under the bakame namespace is an extension to League\Csv which takes advantage of Doctrine Collections features to better manipulate CSV documents.

CSV Filtering

When working with imported CSV documents it always comes down to filtering and querying data from the document to apply some business logic to them. One of the often suggested method is to import the CSV document data into a RBMS and then use its filtering capabilities to process the data. While this might be tempting it also means wasting times in thinking about tables structure and resource availability while often you only want to consume your data and move on to other tasks.

League Csv Statement

For that reason I always prefer using the League\Csv\Statement class. The class uses closures and callables to filter the CSV document as shown below:

Doctrine Collection Criteria

While using the League\Csv\Statement is straightforward, building the correct closure/callable for a given filtering can become a daunting task. Luckily for us, the people behind the Doctrine project have long since resolve this issue with their powerful expression API by introducing the Doctrine\Common\Collections\Criteria class.

Using the Criteria class you can rewrite my previous example as follow:

Which may sound a bit overcomplicated for a simple query but ease debugging and maintenance as filtering requirements become more complex. The expression API is similar to SQL statements so they are easy to understand and use.

From Doctrine Criteria to Csv Statement

This package enables converting any Doctrine\Common\Collections\Criteria object into a League\Csv\Statement object allowing using Doctrine Expression API on any CSV document.

From a Csv Reader to a Doctrine Collection

But that’s not all the package also comes bundle with an implementation of Doctrine Collection, the Bakame\Csv\Extension\RecordCollection that can turn any League\Csv\Reader or League\Csv\ResultSet into a Doctrine Collection.

Final notes

Depending on your CSV size you may prefer using the Bakame\Csv\Extension\RecordCollection class instead of the League\Csv\ResultSet as the latter uses PHP Iterator and streaming capabilities to reduce memory usage.

Last but not least

The League CSV – Doctrine Collection Bridge is an open source project with a MIT License so contributions are more than welcome and will be fully credited. These contributions can be anything from reporting an issue, requesting or adding missing features or simply improving or correcting some typo on the documentation website.

Stream Filtering with SplFileObject

Attention: Les informations de ce billet sont susceptibles d'être obsolètes car vieux de plus 2 ans.

Warning: The information you are reading may be obsolete, this post was published more than 2 years ago.

Since PHP4 days, the language has been able to interact with streams, a resource object that can be read from or written to in a linear fashion. This feature can help us applying filters function on streams while reading from or written to them. This can be interesting if you want, for instance, to compress or decompress, on the fly, a file before or after accessing it.
To apply those filters, the developer must use PHP stream filtering functions stream_filter_* and/or the php meta wrapper php://filter and have access to a PHP file resource. For instance, any resource returned by fopen can be filtered using PHP.

$path = 'path/to/my/file.txt';
$fp = fopen($path, 'r');
$filter1 = stream_filter_append($fp, 'string.toupper');
$filter2 = stream_filter_prepend($fp, 'string.rot13', STREAM_FILTER_READ);
$res = fgets($fp); //the output text is ROT13 and then uppercased
stream_filter_remove($filter2);
rewind($fp);
$res = fgets($fp); //the output text is only uppercased

The stream filters can be applied when reading, writing or on both actions. They can be removed at any given time. This feature is truly powerful. (suite…)

Handling CSV in PHP with Bakame.csv