Last updated 12 October 2026
Comparison by configuration
Size ratio by configuration
The size ratio is the size of the other output divided by the size of the Polypress archive. A ratio above 1 means that the Polypress archive is smaller. The pyarrow library could not read 100 of the tables, and ORC was not attempted on tables with more than 1,000 columns, so the columnar formats were compared on fewer tables.
| Configuration | Tables | Polypress smaller | Median ratio | Lowest ratio |
|---|---|---|---|---|
gzip -9 | 1,000 | 1,000 | 2.39 | 1.06 |
bzip2 -9 | 1,000 | 978 | 1.57 | 0.87 |
xz -9e | 1,000 | 990 | 1.38 | 0.82 |
zstd -3 | 1,000 | 1,000 | 2.27 | 1.13 |
zstd -19 | 1,000 | 997 | 1.49 | 0.88 |
zstd --ultra -22 | 1,000 | 997 | 1.48 | 0.88 |
brotli -q 11 | 1,000 | 978 | 1.40 | 0.79 |
lz4 -9 | 1,000 | 1,000 | 2.86 | 1.33 |
zip -9 | 1,000 | 1,000 | 2.40 | 1.08 |
| 7-Zip, LZMA2, level 9 | 1,000 | 992 | 1.45 | 0.84 |
| 7-Zip, PPMd, order 6 | 1,000 | 896 | 1.40 | 0.79 |
| 7-Zip, PPMd, order 12 | 1,000 | 821 | 1.23 | 0.75 |
| 7-Zip, PPMd, order 16 | 1,000 | 809 | 1.20 | 0.75 |
| Parquet, snappy | 900 | 900 | 2.68 | 1.04 |
| Parquet, gzip level 9 | 900 | 899 | 1.99 | 0.80 |
| Parquet, brotli level 11 | 900 | 899 | 1.74 | 0.67 |
| Parquet, zstd level 22 | 900 | 899 | 1.83 | 0.73 |
| ORC, snappy | 877 | 877 | 3.17 | 1.09 |
| ORC, zlib | 877 | 875 | 2.20 | 0.71 |
| ORC, zstd | 877 | 872 | 2.08 | 0.65 |
| Feather, lz4 | 900 | 900 | 5.85 | 1.47 |
| Feather, zstd | 900 | 900 | 3.56 | 1.18 |
Effect of the final-stage compressor
Polypress compresses its modelled streams with LZMA2, the method used by xz. Parquet does not offer LZMA2. To separate the effect of the modelling from the choice of final-stage compressor, the modelled streams were also compressed with zstd at level 22 and with brotli at level 11, and compared with outputs that use the same compressor at the same level.
| Polypress streams | Compared with | Polypress smaller | Median ratio |
|---|---|---|---|
| zstd, level 22 | Parquet, zstd level 22 | 899 of 900 | 1.65 |
| zstd, level 22 | zstd --ultra -22 | 984 of 1,000 | 1.35 |
| brotli, level 11 | Parquet, brotli level 11 | 899 of 900 | 1.72 |
| brotli, level 11 | brotli -q 11 | 988 of 1,000 | 1.38 |
Tables on which another configuration was smaller
Another configuration produced a smaller output on 219 tables: 7-Zip with PPMd on 206, brotli on 6, ORC on 5, and xz and bzip2 on 1 each. PPMd is a context-modelling method designed for text. The largest difference was on a table of SARS-CoV-2 variant proportions, where the output of ORC with zstd was 35% smaller than the Polypress archive.
Fidelity of text in Parquet files
The pyarrow library inferred the column types when it read each CSV file. Converting the
resulting Parquet file back to text reproduced every cell for 172 tables and changed at least one
cell for 702. The remaining 126 could not be checked. A typical change is 1.50
becoming 1.5. For these tables, the Parquet output is smaller partly because it
discards the original formatting, which favours Parquet in the comparison.