If there’s one thing that benefits web applications, it’s sending lots of data, faster.
To do this we’ve typically compressed our responses from the server to the client using a myriad of different schemes, the most popular of which has been GZIP.
Unfortunately, GZIP was conceived way back in the 1970’s under Unix, and has not been changed at all since then.
In this modern day and age, a new streaming compression format has been developed specifically for use in the age of the web, and is also now implemented directly in all the evergreen browsers.
That compression system is called “Brotli Compression”, its high performance, streaming and has compression ratios that are far superior to the standard GZIP format.
The biggest use of Brotli is in HTTP2, where the compression systems handling of stream-based data sources, rather than random access-based compression, really shines.
Brotli was designed to be at its fastest and most usable on stream-based data sources such as those coming over an HTTP based connection.
If you want to know more about it, there’s a great deal of back ground information on the following Wikipedia Page: https://en.wikipedia.org/wiki/Brotli
For this blog post however, we’re just going to demonstrate how to use it, with a simple command line program that will compress and decompress “.br” static files, so you can “pre compress” static assets on your web server.
First Things First
To get started we need a simple dot net core command line program, cli or visual studio based it’s up to you.
Create yourself a command line application project and get set up with the normal “Hello World” ready to run template. In my case I’m using Rider, but for you it really doesn’t matter, you can use visual studio, code, or any other editor capable of compiling .net core/new .net projects.
Once you have your project set up, replace all the code in your “Program.cs” file with the following code:
using System;
using System.IO;
using System.IO.Compression;
namespace brotlicompression
{
class Program
{
static void Main()
{
string inputFileName = @"P:\brotli\tiftest.tif";
string outputFileName = inputFileName + ".br";
using (FileStream input = File.OpenRead(inputFileName))
{
Console.WriteLine("Opening Input File");
using (FileStream output = File.Create(outputFileName))
{
Console.WriteLine("Opening Output File");
using (BrotliStream compressor = new BrotliStream(output, CompressionMode.Compress))
{
Console.WriteLine("Copy input to output.");
input.CopyTo(compressor);
}
}
}
Console.WriteLine("Complete");
}
}
}
As you can see, I’m using some test files of my own, I’m not going to provide those files with the project however, so you’ll need to change the “inputFile” variable to be one of your own test files, and I’ll leave adding a command line parameter and parser as an exercise to the reader to complete.
I chose for my testing, a bunch of average sized, normal files that you might find on a web server.
The tif file is the largest followed by the wave file. Totalled up, we get something along the lines of the following
Which comes out at approx. 30mb in total size.
One by one we change the input file name, and run the program to compress each file. The compression may not be mega speedy, but that’s largely because it’s not designed to be used as a real-time file compressor, Brotli is very much intended to be employed in a build step for static files.
Once we’ve compressed the files this is what we end up with.
You can see from Figure 3 that some sizes don’t change much (or even at all in the case of the JPG).
This is not surprising and most compression schemes will have the same, with one small difference.
I’ve seen in cases with deflate and GZ that an already compressed file can actually get larger, Brotli on the other hand can detect this, and if it thinks it can’t make any space gains, then it will simply not even try to do so, and may even just pass the file through unaltered.
In the case of our JPG file because the data is already densely compacted, it’s compressed pretty much the only parts in can, in the header with very little change.
The Tif test however has a pretty fantastic saving to about a 3rd of the size of the original file
In general, though, across the entire group of files, we achieved a compression ratio of just over 50%
Looking back at the code you can see that it’s all “stream” based, so you can stream from an input file into memory, out of a socket, a serial connection, a named pipe or just directly to another file as we did in the sample code above.
To decompress a file, you simply just need to switch the input/output file streams, change the “output” stream for the “input” stream in the Brotli compressor change the mode to “Decompress” and tell the decompressor to copy the stream to the output.
TO create a decompressor program, either create a new project, or alter your existing one and change “Program.cs” so that it has the following code in, making sure you substitute the files names for your own in the same manner as you did for the previous example:
using System;
using System.IO;
using System.IO.Compression;
namespace brotlicompression
{
class Program
{
static void Main()
{
string inputFileName = @"P:\brotli\tiftest.tif.br";
string outputFileName = inputFileName + ".tif";
using (FileStream input = File.OpenRead(inputFileName))
{
Console.WriteLine("Opening Input File");
using (FileStream output = File.Create(outputFileName))
{
Console.WriteLine("Opening Output File");
using (BrotliStream compressor = new BrotliStream(input, CompressionMode.Decompress))
{
Console.WriteLine("Copy input to output.");
compressor.CopyTo(output);
}
}
}
Console.WriteLine("Complete");
}
}
}
I’ve been lazy and just appended a new extension onto the file name, you will probably want to do a better job, and replace the extension, and maybe also check to see if the file already exists.
If you time the compression & decompression steps you should also have noticed is that the decompression is significantly faster than the compression, and gets faster the larger the file is.
Brotli is designed to be faster to decompress so that the effect on a client accessing a compressed resource is much quicker than the time taken for a server to compress things, which means that using the dotnet code above, and being on a server that’s compatible with Brotli, you can probably make some quite significant gains by pre-compressing often used resources such as css, web fonts and large images.
You will need to configure your server to make sure the correct mime type is sent, brotli despite the time it’s been around still does not have an official IETF type, most however have settled on “application/x-br” as of the time I write this.
All the ever-green browsers as of the date of me writing this (May 2020) support Brotli compression, but as with everything web, if you’re wanting to check you should always look at sites such as “can I use” and “The Mozilla developer network”
Shawty's Live Space 