Worked Example

This step-by-step example demonstrates how different features and components of Unpacker can be used to compact an object.

Example Object

In the following JSON object, there's lots of repeating data. We've highlighted some data orange and some blue:

{ "name": "Widget Company Ltd", "strapline": "Making the best widgets", "website": "https://www.widgetcompany.com", "employees": [{ "name": "Jane Smith", "position": "Chief Executive Officer", "bio": "https://www.widgetcompany.com/team/janesmith", "linkedin": "https://www.linkedin.com/in/janesmith", "twitter": "https://www.twitter.com/janesmithwidgets" }, { "name": "John Wilson", "position": "Chief Technology Officer", "bio": "https://www.widgetcompany.com/team/johnwilson", "linkedin": "https://www.linkedin.com/in/johnwilson", "twitter": "https://www.twitter.com/jono" }, { "name": "Dashna Anand", "position": "Chief Marketing Officer", "bio": "https://www.widgetcompany.com/team/dashnaanand", "linkedin": "https://www.linkedin.com/in/dashnaanand", "twitter": "https://www.twitter.com/dashnaanand" }] }
An example object with lots of repeating data. The object is 764 characters minified.

To reduce the size of our object, we must first consider which data will be repeated across multiple objects (e.g. many organisations in this example), in the example above and throughout this page we've highlighted it orange. In the example object it includes:

Then, consider which repeated data is specific to this object (Widget Company Ltd), in the example above and throughout this page we've highlighted it blue. In the example object it includes:

Packing the object

We can reduce the original object down to its most compact form (packing the object) for storage, then Unpacker can be used to unpack it to its expanded form.

Step 1. Rewriting keys

We can reduce the size of the repeated data by using shorter keys and expand these using a Transformation Object. The Transformation Object is an object specifying each key value pair to transform, you can rewrite the key of the key value pair using rewriteKey, for example the first instruction below is to convert the key n to the key name:

{ "n": { "rewriteKey": "name" }, "s": { "rewriteKey": "strapline" }, "w": { "rewriteKey": "website" }, "e": { "rewriteKey": "employees" }, "p": { "rewriteKey": "position" }, "b": { "rewriteKey": "bio" }, "l": { "rewriteKey": "linkedin" }, "t": { "rewriteKey": "twitter" } }
Version 0.1 of our Transformation Object

We can now pack our object tighter:

{ "n": "Widget Company Ltd", "s": "Making the best widgets", "w": "https://www.widgetcompany.com", "e": [{ "n": "Jane Smith", "p": "Chief Executive Officer", "b": "https://www.widgetcompany.com/team/janesmith", "l": "https://www.linkedin.com/in/janesmith", "t": "https://www.twitter.com/janesmithwidgets" }, { "n": "John Wilson", "p": "Chief Technology Officer", "b": "https://www.widgetcompany.com/team/johnwilson", "l": "https://www.linkedin.com/in/johnwilson", "t": "https://www.twitter.com/jono" }, { "n": "Dashna Anand", "p": "Chief Marketing Officer", "b": "https://www.widgetcompany.com/team/dashnaanand", "l": "https://www.linkedin.com/in/dashnaanand", "t": "https://www.twitter.com/dashnaanand" }] }
The object is now 664 characters minified

Step 2. Rewriting values

We can rewrite the value of a pair too, in the Transformation Object below notice we're rewriting the value of l and t, in this case the value references itself using the syntax %self – in both cases prefixing a URL to the value:

{ "n": { "rewriteKey": "name" }, "s": { "rewriteKey": "strapline" }, "w": { "rewriteKey": "website" }, "e": { "rewriteKey": "employees" }, "p": { "rewriteKey": "position" }, "b": { "rewriteKey": "bio" }, "l": { "rewriteKey": "linkedin", "rewriteValue" : "https://www.linkedin.com/in/%self" }, "t": { "rewriteKey": "twitter", "rewriteValue" : "https://www.twitter.com/%self" } }
Version 0.2 of our Transformation Object

We can now pack our object tighter, the values for linkedin (l) and twitter (t) will automatically be rewritten to include the URL:

{ "n": "Widget Company Ltd", "s": "Making the best widgets", "w": "https://www.widgetcompany.com", "e": [{ "n": "Jane Smith", "p": "Chief Executive Officer", "b": "https://www.widgetcompany.com/team/janesmith", "l": "janesmith", "t": "janesmithwidgets" }, { "n": "John Wilson", "p": "Chief Technology Officer", "b": "https://www.widgetcompany.com/team/johnwilson", "l": "johnwilson", "t": "jono" }, { "n": "Dashna Anand", "p": "Chief Marketing Officer", "b": "https://www.widgetcompany.com/team/dashnaanand", "l": "dashnaanand", "t": "dashnaanand" }] }
The object is now 508 characters minified

We've removed quite a bit of repeating data but there's more work to do.

Step 3. Removing keys

If we modify the structure of our Compact Object, we can use assignKeys in our Transformation Object to remove keys altogether.

Step 3a. Removing keys for the organisation

If organisations always have a similar structure (e.g. a name, strapline, website and list of employees), then we don't need to provide these keys. We can pass just the values in an array and Unpacker can assign keys to the values. When we're assigning keys to array items we need to tell Unpacker how to assign keys to the values using assignKeys, notice we've also been able to remove some of the rewriteKey instructions:

{ "o": { "assignKeys": ["name", "strapline", "website", "employees"] }, "n": { "rewriteKey": "name" }, "p": { "rewriteKey": "position" }, "b": { "rewriteKey": "bio" }, "l": { "rewriteKey": "linkedin", "rewriteValue" : "https://www.linkedin.com/in/%self" }, "t": { "rewriteKey": "twitter", "rewriteValue" : "https://www.twitter.com/%self" } }
Version 0.3 of our Transformation Object

The Compact Object needs to change to benefit from this key assignment, we need to add an o key so Unpacker knows how to assign the keys. Our new Compact Object is shown below, notice we're passing the organisation name, strapline, website and employees as array items now:

{ "o": [ "Widget Company Ltd", "Making the best widgets", "https://www.widgetcompany.com", [{ "n": "Jane Smith", "p": "Chief Executive Officer", "b": "https://www.widgetcompany.com/team/janesmith", "l": "janesmith", "t": "janesmithwidgets" }, { "n": "John Wilson", "p": "Chief Technology Officer", "b": "https://www.widgetcompany.com/team/johnwilson", "l": "johnwilson", "t": "jono" }, { "n": "Dashna Anand", "p": "Chief Marketing Officer", "b": "https://www.widgetcompany.com/team/dashnaanand", "l": "dashnaanand", "t": "dashnaanand" }] ] }
The object is now 498 characters minified

Changing the structure of our Compact Object will also change the structure of our Expanded Object, which now looks like this (notice the o at the top of the object):

{ "o": { "name": "Widget Company Ltd", "strapline": "Making the best widgets", "website": "https://www.widgetcompany.com", "employees": [{ "name": "Jane Smith", "position": "Chief Executive Officer", "bio": "https://www.widgetcompany.com/team/janesmith", "linkedin": "https://www.linkedin.com/in/janesmith", "twitter": "https://www.twitter.com/janesmithwidgets" }, { "name": "John Wilson", "position": "Chief Technology Officer", "bio": "https://www.widgetcompany.com/team/johnwilson", "linkedin": "https://www.linkedin.com/in/johnwilson", "twitter": "https://www.twitter.com/jono" }, { "name": "Dashna Anand", "position": "Chief Marketing Officer", "bio": "https://www.widgetcompany.com/team/dashnaanand", "linkedin": "https://www.linkedin.com/in/dashnaanand", "twitter": "https://www.twitter.com/dashnaanand" }] } }
This new expanded object has a different structure to the original object

This is often not the desired outcome and can be easily overcome by modifying the Transformation Object to use replacePair. With this modification the o pair is replaced with the pairs specified within o itself. This will retain the original object structure:

{ "o": { "assignKeys": ["name", "strapline", "website", "employees"], "replacePair" : "%self" }, "n": { "rewriteKey": "name" }, "p": { "rewriteKey": "position" }, "b": { "rewriteKey": "bio" }, "l": { "rewriteKey": "linkedin", "rewriteValue" : "https://www.linkedin.com/in/%self" }, "t": { "rewriteKey": "twitter", "rewriteValue" : "https://www.twitter.com/%self" } }
Version 0.4 of our Transformation Object

Step 3b. Removing keys for employees

We can remove keys for the employee objects too. As with organisation, we need to tell Unpacker what type of object it is. To do this we add the key em to our Transformation Object as seen below and use replacePair, notice we can now remove all of the rewriteKey rules:

{ "o": { "assignKeys": [ "n", "s", "w", "e" ], "replacePair": { "name": "%n", "strapline": "%s", "website": "%w", "employees": "%e" } }, "em": { "replacePair": { "name": "%n", "position": "%p", "bio": "%b", "linkedin": "https://www.linkedin.com/in/%l", "twitter": "https://www.twitter.com/%t" } } }
Version 0.5 of our Transformation Object

We can update our Compact Object and pack it tighter but notice we've also introduced new repeating data em:

{ "o": [ "Widget Company Ltd", "Making the best widgets", "https://www.widgetcompany.com", [{ "em" : [ "Jane Smith", "Chief Executive Officer", "https://www.widgetcompany.com/team/janesmith", "janesmith", "janesmithwidgets" ] }, { "em" : [ "John Wilson", "Chief Technology Officer", "https://www.widgetcompany.com/team/johnwilson", "johnwilson", "jono" ] }, { "em" : [ "Dashna Anand", "Chief Marketing Officer", "https://www.widgetcompany.com/team/dashnaanand", "dashnaanand", "dashnaanand" ] }] ] }
The object is now 456 characters minified

The em key doesn't appear in our Output object because we've used replacePair,

Step 4. Assign map structures to array items

The repeated em key is taking up valuable space in our Compact Object. As a final step we can remove the key em by telling Unpacker the structure of array items inside employees using arrayItems:

{ "o": { "assignKeys": ["name", "strapline", "website", "employees"], "replacePair": "%self" }, "employees" : { "arrayItems" : "em" }, "em": { "assignKeys": ["n", "p", "b", "l", "t"], "replacePair": { "name": "%n", "position": "%p", "bio": "%b", "linkedin": "https://www.linkedin.com/in/%l", "twitter": "https://www.twitter.com/%t" } } }
Version 0.6 of our Transformation Object

Now we can remove em and pass an array of arrays to employees:

{ "o": [ "Widget Company Ltd", "Making the best widgets", "https://www.widgetcompany.com", [ [ "Jane Smith", "Chief Executive Officer", "https://www.widgetcompany.com/team/janesmith", "janesmith", "janesmithwidgets" ], [ "John Wilson", "Chief Technology Officer", "https://www.widgetcompany.com/team/johnwilson", "johnwilson", "jono" ], [ "Dashna Anand", "Chief Marketing Officer", "https://www.widgetcompany.com/team/dashnaanand", "dashnaanand", "dashnaanand" ] ] ] }
The object is now 438 characters minified

We've now removed all unnecessary repeating structure, previously highlighted orange, from the Compact Object. We still have some repeated data within this object, highlighted blue.

Step 5: Adding a Variable Index

We can remove data that repeats only inside this object using the Variable Index. The Variable Index is defined using the key ?. Values in the index are referenced elsewhere in the object using their position in the array, e.g. %0.

{ "?": ["https://www.widgetcompany.com", "team", "janesmith", "johnwilson", "dashnaanand"], "o": [ "Widget Company Ltd", "Making the best widgets", "%0", [ [ "Jane Smith", "Chief Executive Officer", "%0%/%1%/%2", "%2", "%2%widgets" ], [ "John Wilson", "Chief Technology Officer", "%0%/%1%/%3", "%3", "jono" ], [ "Dashna Anand", "Chief Marketing Officer", "%0%/%1%/%4", "%4", "%4" ] ] ] }
The object is now 351 characters minified

The Variable Index doesn't appear in the Expanded Object.

Step 6. Adding a Substitution Object

For common data that repeats across objects, the Substitution Object can be used. In our example, we could say that the titles "Chief Executive Officer", "Chief Technology Officer" and "Chief Marketing Officer" are likely to repeat across many different organisations.

If we define a simple Substitution Object like this:

{ "ceo": "Chief Executive Officer", "cto": "Chief Technology Officer", "cmo": "Chief Marketing Officer" }
A simple substitution object

We can reference these variables and pack our object further:

{ "?": ["https://www.widgetcompany.com", "team", "janesmith", "johnwilson", "dashnaanand"], "o": [ "Widget Company Ltd", "Making the best widgets", "%0", [ [ "Jane Smith", "%ceo", "%0%/%1%/%2", "%2", "%2%widgets" ], [ "John Wilson", "%cto", "%0%/%1%/%3", "%3", "jono" ], [ "Dashna Anand", "%cmo", "%0%/%1%/%4", "%4", "%4" ] ] ] }
The object is now 294 characters minified

Comparison

Our original object was 764 characters minified, using the techniques above we've been able to reduce the object down to a Compact Object of 294 characters – a reduction of over 60%.