Unpacker Specification

"

Introduction

Unpacker is designed to unpack a character-efficient Compact Object into a developer-friendly Expanded Object.

Status

Unpacker was first published on 19th May 2021, we are inviting feedback from the technical community.

Authors

Unpacker was created by Elliott Brown and Tony Walmsley of NUM Technology Ltd Content on another site..

Examples

The examples provided in this specification are intended to demonstrate functionality in the simplest way possible, they are not provided to demonstrate character-efficiency, character-efficiency is better demonstrated in the worked example Content elsewhere on this site..

Components

The components of Unpacker are:

  • Compact Object – the data in its most compact form. Required.
  • Substitution Object – reduce repeating data across objects by storing it once and referring to in many Compact Objects. Optional.
  • Transformation Object – contains instructions to transform the Compact Object. Optional.
  • Expanded Object – the result of the Unpacker process.

Compact Object

This object is required and may include a Variable Index:

Variable Index

Reduce repeated data in your Compact Object by defining it once and referring to it multiple times in the same object. The Variable Index can only be defined at the root of the Compact Object using the ? key. The Variable Index never appears in the Expanded Object.

Variables are stored as array items and referenced using their array position:

Compact Object
Expanded Object

The Variable Index may include maps and arrays:

Compact Object
Expanded Object

Any item in the Variable Index can be referred to specifically using deep referencing:

Compact Object
Expanded Object

Substitution Object

Reduce repeated data across objects by storing common data in a Substitution Object and referring to it in many Compact Objects. Keys in the substitution object must not be numbers:

Compact Object
Substitution Object
Expanded Object

As with the Variable Index, you can use deep referencing to refer to items in the Substitution Object:

Compact Object
Substitution Object
Expanded Object

Transformation Object

The Transformation Object is the most powerful component of Unpacker with many options available to transform the Compact Object. Each pair in the Transformation Object provides a set of instructions on how to transform the corresponding pair in the Compact Object.

Global Instructions

Keys at the top level of the Transformation Object are global instructions and will transform all matching keys at any level of the Compact Object. This example uses the rewriteKey instruction to rewrite all keys named t in the Compact Object:

Compact Object
Transformation Object
Expanded Object

Nested Instructions

Nested keys in the Transformation Object will transform corresponding keys in the Compact Object and override any global instructions:

Compact Object
Transformation Object
Expanded Object

rewriteKey

As shown in the examples above, rewriteKey instructs Unpacker to change the key of a pair in the Compact Object.

Compact Object
Transformation Object
Expanded Object

rewriteValue

The rewriteValue instruction instructs Unpacker to change the value of a pair in the Compact Object. The keyword %self can be used to refer to the original value.

Augment a value
Compact Object
Transformation Object
Expanded Object
Restructure a value
Compact Object
Transformation Object
Expanded Object

replacePair

The replacePair instruction instructs Unpacker to replace the pair identified by the key with the contents of replacePair:

Compact Object
Transformation Object
Expanded Object

assignKeys

The assignKeys instruction takes an array of values and transforms it into an object by assigning each value with a key:

Compact Object
Transformation Object
Expanded Object

arrayItems

The arrayItems instruction takes an array of arrays and transforms each nested array into an object, enabling key assignment for each of them.

Compact Object
Transformation Object
Expanded Object

Expanded Object

The Expanded Object is simply the result of the Unpacker process. It never includes the Variable Index Content further up this document.

Referencing

References are encapsulated in %, with the closing % not required if followed by a space or terminator ("). In the example below we demonstrate the different ways a value can be referenced:

Compact Object
Expanded Object

Deep Referencing

To reference a value within an array or map, use dot notation. In the example below, x and y reference a value within the Variable Index and z references a value in the Substitution Object:

Compact Object
Substitution Object
Expanded Object

Referencing Scope

Each object has its own referencing scope.

Compact Object

Numeric references in the Compact Object always refer to the Variable Index Content further up this document., non-numeric references always refer to the Substitution Object Content further up this document.. The Compact Object can't reference any part of itself other than the Variable Index.

Substitution Object

References are not allowed in the Substitution Object.

Transformation Object

References in the Transformation Object are for values passed through from the Compact Object:

Compact Object
Transformation Object
Expanded Object

Or for keys assigned in the Transformation Object:

Compact Object
Transformation Object
Expanded Object

The Transformation Object can reference specific parts of the Compact Object and Substitution Object using the namespace /compact and /subs respectively, for example:

Compact Object
Transformation Object
Substitution Object
Expanded Object

Library Author Guide

In the following section, rules have been provided for Unpacker library authors:

Variable Index

If the Compact Object is a map and includes a ? key at the top level with an array as its value, the array must be considered the Variable Index. The values within the index are variables that can be referenced elsewhere in the object using their array position.

The Variable Index must never appear in the unpacked object.

  • Variables are referenced with their array position (starting from 0)
  • References are encapsulated in %, with the closing % not required if followed by a space or value terminator (e.g. " in JSON)
  • The variable index may include arbitrarily nested maps, arrays, and primitives. References such as %0.x and %0.var are allowed

References

A reference may be percent-prefixed (%ref) if the reference is followed by a space or terminator ("). If the reference is not followed by a space or terminator it must be percent encapsulated (%ref%).

To escape a reference we use another percent. If a reference cannot be resolved and is a standalone reference (not part of a larger string) it should be resolved as null. If it is not a standalone reference it should be resolved as blank.

In an example where var is defined in the Substitution Object as "var":1:

{"foo":"%var"} -> {"foo":1}

An escaped example:

{"foo":"%%var"} -> {"foo":"%var"}

An example where var is undefined:

{"foo":"%var"} -> {"foo":null}

An example where the undefined reference is interpolated:

{"foo":"this:%var"} -> {"foo":"this:"}

Scope

Scoping rules are different for each object:

Compact Object
  • number references (%0) always refer to the Variable Index
  • string references (%a) always refer to the Substitution Object
  • The Compact Object can't reference any part of itself, other than the Variable Index
Substitution Object
  • Referencing is not allowed in the Substitution Object
Transformation Object
  • The default namespace for references is the instruction within the Transformation Object. References will resolve to keys assigned in the instruction or passed through from the Compact Object.
  • The namespace can be changed to the compact object or the substitution object using the reference prefix /compact. and /subs. respectively, deep referencing can be used in both cases.

Transformation Object

  • Each pair in the Transformation Object provides a set of instructions showing how to transform the pair in the Compact Object with the corresponding key. Each pair is known as an instruction.
  • When replacePair is null this should remove the pair
  • rewriteKey and replacePair are mutually exclusive, specifying both for the same key should return a fatal error
  • rewriteValue and replacePair are mutually exclusive, specifying both for the same key should return a fatal error
  • If rewriteValue is undefined then the original value should pass-through
  • arrayItems can only be specified for arrays, specifying it for a non-array should return a fatal error.
  • When a reference is undefined it should be resolved to null if it's a standalone reference and "" if the reference is not a standalone reference (when it is interpolated in a string)
  • rewriteKey cannot be "" or null – raise a fatal error
  • replacePair cannot be "" – raise a fatal error
  • arrayItems cannot be "" or null – raise a fatal error
  • assignKeys cannot be [] or "" – raise a fatal error
  • rewriteValue can be a map, array or primitive
  • replacePair must be a pair – raise a fatal error if not
  • assignKeys is irrelevant if the object is a map, it can be ignored