design/protocol.md

@@ -35,25 +35,17 @@ fucking with you.
 ## Table Pair Encoding (TAPE)
 The Table Pair Encoding (TAPE) scheme is a method for encoding structured data
 within HOPP messages. It defines standard binary encoding methods for common
-data types, as well as a corruption-resistant table structure that maps numeric
-IDs to values. It is designed to allow applications to be presented with data
-they are not equipped to handle while continuing to function normally. This
-enables backwards compatibile application protocol changes.
+data types, as well as aggregate data types such as tables and arrays. It is
+designed to allow applications to be presented with data they are not equipped
+to handle while continuing to function normally. This enables backwards
+compatibile application protocol changes.
 
-### Table Structure
-A table is divided into two sections: the header, and the values. The header
-begins with the number (U16) of pairs in the table, which is then followed by
-that many tag-offset pairs. A tag-offset pair consists of a numerical (U16) tag,
-followed the position (U16) of the value relative to the start of the values
-section. The values section contains the value data for each pair, where the
-start of each value is determined by its offset, and the end is determined by
-the offset of the next value, or the end of the message if there is no value
-after it.
-
-Both sections must be in the same order, and because of this, each value offset
-must be greater than or equal to the last. If a message has erratic structure
-(such as unordered or out-of-bounds offsets), implementations may opt to discard
-only the erratic pairs, as well as the pairs directly before those.
+The length of a TAPE structure is assumed to be given by the surrounding
+protocol, which is usually METADAPT-A or B. The root of a TAPE structure can be
+any data value, but is usually a table, which can contain several values that
+each have a numeric key. Values can also be nested. Both sides of the connection
+must agree on what data type should be the root value, the data type of each
+known table value, etc.
 
 ### Data Value Types
 The table below lists all data value types supported by TAPE.
@@ -68,9 +60,10 @@ The table below lists all data value types supported by TAPE.
 | U16         |               2 | An unsigned 16-bit integer  | BEU
 | U32         |               4 | An unsigned 32-bit integer  | BEU
 | U64         |               8 | An unsigned 64-bit integer  | BEU
-| Array[^1]   |         SOP[^2] | An array of any above type  | PASTA
-| String      |             N/A | A UTF-8 string              | UTF-8
-| StringArray | n * 2 + SOP[^2] | An array the String type    | VILA
+| Array[^1]   |                 | An array of any above type  | PASTA
+| String      |                 | A UTF-8 string              | UTF-8
+| StringArray |                 | An array the String type    | VILA
+| Table       |                 | A table of any type         | TTLV
 
 [^1]: Array types are written as <E>Array, where <E> is the element type. For
 example, an array of I32 would be written as I32Array. StringArray still follows
@@ -95,6 +88,15 @@ Big-Endian, Unsigned integer. The size is defined as the least amount of whole
 octets which can fit all bits in the integer, regardless if the bits are on or
 off. Therefore, the size cannot change at runtime.
 
+#### GBEU
+Growing Big-Endian, Unsigned integer. The integer is broken up into 8-bit
+chunks, where the first bit of each chunk is a CCB. The chunk with its CCB set
+to zero instead of one is the last chunk in the integer. Chunks are ordered from
+most significant to least significant (big endian). The size is defined as the
+least amount of whole octets which can fit all chunks of the integer. The size
+of this type is not fixed and may change at runtime, so this needs to be
+accounted for during use.
+
 #### PASTA
 Packed Single-Type Array. The size is defined as the size of an individual item
 times the number of items. Items are placed one after the other with no gaps
@@ -110,13 +112,23 @@ for during use.
 
 #### VILA
 Variable Item Length Array. The size is defined as the least amount of whole
-octets which can fit each item plus one U16 per item. The size of this type is
-not fixed and may change at runtime, so this needs to be accounted for during
-use. The amount of items must be greater than zero. Items are each prefixed by
-their size (in octets) encoded as a U16, and they are placed one after the other
-with no gaps in-between them, except as required to align the start of each item
-to the nearest whole octet. Items should be of the same type but do not need to
-be of the same size.
+octets which can fit each item plus one GBEU per item describing that item's
+size. The size of this type is not fixed and may change at runtime, so this
+needs to be accounted for during use. The amount of items must be greater than
+zero. Items are each prefixed by their size (in octets) encoded as a GBEU, and
+they are placed one after the other with no gaps in-between them, except as
+required to align the start of each item to the nearest whole octet. Items
+should be of the same type but do not need to be of the same size.
+
+#### TTLV
+TAPE Tag Length Value. The size is defined as the least amount of whole octets
+which can fit each item plus one U16 and one GBEU per item, where the latter of
+which describes that item's size. The size of this type is not fixed and may
+change at runtime, so this needs to be accounted for during use. Items are each
+prefixed by their numerical tag encoded as a U16, and their size (in octets)
+encoded as a GBEU. Items are placed one after the other with no gaps in-between
+them, except as required to align the start of each item to the nearest whole
+octet. Items need not be of the same type nor the same size.
 
 ## Transports
 A transport is a protocol that HOPP connections can run on top of. HOPP