generate: Generator compiles

This is a stupid fucking restriction

.gitignore

@@ -0,0 +1 @@
+/generate/test

design/branched-generated-encoder.md

@@ -0,0 +1,128 @@
+# Branched Generated Decoder
+
+Pasted here because Tebitea is down
+
+## The problem
+
+TAPE is designed so that the decoder can gloss over data it does not understand.
+Technically the protocol allows for this, but I completely forgot to implement
+this in the generated decoder, oops. This would be trivial if TAPE messages were
+still flat tables, but they aren't, because those aren't useful enough. So,
+let's analyze the problem.
+
+## When it happens
+
+There are two reasons something might not match up with the expected data:
+
+The first and most obvious is unrecognized keys. If the key is not in the set of
+recognized keys for a KTV, it should leave the corresponding struct field blank.
+Once #6 has been implemented, throw an error if the data was not optional.
+
+The second is wrong types. If we are expecting KTV and get SBA, we should leave
+the data as empty. The aforementioned concern about #6 also applies here. We
+don't need to worry about special cases at the structure root, because it would
+be technically possible to make the structure root an option, so it really is
+just a normal value. Until #6, we will leave that blank too.
+
+## Preliminary ideas
+
+The first is going to be pretty simple. All we need to do is have a skimmer
+function that skims over TAPE data very, and then call that on the KTV value
+each time we run into a mystery key. It should only return an error if the
+structure of the data is malformed in such a way that it cannot continue to the
+next one. This should be stored in the tape package alongside the dynamic
+decoding functions, because they will essentially function the same way and
+could probably share lots of code.
+
+The second is a bit more complicated because of the existence of KTV and OTA
+because they are aggregate types. Go types work a bit differently, as if you
+have an array of an array of an array of ints, that information is represented
+in one place, whereas TAPE doesn't really do that. All of that information is
+sort of buried within the data structure, so we don't know what we will be
+decoding before we actually do it. Whenever we encounter a type we don't expect,
+we would need to abort decoding of the entire data structure, and then skim over
+whatever detritus is left, which would literally be in a half-decoded state. The
+fact that the code is generated flat and thus cannot use return or defer
+statements contributes to the complexity of this problem. We need to go up, but
+we can't. There is no up, only forward.
+
+Of course, the dynamic decoder does not have this problem in the first place
+because it doesn't expect anything, and constructs the destination to fit
+whatever it sees in the TAPE structure as it is decoding it. KTVs are completely
+dynamic because they are implemented as maps, so the only time it needs to
+completely comprehend a type is with OTAs. There is a function called typeOf
+that gets the type of the current tag and returns it as a reflect.Type, which
+necessitates recursion and peeking at OTAs and their elements.
+
+We could try to do the same thing in the generated decoder, comparing the
+determined type against the expected type to try to figure out whether we should
+decode an array or a table, etc. This is immediately problematic as it requires
+memory to be allocated, both for the peek buffer and the resulting tree of type
+information. If we end up with some crazy way to keep track of the types, that's
+only one half of the allocation problem and we would still be spending extra
+cycles going over all of that twice.
+
+## Performance constraints
+
+The generated decoder is supposed to blaze through data, and it can't do that if
+it does all the singing and dancing that the dynamic decoder does. It's time for
+some performance constraints:
+
+- No allocations, except as required to build the destination for the data
+- No redundant work
+- So, no freaking peeking
+- It should take well under 500 lines of generated code to decode one message of
+reasonable size (i.e. be careful not to bloat the binary)
+
+I'm not really going to do my usual thing here of making a slow version and
+speeding it up over time based on evidence and experimentation because these
+constraints inform the design so much it would be impossible to continue without
+them. I am 99% confident that these constraints will allow for an acceptable
+baseline of performance (for generated code) and we can still profile and
+micro-optimize later. This is good enough for me.
+Heavy solution
+
+There is a solution that might work very well which involves completely redoing
+the generated decoding code. We could create a function for every source type to
+destination type mapping that exists in protocol, and then compose them all
+together. The decoding methods for each message or type would be wrappers around
+the correct function for their root TAPE -> Go type mapping. The main benefit of
+this is it would make this problem a lot more manageable because the interface
+points between the data would be represented by function boundaries. This would
+allow the use of return and defer statements, and would allow more code sharing,
+producing a smaller binary. Go would probably inline these where needed.
+
+Would this work? Probably. More investigation is required to make sure. I want
+to stop re-writing things I don't need to. On the other hand, it is just the
+decoder.
+
+## Light solution
+
+TODO: find a solution that satisfies the performance constraints, keeps the same
+identical interface, and works off the same code. I am convinced this is doable,
+and it might even allow us to extract more data from an unexpected structure.
+However, continuing this way might introduce unmanageable complexity. It is
+already a little unmanageable and I am just one pony (kind of).
+
+## Implementation
+
+Heavy solution is going to work here, applied to only the points of
+`Generator.generateDecodeValue` where it decodes an aggregate data structure.
+That way, only minimal amounts of code need to be redone.
+
+Whenever a branch needs to happen, a call shall be generated, a deferred
+implementation request shall be added to a special FIFO queue within the
+generator. After generating data structures and their root decoding functions,
+the generator shall pick away at this queue until no requests remain. The
+generator shall accept new items during this process, so that recursion is
+possible. This is all to ensure it is only ever writing one function at a time
+
+The functions shall take a pointer to a type that accepts any type like (~) the
+destination's base type. We should also probably just call
+`Generator.generateDecodeValue` directly on user defined types this way, keeping
+their public `Decode` methods just for convenience.
+
+The tape package shall contain a skimming function that takes a decoder and a
+tag, and recursively consumes the decoder given the context of the tag. This
+shall be utilized by the decoder functions to skip over values if their tags
+or keys do not match up with what is expected.

generate/generate.go

@@ -6,22 +6,20 @@ import "maps"
 import "math"
 import "slices"
 import "strings"
+import "encoding/hex"
 import "git.tebibyte.media/sashakoshka/hopp/tape"
 
 const imports =
 `
-import "git.teibibyte.media/sashakoshka/hopp/tape"
+import "git.tebibyte.media/sashakoshka/hopp/tape"
 `
 
 const preamble = `
-/* # Do not edit this package by hand!
- * 
- * This file was automatically generated by the Holanet PDL compiler. The
- * source file is located at <path>
- * Please edit that file instead, and re-compile it to this location.
- *
- * HOPP, TAPE, METADAPT, PDL/0 (c) 2025 holanet.xyz
- */
+// Code generated by the Holanet PDL compiler. DO NOT EDIT.
+// The source file is located at <path>
+// Please edit that file instead, and re-compile it to this location.
+// HOPP, TAPE, METADAPT, PDL/0 (c) 2025 holanet.xyz
+
 `
 
 const static = `
@@ -36,6 +34,19 @@ type Message interface {
 	// Method returns the method code of the message.
 	Method() uint16
 }
+
+// canAssign determines if data from the given source tag can be assigned to
+// a Go type represented by destination. It is designed to receive destination
+// values from [generate.Generator.generateCanAssign]. The eventual Go type and
+// the destination tag must come from the same (or hash-equivalent) PDL type.
+func canAssign(destination, source tape.Tag) bool {
+	if destination.Is(source) { return true }
+	if (destination == tape.SBA || destination == tape.LBA) &&
+		(source == tape.SBA || source == tape.LBA) {
+		return true
+	}
+	return false
+}
 `
 
 // Generator converts protocols into Go code.
@@ -47,7 +58,16 @@ type Generator struct {
 	PackageName string
 	
 	nestingLevel int
+	temporaryVar int
 	protocol     *Protocol
+
+	decodeBranchRequestQueue []decodeBranchRequest
+}
+
+type decodeBranchRequest struct {
+	hash [16]byte
+	typ  Type
+	name string
 }
 
 func (this *Generator) Generate(protocol *Protocol) (n int, err error) {
@@ -81,6 +101,14 @@ func (this *Generator) Generate(protocol *Protocol) (n int, err error) {
 		n += nn; if err != nil { return n, err }
 	}
 
+	// request queue
+	for {
+		hash, typ, name, ok := this.pullDecodeBranchRequest()
+		if !ok { break }
+		nn, err := this.generateDecodeBranch(hash, typ, name)
+		n += nn; if err != nil { return n, err }
+	}
+
 	return n, nil
 }
 
@@ -97,7 +125,7 @@ func (this *Generator) generateTypedef(name string, typ Type) (n int, err error)
 	nn, err = this.println()
 	n += nn; if err != nil { return n, err }
 
-	// Tag method
+	// 'Tag' method
 	// to be honest we probably don't need this method at all
 	// nn, err = this.iprintf("\n// Tag returns the preferred TAPE tag.\n")
 	// n += nn; if err != nil { return n, err }
@@ -146,7 +174,27 @@ func (this *Generator) generateTypedef(name string, typ Type) (n int, err error)
 		name)
 	n += nn; if err != nil { return n, err }
 	this.push()
-		nn, err = this.generateDecodeValue(typ, "this", "tag")
+		nn, err = this.iprintf("var nn int\n")
+		n += nn; if err != nil { return n, err }
+
+		nn, err = this.iprintf("if !(")
+		n += nn; if err != nil { return n, err }
+		nn, err = this.generateCanAssign(typ, "tag")
+		n += nn; if err != nil { return n, err }
+		nn, err = this.printf(") {\n")
+		n += nn; if err != nil { return n, err }
+		this.push()
+			nn, err = this.iprintf("nn, err = tape.Skim(decoder, tag)\n")
+			n += nn; if err != nil { return n, err }
+			nn, err = this.generateErrorCheck()
+			n += nn; if err != nil { return n, err }
+			nn, err = this.iprintf("return n, nil\n")
+			n += nn; if err != nil { return n, err }
+		this.pop()
+		nn, err = this.iprintf("}\n")
+		n += nn; if err != nil { return n, err }
+		
+		nn, err = this.generateDecodeValue(typ, name, "this", "tag")
 		n += nn; if err != nil { return n, err }
 		nn, err = this.iprintf("return n, nil\n")
 		n += nn; if err != nil { return n, err }
@@ -170,11 +218,20 @@ func (this *Generator) generateMessage(method uint16, message Message) (n int, e
 	nn, err = this.println()
 	n += nn; if err != nil { return n, err }
 
+	// Method method
+	nn, err = this.iprintf("\n// Method returns the message's method number.\n")
+	n += nn; if err != nil { return n, err }
+	nn, err = this.iprintf(
+		"func(this *%s) Method() uint16 { return 0x%04X }\n",
+		this.resolveMessageName(message.Name),
+		method)
+	n += nn; if err != nil { return n, err }
+
 	// Encode method
 	nn, err = this.iprintf("\n// Encode encodes this message's tag and value.\n")
 	n += nn; if err != nil { return n, err }
 	nn, err = this.iprintf(
-		"func(this %s) Encode(encoder *tape.Encoder) (n int, err error) {\n",
+		"func(this *%s) Encode(encoder *tape.Encoder) (n int, err error) {\n",
 		this.resolveMessageName(message.Name))
 	n += nn; if err != nil { return n, err }
 	this.push()
@@ -184,7 +241,7 @@ func (this *Generator) generateMessage(method uint16, message Message) (n int, e
 		n += nn; if err != nil { return n, err }
 		nn, err = this.println()
 		n += nn; if err != nil { return n, err }
-		nn, err = this.iprintf("nn, err := encoder.WriteUint8()\n")
+		nn, err = this.iprintf("nn, err := encoder.WriteTag(tag)\n")
 		n += nn; if err != nil { return n, err }
 		nn, err = this.generateErrorCheck()
 		n += nn; if err != nil { return n, err }
@@ -196,7 +253,43 @@ func (this *Generator) generateMessage(method uint16, message Message) (n int, e
 	nn, err = this.iprintf("}\n")
 	n += nn; if err != nil { return n, err }
 
-	// TODO decode method
+	// Decode method
+	nn, err = this.iprintf("\n// Decode decodes this message's tag and value.\n")
+	n += nn; if err != nil { return n, err }
+	nn, err = this.iprintf(
+		"func(this *%s) Decode(decoder *tape.Decoder) (n int, err error) {\n",
+		this.resolveMessageName(message.Name))
+	n += nn; if err != nil { return n, err }
+	this.push()
+		nn, err = this.iprintf("tag, nn, err := decoder.ReadTag()\n")
+		n += nn; if err != nil { return n, err }
+		nn, err = this.generateErrorCheck()
+		n += nn; if err != nil { return n, err }
+
+		nn, err = this.iprintf("if !(")
+		n += nn; if err != nil { return n, err }
+		nn, err = this.generateCanAssign(message.Type, "tag")
+		n += nn; if err != nil { return n, err }
+		nn, err = this.printf(") {\n")
+		n += nn; if err != nil { return n, err }
+		this.push()
+			nn, err = this.iprintf("nn, err = tape.Skim(decoder, tag)\n")
+			n += nn; if err != nil { return n, err }
+			nn, err = this.generateErrorCheck()
+			n += nn; if err != nil { return n, err }
+			nn, err = this.iprintf("return n, nil\n")
+			n += nn; if err != nil { return n, err }
+		this.pop()
+		nn, err = this.iprintf("}\n")
+		n += nn; if err != nil { return n, err }
+		
+		nn, err = this.generateDecodeValue(message.Type, this.resolveMessageName(message.Name), "this", "tag")
+		n += nn; if err != nil { return n, err }
+		nn, err = this.iprintf("return n, nil\n")
+		n += nn; if err != nil { return n, err }
+	this.pop()
+	nn, err = this.iprintf("}\n")
+	n += nn; if err != nil { return n, err }
 	
 	return n, nil
 }
@@ -220,13 +313,17 @@ func (this *Generator) generateEncodeValue(typ Type, valueSource, tagSource stri
 			// SI stores the value in the tag, so we write nothing here
 			break
 		}
-		nn, err := this.iprintf("nn, err = encoder.WriteInt%d(%s)\n", bitsToBytes(typ.Bits), valueSource)
+		prefix := "WriteUint"
+		if typ.Signed {
+			prefix = "WriteInt"
+		}
+		nn, err := this.iprintf("nn, err = encoder.%s%d(%s)\n", prefix, typ.Bits, valueSource)
 		n += nn; if err != nil { return n, err }
 		nn, err = this.generateErrorCheck()
 		n += nn; if err != nil { return n, err }
 	case TypeFloat:
 		// FP: <value: FloatN>
-		nn, err := this.iprintf("nn, err = encoder.WriteFloat%d(%s)\n", bitsToBytes(typ.Bits), valueSource)
+		nn, err := this.iprintf("nn, err = encoder.WriteFloat%d(%s)\n", typ.Bits, valueSource)
 		n += nn; if err != nil { return n, err }
 		nn, err = this.generateErrorCheck()
 		n += nn; if err != nil { return n, err }
@@ -241,13 +338,13 @@ func (this *Generator) generateEncodeValue(typ Type, valueSource, tagSource stri
 		n += nn; if err != nil { return n, err }
 		this.push()
 			nn, err = this.iprintf(
-				"nn, err = encoder.WriteUintN(%s.CN(), uint64(len(%s)))\n",
-				tagSource, valueSource)
+				"nn, err = encoder.WriteUintN(uint64(len(%s)), %s.CN())\n",
+				valueSource, tagSource)
 			n += nn; if err != nil { return n, err }
 			nn, err = this.generateErrorCheck()
 			n += nn; if err != nil { return n, err }
 		this.pop()
-		nn, err = this.iprintf("}\n", tagSource)
+		nn, err = this.iprintf("}\n")
 		n += nn; if err != nil { return n, err }
 		
 		nn, err = this.iprintf("nn, err = encoder.Write([]byte(%s))\n", valueSource)
@@ -257,8 +354,8 @@ func (this *Generator) generateEncodeValue(typ Type, valueSource, tagSource stri
 	case TypeArray:
 		// OTA: <length: UN> <elementTag: tape.Tag> <values>*
 		nn, err := this.iprintf(
-			"nn, err = encoder.WriteUintN(%s.CN(), uint64(len(%s)))\n",
-			tagSource, valueSource)
+			"nn, err = encoder.WriteUintN(uint64(len(%s)), %s.CN())\n",
+			valueSource, tagSource)
 		n += nn; if err != nil { return n, err }
 		nn, err = this.generateErrorCheck()
 		n += nn; if err != nil { return n, err }
@@ -271,24 +368,33 @@ func (this *Generator) generateEncodeValue(typ Type, valueSource, tagSource stri
 			n += nn; if err != nil { return n, err }
 			nn, err = this.println()
 			n += nn; if err != nil { return n, err }
+			// TODO: we don't have to do this for loop for some
+			// types such as integers because the CN will be the
+			// same
 			nn, err = this.iprintf("for _, item := range %s {\n", valueSource)
 			n += nn; if err != nil { return n, err }
 			this.push()
+				nn, err = this.iprintf("_ = item\n")
+				n += nn; if err != nil { return n, err }
 				nn, err = this.iprintf("tag := ")
 				n += nn; if err != nil { return n, err }
 				nn, err = this.generateTag(typ.Element, "item")
 				n += nn; if err != nil { return n, err }
 				nn, err = this.println()
 				n += nn; if err != nil { return n, err }
-				nn, err = this.iprintf("tag.Is(tape.SBA) { continue }\n")
+				nn, err = this.iprintf("if tag.Is(tape.SBA) { continue }\n")
 				n += nn; if err != nil { return n, err }
-				nn, err = this.iprintf("tag.CN() > itemTag.CN() { largest = tag }\n")
+				nn, err = this.iprintf("if tag.CN() > itemTag.CN() { itemTag = tag }\n")
 				n += nn; if err != nil { return n, err }
 			this.pop()
 			nn, err = this.iprintf("}\n")
 			n += nn; if err != nil { return n, err }
 			nn, err = this.iprintf("if itemTag.Is(tape.SBA) { itemTag += 1 << 5 }\n")
 			n += nn; if err != nil { return n, err }
+			nn, err = this.iprintf("nn, err = encoder.WriteTag(itemTag)\n")
+			n += nn; if err != nil { return n, err }
+			n += nn; if err != nil { return n, err }
+			nn, err = this.generateErrorCheck()
 			nn, err = this.iprintf("for _, item := range %s {\n", valueSource)
 			n += nn; if err != nil { return n, err }
 			this.push()
@@ -311,8 +417,8 @@ func (this *Generator) generateEncodeValue(typ Type, valueSource, tagSource stri
 	case TypeTableDefined:
 		// KTV: <length: UN> (<key: U16> <tag: Tag> <value>)*
 		nn, err := this.iprintf(
-			"nn, err = encoder.WriteUintN(%s.CN(), %d)\n",
-			tagSource, len(typ.Fields))
+			"nn, err = encoder.WriteUintN(%d, %s.CN())\n",
+			len(typ.Fields), tagSource)
 		n += nn; if err != nil { return n, err }
 		nn, err = this.generateErrorCheck()
 		n += nn; if err != nil { return n, err }
@@ -349,8 +455,10 @@ func (this *Generator) generateEncodeValue(typ Type, valueSource, tagSource stri
 		n += nn; if err != nil { return n, err }
 		nn, err = this.generateErrorCheck()
 		n += nn; if err != nil { return n, err }
+	default:
+		panic(fmt.Errorf("unknown type: %T", typ))
 	}
-
+	
 	return n, nil
 }
 
@@ -364,8 +472,357 @@ func (this *Generator) generateEncodeValue(typ Type, valueSource, tagSource stri
 //   - n int
 //   - err error
 //   - nn int
-func (this *Generator) generateDecodeValue(typ Type, valueSource, tagSource string) (n int, err error) {
-	// TODO
+//
+// The typeName paramterer is handled in the way described in the documentation
+// for [Generator.generateDecodeBranch].
+func (this *Generator) generateDecodeValue(typ Type, typeName, valueSource, tagSource string) (n int, err error) {
+	switch typ := typ.(type) {
+	case TypeInt:
+		// SI: (none)
+		// LI: <value: IntN>
+		if typ.Bits <= 5 {
+			// SI stores the value in the tag
+			nn, err := this.iprintf("*%s = uint8(%s.CN())\n", valueSource, tagSource)
+			n += nn; if err != nil { return n, err }
+			break
+		}
+		prefix := "ReadUint"
+		if typ.Signed {
+			prefix = "ReadInt"
+		}
+		nn, err := this.iprintf("*%s, nn, err = decoder.%s%d()\n", valueSource, prefix, typ.Bits)
+		n += nn; if err != nil { return n, err }
+		nn, err = this.generateErrorCheck()
+		n += nn; if err != nil { return n, err }
+	case TypeFloat:
+		// FP: <value: FloatN>
+		nn, err := this.iprintf("*%s, nn, err = decoder.ReadFloat%d()\n", valueSource, typ.Bits)
+		n += nn; if err != nil { return n, err }
+		nn, err = this.generateErrorCheck()
+		n += nn; if err != nil { return n, err }
+	case TypeString, TypeBuffer:
+		// SBA: <data: U8>*
+		// LBA: <length: UN> <data: U8>*
+		lengthVar := this.newTemporaryVar("length")
+		nn, err := this.iprintf("var %s uint64\n", lengthVar)
+		n += nn; if err != nil { return n, err }
+		nn, err = this.iprintf("if %s.Is(tape.LBA) {\n", tagSource)
+		n += nn; if err != nil { return n, err }
+		this.push()
+			nn, err = this.iprintf(
+				"%s, nn, err = decoder.ReadUintN(int(%s.CN()))\n",
+				lengthVar, tagSource)
+			n += nn; if err != nil { return n, err }
+			nn, err = this.generateErrorCheck()
+			n += nn; if err != nil { return n, err }
+		this.pop()
+		nn, err = this.iprintf("} else {\n")
+		n += nn; if err != nil { return n, err }
+		this.push()
+			nn, err = this.iprintf(
+				"%s = uint64(%s.CN())\n",
+				lengthVar, tagSource)
+			n += nn; if err != nil { return n, err }
+		this.pop()
+		nn, err = this.iprintf("}\n")
+		n += nn; if err != nil { return n, err }
+		nn, err = this.iprintf("buffer := make([]byte, int(%s))\n", lengthVar)
+		n += nn; if err != nil { return n, err }
+		nn, err = this.iprintf("nn, err = decoder.Read(buffer)\n")
+		n += nn; if err != nil { return n, err }
+		nn, err = this.generateErrorCheck()
+		n += nn; if err != nil { return n, err }
+		if _, ok := typ.(TypeString); ok {
+			nn, err = this.iprintf("*%s = string(buffer)\n", valueSource)
+			n += nn; if err != nil { return n, err }
+		} else {
+			nn, err = this.iprintf("*%s = buffer\n", valueSource)
+			n += nn; if err != nil { return n, err }
+		}
+	case TypeArray:
+		// OTA: <length: UN> <elementTag: tape.Tag> <values>*
+		nn, err := this.generateDecodeBranchCall(typ, typeName, valueSource, tagSource)
+		n += nn; if err != nil { return n, err }
+	case TypeTable:
+		// KTV: <length: UN> (<key: U16> <tag: Tag> <value>)*
+		nn, err := this.iprintf(
+			"nn, err = tape.DecodeAny(decoder, %s, %s)\n",
+			valueSource, tagSource)
+		n += nn; if err != nil { return n, err }
+		nn, err = this.generateErrorCheck()
+		n += nn; if err != nil { return n, err }
+	case TypeTableDefined:
+		// KTV: <length: UN> (<key: U16> <tag: Tag> <value>)*
+		nn, err := this.generateDecodeBranchCall(typ, typeName, valueSource, tagSource)
+		n += nn; if err != nil { return n, err }
+	case TypeNamed:
+		// WHATEVER: [WHATEVER]
+		nn, err := this.iprintf("nn, err = %s.DecodeValue(decoder, %s)\n", valueSource, tagSource)
+		n += nn; if err != nil { return n, err }
+		nn, err = this.generateErrorCheck()
+		n += nn; if err != nil { return n, err }
+	default:
+		panic(fmt.Errorf("unknown type: %T", typ))
+	}
+	
+	return n, nil
+}
+
+// generateDecodeBranchCall generates code to call an aggregate decoder function,
+// for a specified type. The definition of the function is deferred so no
+// duplicates are created. The function overwrites memory pointed to by the
+// variable (or parenthetical statement) specified by valueSource, and the value
+// will be encoded according to the tag stored in the variable (or parenthetical
+// statement) specified by tagSource. the code generated is a BLOCK and expects
+// these variables to be defined:
+//
+//   - decoder *tape.Decoder
+//   - n int
+//   - err error
+//   - nn int
+//
+// The typeName paramterer is handled in the way described in the documentation
+// for [Generator.generateDecodeBranch].
+func (this *Generator) generateDecodeBranchCall(typ Type, typeName, valueSource, tagSource string) (n int, err error) {
+	hash := HashType(typ)
+	nn, err := this.iprintf(
+		"nn, err = %s(%s, decoder, %s)\n",
+		this.decodeBranchName(hash, typeName), valueSource, tagSource)
+	n += nn; if err != nil { return n, err }
+	nn, err = this.generateErrorCheck()
+	n += nn; if err != nil { return n, err }
+	this.pushDecodeBranchRequest(hash, typ, typeName)
+	return n, nil
+}
+
+// generateDecodeBranch generates an aggregate decoder function definition for a
+// specified type. It assumes that hash == HashType(typ). If typeName is not
+// empty, it will be used as the type in the argument list instead of the result
+// of [Generator.generateType].
+func (this *Generator) generateDecodeBranch(hash [16]byte, typ Type, typeName string) (n int, err error) {
+	nn, err := this.iprintf("\nfunc %s(this *", this.decodeBranchName(hash, typeName))
+	n += nn; if err != nil { return n, err }
+	if typeName == "" {
+		nn, err = this.generateType(typ)
+		n += nn; if err != nil { return n, err }
+	} else {
+		nn, err = this.print(typeName)
+		n += nn; if err != nil { return n, err }	
+	}
+	nn, err = this.printf(", decoder *tape.Decoder, tag tape.Tag) (n int, err error) {\n")
+	n += nn; if err != nil { return n, err }
+	this.push()
+	
+		nn, err = this.iprintf("var nn int\n")
+		n += nn; if err != nil { return n, err }
+		
+		switch typ := typ.(type) {
+		case TypeArray:
+			// OTA: <length: UN> <elementTag: tape.Tag> <values>*
+			// read header
+			lengthVar := this.newTemporaryVar("length")
+			nn, err := this.iprintf("var %s uint64\n", lengthVar)
+			n += nn; if err != nil { return n, err }
+			nn, err = this.iprintf("%s, nn, err = decoder.ReadUintN(int(tag.CN()))\n", lengthVar)
+			n += nn; if err != nil { return n, err }
+			nn, err = this.generateErrorCheck()
+			n += nn; if err != nil { return n, err }
+			elementTagVar := this.newTemporaryVar("elementTag")
+			nn, err = this.iprintf("var %s tape.Tag\n", elementTagVar)
+			n += nn; if err != nil { return n, err }
+			nn, err = this.iprintf("%s, nn, err = decoder.ReadTag()\n", elementTagVar)
+			n += nn; if err != nil { return n, err }
+			nn, err = this.generateErrorCheck()
+			n += nn; if err != nil { return n, err }
+
+			// abort macro
+			abort := func() (n int, err error) {
+				// skim entire array
+				nn, err = this.iprintf("for _ = range %s {\n", lengthVar)
+				n += nn; if err != nil { return n, err }
+				this.push()
+					nn, err = this.iprintf("nn, err = tape.Skim(decoder, %s)\n", elementTagVar)
+					n += nn; if err != nil { return n, err }
+					nn, err = this.generateErrorCheck()
+					n += nn; if err != nil { return n, err }
+				this.pop()
+				nn, err = this.iprintf("}\n")
+				n += nn; if err != nil { return n, err }
+				nn, err = this.iprintf("return n, nil\n")
+				n += nn; if err != nil { return n, err }
+				return n, nil
+			}
+	
+			// validate header
+			// TODO: here, validate that length is less than the
+			// max, whatever that is configured to be. the reason we
+			// want to read it here is that we would have to skip
+			// the tag anyway so why not.
+			nn, err = this.iprintf("if !(")
+			n += nn; if err != nil { return n, err }
+			nn, err = this.generateCanAssign(typ.Element, elementTagVar)
+			n += nn; if err != nil { return n, err }
+			nn, err = this.printf(") {\n")
+			n += nn; if err != nil { return n, err }
+			this.push()
+				nn, err = abort()
+				n += nn; if err != nil { return n, err }
+			this.pop()
+			nn, err = this.iprintf("}\n")
+			n += nn; if err != nil { return n, err }
+
+			// decode payloads
+			nn, err = this.iprintf("*this = make(")
+			n += nn; if err != nil { return n, err }
+			nn, err = this.generateType(typ)
+			n += nn; if err != nil { return n, err }
+			nn, err = this.printf(", %s)\n", lengthVar)
+			n += nn; if err != nil { return n, err }
+			nn, err = this.iprintf("for index := range int(%s) {\n", lengthVar)
+			n += nn; if err != nil { return n, err }
+			this.push()
+				nn, err = this.generateDecodeValue(typ.Element, "", "(&(*this)[index])", elementTagVar)
+				n += nn; if err != nil { return n, err }
+			this.pop()
+			nn, err = this.iprintf("}\n")
+			n += nn; if err != nil { return n, err }
+		case TypeTableDefined:
+			// KTV: <length: UN> (<key: U16> <tag: Tag> <value>)*
+			// read header
+			lengthVar := this.newTemporaryVar("length")
+			nn, err := this.iprintf("var %s uint64\n", lengthVar)
+			n += nn; if err != nil { return n, err }
+			nn, err = this.iprintf("%s, nn, err = decoder.ReadUintN(int(tag.CN()))\n", lengthVar)
+			n += nn; if err != nil { return n, err }
+			nn, err = this.generateErrorCheck()
+			n += nn; if err != nil { return n, err }
+			
+			// validate header
+			// TODO: here, validate that length is less than the
+			// max, whatever that is configured to be. if not, stop
+			// ALL decoding. skimming huge big ass data could cause
+			// problems
+
+			// read fields
+			nn, err = this.iprintf("for _ = range int(%s) {\n", lengthVar)
+			n += nn; if err != nil { return n, err }
+			this.push()
+				// read field header
+				fieldKeyVar := this.newTemporaryVar("fieldKey")
+				nn, err = this.iprintf("var %s uint16\n", fieldKeyVar)
+				n += nn; if err != nil { return n, err }
+				nn, err = this.iprintf("%s, nn, err = decoder.ReadUint16()\n", fieldKeyVar)
+				n += nn; if err != nil { return n, err }
+				nn, err = this.generateErrorCheck()
+				n += nn; if err != nil { return n, err }
+				fieldTagVar := this.newTemporaryVar("fieldTag")
+				nn, err = this.iprintf("var %s tape.Tag\n", fieldTagVar)
+				n += nn; if err != nil { return n, err }
+				nn, err = this.iprintf("%s, nn, err = decoder.ReadTag()\n", fieldTagVar)
+				n += nn; if err != nil { return n, err }
+				nn, err = this.generateErrorCheck()
+				n += nn; if err != nil { return n, err }
+
+				// abort field macro
+				abortField := func() (n int, err error) {
+					nn, err = this.iprintf("tape.Skim(decoder, %s)\n", fieldTagVar)
+					n += nn; if err != nil { return n, err }
+					nn, err = this.iprintf("continue\n")
+					n += nn; if err != nil { return n, err }
+					return n, nil
+				}
+
+				// switch on tag
+				nn, err = this.iprintf("switch %s {\n", fieldKeyVar)
+				n += nn; if err != nil { return n, err }
+				for _, key := range slices.Sorted(maps.Keys(typ.Fields)) {
+					field := typ.Fields[key]
+					nn, err = this.iprintf("case 0x%04X:\n", key)
+					n += nn; if err != nil { return n, err }
+					this.push()
+
+					// validate field header
+					nn, err = this.iprintf("if !(")
+					n += nn; if err != nil { return n, err }
+					nn, err = this.generateCanAssign(field.Type, fieldTagVar)
+					n += nn; if err != nil { return n, err }
+					nn, err = this.printf(") {\n")
+					n += nn; if err != nil { return n, err }
+					this.push()
+						nn, err = abortField()
+						n += nn; if err != nil { return n, err }
+					this.pop()
+					nn, err = this.iprintf("}\n")
+					n += nn; if err != nil { return n, err }
+
+					// decode payload
+					nn, err = this.generateDecodeValue(
+						field.Type, "",
+						fmt.Sprintf("(&(this.%s))", field.Name), fieldTagVar)
+					n += nn; if err != nil { return n, err }
+					this.pop()
+				}					
+				nn, err = this.iprintf("default:\n")
+				n += nn; if err != nil { return n, err }
+				this.push()
+					abortField()
+				this.pop()
+				nn, err = this.iprintf("}\n")
+				n += nn; if err != nil { return n, err }
+			this.pop()
+			nn, err = this.iprintf("}\n")
+			n += nn; if err != nil { return n, err }
+			
+			// TODO once options are implemented, have a set of
+			// bools for each non-optional field, and check here
+			// that they are all true. a counter will not work
+			// because if someone specifies a non-optional field
+			// twice, they can neglect to specify another
+			// non-optional field and we won't even know because the
+			// count will still be even. we shouldn't use a map
+			// either because its an allocation and its way more
+			// memory than just, like 5 bools (on the stack no less)
+		default: return n, fmt.Errorf("unexpected type: %T", typ)
+		}
+
+		nn, err = this.iprintf("return n, nil\n")
+		
+	this.pop()
+	nn, err = this.iprintf("}\n")
+	n += nn; if err != nil { return n, err }
+	return n, nil
+}
+
+func (this *Generator) decodeBranchName(hash [16]byte, name string) string {
+	if name == "" {
+		return fmt.Sprintf("decodeBranch_%s", hex.EncodeToString(hash[:]))
+	} else {
+		return fmt.Sprintf("decodeBranch_%s_%s", hex.EncodeToString(hash[:]), name)
+	}
+}
+
+// pushDecodeBranchRequest pushes a new branch decode function request to the
+// back of the queue, if it is not already in the queue.
+func (this *Generator) pushDecodeBranchRequest(hash [16]byte, typ Type, name string) {
+	for _, item := range this.decodeBranchRequestQueue {
+		if item.hash == hash && item.name == name { return }
+	}
+	this.decodeBranchRequestQueue = append(this.decodeBranchRequestQueue, decodeBranchRequest {
+		hash: hash,
+		typ:  typ,
+		name: name,
+	})
+}
+
+// pullDecodeBranchRequest pulls a branch decode function request from the front
+// of the queue.
+func (this *Generator) pullDecodeBranchRequest() (hash [16]byte, typ Type, name string, ok bool) {
+	if len(this.decodeBranchRequestQueue) < 1 {
+		return [16]byte { }, nil, "", false
+	}
+	request := this.decodeBranchRequestQueue[0]
+	this.decodeBranchRequestQueue = this.decodeBranchRequestQueue[1:]
+	return request.hash, request.typ, request.name, true
 }
 
 func (this *Generator) generateErrorCheck() (n int, err error) {
@@ -378,35 +835,37 @@ func (this *Generator) generateTag(typ Type, source string) (n int, err error) {
 	switch typ := typ.(type) {
 	case TypeInt:
 		if typ.Bits <= 5 {
-			nn, err := this.printf("tape.TagSI")
+			nn, err := this.printf("tape.SI.WithCN(int(%s))", source)
 			n += nn; if err != nil { return n, err }
 		} else {
-			nn, err := this.printf("tape.TagLI.WithCN(%d)", bitsToCN(typ.Bits))
+			nn, err := this.printf("tape.LI.WithCN(%d)", bitsToCN(typ.Bits))
 			n += nn; if err != nil { return n, err }
 		}
 	case TypeFloat:
-		nn, err := this.printf("tape.TagFP.WithCN(%d)", bitsToCN(typ.Bits))
+		nn, err := this.printf("tape.FP.WithCN(%d)", bitsToCN(typ.Bits))
 		n += nn; if err != nil { return n, err }
 	case TypeString:
-		nn, err := this.generateTag(TypeBuffer { }, source)
+		nn, err := this.printf("tape.StringTag(%s)", source)
 		n += nn; if err != nil { return n, err }
 	case TypeBuffer:
-		nn, err := this.printf("bufferTag(%s)", source)
+		nn, err := this.printf("tape.BufferTag(%s)", source)
 		n += nn; if err != nil { return n, err }
 	case TypeArray:
-		nn, err := this.printf("arrayTag(tape.TagOTA.WithCN(tape.IntBytes(uint64(len(%s))))", source)
+		nn, err := this.printf("tape.OTA.WithCN(tape.IntBytes(uint64(len(%s))))", source)
 		n += nn; if err != nil { return n, err }
 	case TypeTable:
-		nn, err := this.printf("tape.TagKTV.WithCN(tape.IntBytes(uint64(len(%s))))", source)
+		nn, err := this.printf("tape.KTV.WithCN(tape.IntBytes(uint64(len(%s))))", source)
 		n += nn; if err != nil { return n, err }
 	case TypeTableDefined:
-		nn, err := this.printf("tape.TagKTV.WithCN(%d)", tape.IntBytes(uint64(len(typ.Fields))))
+		nn, err := this.printf("tape.KTV.WithCN(%d)", tape.IntBytes(uint64(len(typ.Fields))))
 		n += nn; if err != nil { return n, err }
 	case TypeNamed:
 		resolved, err := this.resolveTypeName(typ.Name)
 		if err != nil { return n, err }
 		nn, err := this.generateTag(resolved, source)
 		n += nn; if err != nil { return n, err }
+	default:
+		panic(fmt.Errorf("unknown type: %T", typ))
 	}
 
 	return n, nil
@@ -420,29 +879,29 @@ func (this *Generator) generateTN(typ Type) (n int, err error) {
 	switch typ := typ.(type) {
 	case TypeInt:
 		if typ.Bits <= 5 {
-			nn, err := this.printf("tape.TagSI")
+			nn, err := this.printf("tape.SI")
 			n += nn; if err != nil { return n, err }
 		} else {
-			nn, err := this.printf("tape.TagLI")
+			nn, err := this.printf("tape.LI")
 			n += nn; if err != nil { return n, err }
 		}
 	case TypeFloat:
-		nn, err := this.printf("tape.TagFP",)
+		nn, err := this.printf("tape.FP",)
 		n += nn; if err != nil { return n, err }
 	case TypeString:
 		nn, err := this.generateTN(TypeBuffer { })
 		n += nn; if err != nil { return n, err }
 	case TypeBuffer:
-		nn, err := this.printf("tape.TagLBA")
+		nn, err := this.printf("tape.LBA")
 		n += nn; if err != nil { return n, err }
 	case TypeArray:
-		nn, err := this.printf("tape.TagOTA")
+		nn, err := this.printf("tape.OTA")
 		n += nn; if err != nil { return n, err }
 	case TypeTable:
-		nn, err := this.printf("tape.TagKTV")
+		nn, err := this.printf("tape.KTV")
 		n += nn; if err != nil { return n, err }
 	case TypeTableDefined:
-		nn, err := this.printf("tape.TagKTV")
+		nn, err := this.printf("tape.KTV")
 		n += nn; if err != nil { return n, err }
 	case TypeNamed:
 		resolved, err := this.resolveTypeName(typ.Name)
@@ -460,6 +919,11 @@ func (this *Generator) generateType(typ Type) (n int, err error) {
 		if err := this.validateIntBitSize(typ.Bits); err != nil {
 			return n, err
 		}
+		if typ.Bits <= 5 {
+			nn, err := this.printf("uint8")
+			n += nn; if err != nil { return n, err }
+			break
+		}
 		if typ.Signed {
 			nn, err := this.printf("int%d", typ.Bits)
 			n += nn; if err != nil { return n, err }
@@ -496,9 +960,7 @@ func (this *Generator) generateType(typ Type) (n int, err error) {
 		nn, err := this.generateTypeTableDefined(typ)
 		n += nn; if err != nil { return n, err }
 	case TypeNamed:
-		actual, err := this.resolveTypeName(typ.Name)
-		if err != nil { return n, err }
-		nn, err := this.generateType(actual)
+		nn, err := this.print(typ.Name)
 		n += nn; if err != nil { return n, err }
 	}
 	return n, nil
@@ -525,9 +987,22 @@ func (this *Generator) generateTypeTableDefined(typ TypeTableDefined) (n int, er
 	return n, nil
 }
 
+// generateCanAssign generates an expression which checks if the tag specified
+// by tagSource can be assigned to a Go destination generated from typ. The
+// generated code is INLINE.
+func (this *Generator) generateCanAssign(typ Type, tagSource string) (n int, err error) {
+	nn, err := this.printf("canAssign(")
+	n += nn; if err != nil { return n, err }
+	nn, err = this.generateTN(typ)
+	n += nn; if err != nil { return n, err }
+	nn, err = this.printf(", %s)", tagSource)
+	n += nn; if err != nil { return n, err }
+	return n, nil
+}
+
 func (this *Generator) validateIntBitSize(size int) error {
 	switch size {
-	case 8, 16, 32, 64: return nil
+	case 5, 8, 16, 32, 64: return nil
 	default: return fmt.Errorf("integers of size %d are unsupported on this platform", size)
 	}
 }
@@ -582,30 +1057,7 @@ func (this *Generator) resolveMessageName(message string) string {
 	return "Message" + message
 }
 
-func (this *Generator) resolveTypeName(name string) (Type, error) {
-	switch name {
-	case "U8":     return TypeInt { Bits: 8   }, nil
-	case "U16":    return TypeInt { Bits: 16  }, nil
-	case "U32":    return TypeInt { Bits: 32  }, nil
-	case "U64":    return TypeInt { Bits: 64  }, nil
-	case "U128":   return TypeInt { Bits: 128 }, nil
-	case "U256":   return TypeInt { Bits: 256 }, nil
-	case "I8":     return TypeInt { Bits: 8,   Signed: true }, nil
-	case "I16":    return TypeInt { Bits: 16,  Signed: true }, nil
-	case "I32":    return TypeInt { Bits: 32,  Signed: true }, nil
-	case "I64":    return TypeInt { Bits: 64,  Signed: true }, nil
-	case "I128":   return TypeInt { Bits: 128, Signed: true }, nil
-	case "I256":   return TypeInt { Bits: 256, Signed: true }, nil
-	case "F16":    return TypeFloat { Bits: 16  }, nil
-	case "F32":    return TypeFloat { Bits: 32  }, nil
-	case "F64":    return TypeFloat { Bits: 64  }, nil
-	case "F128":   return TypeFloat { Bits: 128 }, nil
-	case "F256":   return TypeFloat { Bits: 256 }, nil
-	case "String": return TypeString { }, nil
-	case "Buffer": return TypeBuffer { }, nil
-	case "Table":  return TypeTable { }, nil
-	}
-	
+func (this *Generator) resolveTypeName(name string) (Type, error) {	
 	if typ, ok := this.protocol.Types[name]; ok {
 		if typ, ok := typ.(TypeNamed); ok {
 			return this.resolveTypeName(typ.Name)
@@ -616,6 +1068,11 @@ func (this *Generator) resolveTypeName(name string) (Type, error) {
 	return nil, fmt.Errorf("no type exists called %s", name)
 }
 
+func (this *Generator) newTemporaryVar(base string) string {
+	this.temporaryVar += 1
+	return fmt.Sprintf("%s_%d", base, this.temporaryVar)
+}
+
 func bitsToBytes(bits int) int {
 	return int(math.Ceil(float64(bits) / 8.0))
 }

generate/generate_test.go

@@ -0,0 +1,327 @@
+package generate
+
+// import "fmt"
+import "strings"
+import "testing"
+import "git.tebibyte.media/sashakoshka/goparse"
+
+var testGenerateCorrect =
+`package protocol
+
+/* # Do not edit this package by hand!
+ * 
+ * This file was automatically generated by the Holanet PDL compiler. The
+ * source file is located at input.pdl
+ * Please edit that file instead, and re-compile it to this location.
+ *
+ * HOPP, TAPE, METADAPT, PDL/0 (c) 2025 holanet.xyz
+ */
+
+import "git.tebibyte.media/sashakoshka/hopp/tape"
+
+// Table is a KTV table with an undefined schema.
+type Table map[uint16] any
+
+// Message is any message that can be sent along this protocol.
+type Message interface {
+	tape.Encodable
+	tape.Decodable
+
+	// Method returns the method code of the message.
+	Method() uint16
+}
+
+// User represents the protocol data type User.
+type User struct {
+	Name string
+	Bio string
+	Followers uint32
+}
+
+// EncodeValue encodes the value of this type without the tag. The value is
+// encoded according to the parameters specified by the tag, if possible.
+func (this *User) EncodeValue(encoder *tape.Encoder) (n int, err error) {
+	nn, err := tape.WriteTableHeader(2)
+	n += nn; if err != nil { return n, err }
+	nn, err := encoder.WriteUint16(0x0000)
+	n += nn; if err != nil { return n, err }
+	nn, err := tape.WriteString(encoder, this.Name)
+	n += nn; if err != nil { return n, err }
+	nn, err := encoder.WriteUint16(0x0001)
+	n += nn; if err != nil { return n, err }
+	nn, err := tape.WriteString(encoder, this.Bio)
+	n += nn; if err != nil { return n, err }
+	return n, nil
+}
+
+// Decode replaces the data in this User with information from the decoder.
+func (this *User) Decode(decoder *tape.Decoder) (n int, err error) {
+	pull, nn, err := tape.ReadTableHeader(decoder)
+	n += nn; if err != nil { return n, err }
+
+	for {
+		key, tag, end, nn, err := pull()
+		n += nn; if err != nil { return n, err }
+		if end { break }
+
+		switch key {
+		case 0x0000:
+			value, nn, err := tape.ReadString(decoder)
+			n += nn; if err != nil { return n, err }
+			this.Name = value
+		case 0x0001:
+			value, nn, err := tape.ReadString(decoder)
+			n += nn; if err != nil { return n, err }
+			this.Bio = value 
+		}
+	}
+	
+	return n, nil
+}
+
+// MessageConnect represents the protocol message M0000 Connect.
+type MessageConnect struct {
+	Name string
+	Password string
+}
+
+// Method returns the method code, M0000.
+func (this *MessageConnect) Method() uint16 {
+	return 0x0000
+}
+
+// Encode encodes the message to the encoder.
+func (this *MessageConnect) Encode(encoder *tape.Encoder) (n int, err error) {
+	nn, err := tape.WriteTableHeader(2)
+	n += nn; if err != nil { return n, err }
+	nn, err := encoder.WriteUint16(0x0000)
+	n += nn; if err != nil { return n, err }
+	nn, err := tape.WriteString(encoder, this.Name)
+	n += nn; if err != nil { return n, err }
+	nn, err := encoder.WriteUint16(0x0001)
+	n += nn; if err != nil { return n, err }
+	nn, err := tape.WriteString(encoder, this.Password)
+	n += nn; if err != nil { return n, err }
+	return n, nil
+}
+
+// Decode replaces the data in this message with information from the decoder.
+func (this *MessageConnect) Decode(decoder *tape.Decoder) (n int, err error) {
+	pull, nn, err := tape.ReadTableHeader(decoder)
+	n += nn; if err != nil { return n, err }
+
+	for {
+		key, tag, end, nn, err := pull()
+		n += nn; if err != nil { return n, err }
+		if end { break }
+
+		switch key {
+		case 0x0000:
+			value, nn, err := tape.ReadString(decoder)
+			n += nn; if err != nil { return n, err }
+			this.Name = value
+		case 0x0001:
+			value, nn, err := tape.ReadString(decoder)
+			n += nn; if err != nil { return n, err }
+			this.Password = value 
+		}
+	}
+	
+	return n, nil
+}
+
+// MessageUserList represents the protocol message M0001 UserList.
+type MessageUserList struct {
+	Users []User
+}
+
+// Method returns the method code, M0001.
+func (this *MessageUserList) Method() uint16 {
+	return 0x0001
+}
+
+// TODO methods
+`
+
+func TestGenerate(test *testing.T) {
+	protocol := defaultProtocol()
+	protocol.Messages[0x0000] = Message {
+		Name: "Connect",
+		Type: TypeTableDefined {
+			Fields: map[uint16] Field {
+				0x0000: Field { Name: "Name",      Type: TypeString { } },
+				0x0001: Field { Name: "Password",  Type: TypeString { } },
+			},
+		},
+	}
+	protocol.Messages[0x0001] = Message {
+		Name: "UserList",
+		Type: TypeTableDefined {
+			Fields: map[uint16] Field {
+				0x0000: Field { Name: "Users",  Type: TypeArray { Element: TypeNamed { Name: "User" } } },
+			},
+		},
+	}
+	protocol.Types["User"] = TypeTableDefined {
+		Fields: map[uint16] Field {
+			0x0000: Field { Name: "Name",      Type: TypeString { } },
+			0x0001: Field { Name: "Bio",       Type: TypeString { } },
+			0x0002: Field { Name: "Followers", Type: TypeInt { Bits: 32 } },
+		},
+	}
+
+	correct := testGenerateCorrect
+
+	builder := strings.Builder { }
+	generator := Generator { Output: &builder }
+	/* TODO test n: */ _, err := generator.Generate(&protocol)
+	if err != nil { test.Fatal(parse.Format(err)) }
+	got := builder.String()
+
+	test.Log("CORRECT:")
+	test.Log(correct)
+	test.Log("GOT:")
+	test.Log(got)
+
+	if correct != got {
+		test.Error("not equal")
+		for index := range min(len(correct), len(got)) {
+			if correct[index] == got[index] { continue }
+			test.Log("C:", correct[max(0, index - 8):min(len(correct), index + 8)])
+			test.Log("G:", got[max(0, index - 8):min(len(got), index + 8)])
+			break
+		}
+		test.FailNow()
+	}
+}
+
+func TestGenerateRun(test *testing.T) {
+	protocol := defaultProtocol()
+	protocol.Messages[0x0000] = Message {
+		Name: "Connect",
+		Type: TypeTableDefined {
+			Fields: map[uint16] Field {
+				0x0000: Field { Name: "Name",      Type: TypeString { } },
+				0x0001: Field { Name: "Password",  Type: TypeString { } },
+			},
+		},
+	}
+	protocol.Messages[0x0001] = Message {
+		Name: "UserList",
+		Type: TypeTableDefined {
+			Fields: map[uint16] Field {
+				0x0000: Field { Name: "Users",  Type: TypeArray { Element: TypeNamed { Name: "User" } } },
+			},
+		},
+	}
+	protocol.Messages[0x0002] = Message {
+		Name: "Pulse",
+		Type: TypeTableDefined {
+			Fields: map[uint16] Field {
+				0x0000: Field { Name: "Index",  Type: TypeInt { Bits: 5 } },
+				0x0001: Field { Name: "Offset", Type: TypeInt { Bits: 16, Signed: true }},
+				0x0002: Field { Name: "X",      Type: TypeFloat { Bits: 16 }},
+				0x0003: Field { Name: "Y",      Type: TypeFloat { Bits: 32 }},
+				0x0004: Field { Name: "Z",      Type: TypeFloat { Bits: 64 }},
+			},
+		},
+	}
+	protocol.Messages[0x0003] = Message {
+		Name: "NestedArray",
+		Type: TypeArray { Element: TypeArray { Element: TypeInt { Bits: 8 } } },
+	}
+	protocol.Types["User"] = TypeTableDefined {
+		Fields: map[uint16] Field {
+			0x0000: Field { Name: "Name",      Type: TypeString { } },
+			0x0001: Field { Name: "Bio",       Type: TypeString { } },
+			0x0002: Field { Name: "Followers", Type: TypeInt { Bits: 32 } },
+		},
+	}
+	testGenerateRun(test, &protocol, `
+		// imports
+	`, `
+		// test case
+		log.Println("MessageConnect")
+		messageConnect := MessageConnect {
+			Name: "rarity",
+			Password: "gems",
+		}
+		testEncode(
+			&messageConnect,
+			tu.S(0xC1, 0x02).AddVar(
+				[]byte { 0x00, 0x00, 0x66, 'r', 'a', 'r', 'i', 't', 'y' },
+				[]byte { 0x00, 0x01, 0x64, 'g', 'e', 'm', 's' },
+				))
+		log.Println("MessageUserList")
+		messageUserList := MessageUserList {
+			Users: []User {
+				User {
+					Name: "rarity",
+					Bio: "asdjads",
+					Followers: 0x324,
+				},
+				User {
+					Name: "deez nuts",
+					Bio: "logy",
+					Followers: 0x8000,
+				},
+				User {
+					Name: "creekflow",
+					Bio: "im creekflow",
+					Followers: 0x3894,
+				},
+			},
+		}
+		testEncode(
+			&messageUserList,
+			tu.S(0xC1, 0x01, 0x00, 0x00,
+				0xA1, 0x03, 0xC1,
+					).Add(0x03).AddVar(
+						[]byte { 0x00, 0x00, 0x66, 'r', 'a', 'r', 'i', 't', 'y' },
+						[]byte { 0x00, 0x01, 0x67, 'a', 's', 'd', 'j', 'a', 'd', 's' },
+						[]byte { 0x00, 0x02, 0x23, 0x00, 0x00, 0x03, 0x24 },
+					).Add(0x03).AddVar(
+						[]byte { 0x00, 0x00, 0x69, 'd', 'e', 'e', 'z', ' ', 'n', 'u', 't', 's' },
+						[]byte { 0x00, 0x01, 0x64, 'l', 'o', 'g', 'y' },
+						[]byte { 0x00, 0x02, 0x23, 0x00, 0x00, 0x80, 0x00 },
+					).Add(0x03).AddVar(
+						[]byte { 0x00, 0x00, 0x69, 'c', 'r', 'e', 'e', 'k', 'f', 'l', 'o', 'w' },
+						[]byte { 0x00, 0x01, 0x6C, 'i', 'm', ' ', 'c', 'r', 'e', 'e', 'k', 'f',
+							'l', 'o', 'w' },
+						[]byte { 0x00, 0x02, 0x23, 0x00, 0x00, 0x38, 0x94 },
+				))
+		log.Println("MessagePulse")
+		messagePulse := MessagePulse {
+			Index: 9,
+			Offset: -0x3521,
+			X: 45.389,
+			Y: 294.1,
+			Z: 384729384.234892034,
+		}
+		testEncode(
+			&messagePulse,
+			tu.S(0xC1, 0x05).AddVar(
+				[]byte { 0x00, 0x00, 0x09 },
+				[]byte { 0x00, 0x01, 0x21, 0xCA, 0xDF },
+				[]byte { 0x00, 0x02, 0x41, 0x51, 0xAC },
+				[]byte { 0x00, 0x03, 0x43, 0x43, 0x93, 0x0C, 0xCD },
+				[]byte { 0x00, 0x04, 0x47, 0x41, 0xB6, 0xEE, 0x81, 0x28, 0x3C, 0x21, 0xE2 },
+				))
+		log.Println("MessageNestedArray")
+		uint8s := func(n int) []uint8 {
+			array := make([]uint8, n)
+			for index := range array {
+				array[index] = uint8(index + 1) | 0xF0
+			}
+			return array
+		}
+		messageNestedArray := MessageNestedArray {
+			uint8s(6),
+			uint8s(35),
+		}
+		testEncode(
+			&messageNestedArray,
+			tu.S(0xA1, // TODO
+				))
+	`)
+}

generate/misc_test.go

@@ -0,0 +1,73 @@
+package generate
+
+import "os"
+import "fmt"
+import "os/exec"
+import "testing"
+import "path/filepath"
+
+func testGenerateRun(test *testing.T, protocol *Protocol, imports string, testCase string) {
+	// reset data directory
+	dir := "test/generate-run"
+	err := os.RemoveAll(dir)
+	if err != nil { test.Fatal(err) }
+	err = os.MkdirAll(dir, 0750)
+	if err != nil { test.Fatal(err) }
+
+	// open files
+	sourceFile, err := os.Create(filepath.Join(dir, "protocol.go"))
+	if err != nil { test.Fatal(err) }
+	defer sourceFile.Close()
+	mainFile, err := os.Create(filepath.Join(dir, "main.go"))
+	if err != nil { test.Fatal(err) }
+	defer mainFile.Close()
+
+	// generate protocol
+	generator := Generator {
+		Output: sourceFile,
+		PackageName: "main",
+	}
+	_, err = generator.Generate(protocol)
+	if err != nil { test.Fatal(err) }
+
+	// build static source files
+	imports = `
+		import "log"
+		import "bytes"
+		import "git.tebibyte.media/sashakoshka/hopp/tape"
+		import tu "git.tebibyte.media/sashakoshka/hopp/internal/testutil"
+		` + imports
+	setup := `log.Println("*** BEGIN TEST CASE OUTPUT ***")`
+	teardown := `log.Println("--- END TEST CASE OUTPUT ---")`
+	static := `
+		func testEncode(message Message, correct tu.Snake) {
+			buffer := bytes.Buffer { }
+			encoder := tape.NewEncoder(&buffer)
+			n, err := message.Encode(encoder)
+			if err != nil { log.Fatalf("at %d: %v\n", n, err) }
+			encoder.Flush()
+			got := buffer.Bytes()
+			log.Printf("got:     [%s]", tu.HexBytes(got))
+			log.Println("correct:", correct)
+			if n != len(got) {
+				log.Fatalf("n incorrect: %d != %d\n", n, len(got))
+			}
+			if ok, n := correct.Check(got); !ok {
+				log.Fatalln("not equal at", n)
+			}
+		}
+		`
+	fmt.Fprintf(
+		mainFile, "package main\n%s\nfunc main() {\n%s\n%s\n%s\n}\n%s",
+		imports, setup, testCase, teardown, static)
+
+	// build and run test
+	command := exec.Command("go", "run", "./generate/test/generate-run")
+	workingDirAbs, err := filepath.Abs("..")
+	if err != nil { test.Fatal(err) }
+	command.Dir = workingDirAbs
+	command.Env = os.Environ()
+	output, err := command.CombinedOutput()
+	test.Logf("output of %v:\n%s", command, output)
+	if err != nil { test.Fatal(err) }
+}

generate/parse.go

@@ -95,6 +95,28 @@ func (this *parser) parseType() (Type, error) {
 
 	switch this.Kind() {
 	case TokenIdent:
+		switch this.Value() {
+		case "U8":     return TypeInt { Bits: 8   }, this.Next()
+		case "U16":    return TypeInt { Bits: 16  }, this.Next()
+		case "U32":    return TypeInt { Bits: 32  }, this.Next()
+		case "U64":    return TypeInt { Bits: 64  }, this.Next()
+		case "U128":   return TypeInt { Bits: 128 }, this.Next()
+		case "U256":   return TypeInt { Bits: 256 }, this.Next()
+		case "I8":     return TypeInt { Bits: 8,   Signed: true }, this.Next()
+		case "I16":    return TypeInt { Bits: 16,  Signed: true }, this.Next()
+		case "I32":    return TypeInt { Bits: 32,  Signed: true }, this.Next()
+		case "I64":    return TypeInt { Bits: 64,  Signed: true }, this.Next()
+		case "I128":   return TypeInt { Bits: 128, Signed: true }, this.Next()
+		case "I256":   return TypeInt { Bits: 256, Signed: true }, this.Next()
+		case "F16":    return TypeFloat { Bits: 16  }, this.Next()
+		case "F32":    return TypeFloat { Bits: 32  }, this.Next()
+		case "F64":    return TypeFloat { Bits: 64  }, this.Next()
+		case "F128":   return TypeFloat { Bits: 128 }, this.Next()
+		case "F256":   return TypeFloat { Bits: 256 }, this.Next()
+		case "String": return TypeString { }, this.Next()
+		case "Buffer": return TypeBuffer { }, this.Next()
+		case "Table":  return TypeTable { }, this.Next()
+		}
 		return this.parseTypeNamed()
 	case TokenLBracket:
 		return this.parseTypeArray()

generate/parse_test.go

@@ -11,8 +11,8 @@ func TestParse(test *testing.T) {
 		Name: "Connect",
 		Type: TypeTableDefined {
 			Fields: map[uint16] Field {
-				0x0000: Field { Name: "Name",      Type: TypeNamed { Name: "String" } },
-				0x0001: Field { Name: "Password",  Type: TypeNamed { Name: "String" } },
+				0x0000: Field { Name: "Name",      Type: TypeString { } },
+				0x0001: Field { Name: "Password",  Type: TypeString { } },
 			},
 		},
 	}
@@ -26,9 +26,9 @@ func TestParse(test *testing.T) {
 	}
 	correct.Types["User"] = TypeTableDefined {
 		Fields: map[uint16] Field {
-			0x0000: Field { Name: "Name",      Type: TypeNamed { Name: "String" } },
-			0x0001: Field { Name: "Bio",       Type: TypeNamed { Name: "String" } },
-			0x0002: Field { Name: "Followers", Type: TypeNamed { Name: "U32" } },
+			0x0000: Field { Name: "Name",      Type: TypeString { } },
+			0x0001: Field { Name: "Bio",       Type: TypeString { } },
+			0x0002: Field { Name: "Followers", Type: TypeInt { Bits: 32 } },
 		},
 	}
 	test.Log("CORRECT:", &correct)

generate/protocol.go

@@ -1,5 +1,10 @@
 package generate
 
+import "fmt"
+import "maps"
+import "slices"
+import "crypto/md5"
+
 type Protocol struct {
 	Messages map[uint16] Message
 	Types    map[string] Type
@@ -11,7 +16,7 @@ type Message struct {
 }
 
 type Type interface {
-	
+	fmt.Stringer
 }
 
 type TypeInt struct {
@@ -19,29 +24,84 @@ type TypeInt struct {
 	Signed bool
 }
 
+func (typ TypeInt) String() string {
+	output := ""
+	if typ.Signed {
+		output += "I"
+	} else {
+		output += "U"
+	}
+	output += fmt.Sprint(typ.Bits)
+	return output
+}
+
 type TypeFloat struct {
 	Bits int
 }
 
+func (typ TypeFloat) String() string {
+	return fmt.Sprintf("F%d", typ.Bits)
+}
+
 type TypeString struct { }
 
+func (TypeString) String() string {
+	return "String"
+}
+
 type TypeBuffer struct { }
 
+func (TypeBuffer) String() string {
+	return "Buffer"
+}
+
 type TypeArray struct {
 	Element Type
 }
 
+func (typ TypeArray) String() string {
+	return fmt.Sprintf("[]%v", typ.Element)
+}
+
 type TypeTable struct { }
 
+func (TypeTable) String() string {
+	return "Table"
+}
+
 type TypeTableDefined struct {
 	Fields map[uint16] Field
 }
 
+func (typ TypeTableDefined) String() string {
+	output := "{"
+	for _, key := range slices.Sorted(maps.Keys(typ.Fields)) {
+		output += fmt.Sprintf("%04X %v", key, typ.Fields[key])
+	}
+	output += "}"
+	return output
+}
+
 type Field struct {
 	Name string
 	Type Type
 }
 
+func (field Field) String() string {
+	return fmt.Sprintf("%s %v", field.Name, field.Type)
+}
+
 type TypeNamed struct {
 	Name string
 }
+
+func (typ TypeNamed) String() string {
+	return typ.Name
+}
+
+func HashType(typ Type) [16]byte {
+	// TODO: if we ever want to make the compiler more efficient, this would
+	// be a good place to start, complex string concatenation in a hot path
+	// (sorta)
+	return md5.Sum([]byte(typ.String()))
+}

internal/testutil/testutil.go

@@ -0,0 +1,161 @@
+package testutil
+
+import "fmt"
+import "slices"
+import "strings"
+import "reflect"
+
+// Snake lets you compare blocks of data where the ordering of certain parts may
+// be swapped every which way. It is designed for comparing the encoding of
+// maps where the ordering of individual elements is inconsistent.
+//
+// The snake is divided into sectors, which hold a number of variations. For a
+// sector to be satisfied by some data, some ordering of it must match the data
+// exactly. for the snake to be satisfied by some data, its sectors must match
+// the data in order, but the internal ordering of each sector doesn't matter.
+type Snake []    []     []byte
+//         snake sector variation
+
+// S returns a new snake.
+func S(data ...byte) Snake {
+	return (Snake { }).Add(data...)
+}
+
+// AddVar returns a new snake with the given sector added on to it. Successive
+// calls of this method can be chained together to create a big ass snake.
+func (sn Snake) AddVar(sector ...[]byte) Snake {
+	slice := make(Snake, len(sn) + 1)
+	copy(slice, sn)
+	slice[len(slice) - 1] = sector
+	return slice
+}
+
+// Add is like AddVar, but adds a sector with only one variation, which means it
+// does not vary, hence why the method is called that.
+func (sn Snake) Add(data ...byte) Snake {
+	return sn.AddVar(data)
+}
+
+// Check determines if the data satisfies the snake.
+func (sn Snake) Check(data []byte) (ok bool, n int) {
+	left := data
+	variations := map[int] []byte { }
+	for _, sector := range sn {
+		clear(variations)
+		for key, variation := range sector {
+			variations[key] = variation
+		}
+		for len(variations) > 0 {
+			found := false
+			for key, variation := range variations {
+				if len(left) < len(variation) { continue }
+				if !slices.Equal(left[:len(variation)], variation) { continue }
+				n += len(variation)
+				left = data[n:]
+				delete(variations, key)
+				found = true
+			}
+			if !found { return false, n }
+		}
+	}
+	if n < len(data) {
+		return false, n
+	}
+	return true, n
+}
+
+func (sn Snake) String() string {
+	if len(sn) == 0 || len(sn[0]) == 0 || len(sn[0][0]) == 0 {
+		return "EMPTY"
+	}
+
+	out := strings.Builder { }
+	for index, sector := range sn {
+		if index > 0 { out.WriteString(" : ") }
+		out.WriteRune('[')
+		for index, variation := range sector {
+			if index > 0 { out.WriteString(" / ") }
+			for _, byt := range variation {
+				fmt.Fprintf(&out, "%02x", byt)
+			}
+		}
+		out.WriteRune(']')
+	}
+	return out.String()
+}
+
+// HexBytes formats bytes into a hexadecimal string.
+func HexBytes(data []byte) string {
+	if len(data) == 0 { return "EMPTY" }
+	out := strings.Builder { }
+	for _, byt := range data {
+		fmt.Fprintf(&out, "%02x", byt)
+	}
+	return out.String()
+}
+
+// Describe returns a string representing the type and data of the given value.
+func Describe(value any) string {
+	desc := describer { }
+	desc.describe(reflect.ValueOf(value))
+	return desc.String()
+}
+
+type describer struct {
+	strings.Builder
+	indent int
+}
+
+func (this *describer) describe(value reflect.Value) {
+	value = reflect.ValueOf(value.Interface())
+	switch value.Kind() {
+	case reflect.Array, reflect.Slice:
+		this.printf("[\n")
+		this.indent += 1
+		for index := 0; index < value.Len(); index ++ {
+			this.iprintf("")
+			this.describe(value.Index(index))
+			this.iprintf("\n")
+		}
+		this.indent -= 1
+		this.iprintf("]")
+	case reflect.Struct:
+		this.printf("struct {\n")
+		this.indent += 1
+		typ := value.Type()
+		for index := range typ.NumField() {
+			indexBuffer := [1]int { index }
+			this.iprintf("%s: ", typ.Field(index).Name)
+			this.describe(value.FieldByIndex(indexBuffer[:]))
+			this.iprintf("\n")
+		}
+		this.indent -= 1
+		this.iprintf("}\n")
+	case reflect.Map:
+		this.printf("map {\n")
+		this.indent += 1
+		iter := value.MapRange()
+		for iter.Next() {
+			this.iprintf("")
+			this.describe(iter.Key())
+			this.printf(": ")
+			this.describe(iter.Value())
+			this.iprintf("\n")
+		}
+		this.indent -= 1
+		this.iprintf("}\n")
+	case reflect.Pointer:
+		this.printf("& ")
+		this.describe(value.Elem())
+	default:
+		this.printf("<%v %v>", value.Type(), value.Interface())
+	}
+}
+
+func (this *describer) printf(format string, v ...any) {
+	fmt.Fprintf(this, format, v...)
+}
+
+func (this *describer) iprintf(format string, v ...any) {
+	fmt.Fprintf(this, strings.Repeat("\t", this.indent) + format, v...)
+}

internal/testutil/testutil_test.go

@@ -0,0 +1,66 @@
+package testutil
+
+import "testing"
+
+func TestSnakeA(test *testing.T) {
+	snake := S(1, 6).AddVar(
+		[]byte { 1 },
+		[]byte { 2 },
+		[]byte { 3 },
+		[]byte { 4 },
+		[]byte { 5 },
+	).Add(9)
+
+	test.Log(snake)
+
+	ok, n := snake.Check([]byte { 1, 6, 1, 2, 3, 4, 5, 9 })
+	if !ok { test.Fatal("false negative:", n) }
+	ok, n = snake.Check([]byte { 1, 6, 5, 4, 3, 2, 1, 9 })
+	if !ok { test.Fatal("false negative:", n) }
+	ok, n = snake.Check([]byte { 1, 6, 3, 1, 4, 2, 5, 9 })
+	if !ok { test.Fatal("false negative:", n) }
+	
+	ok, n = snake.Check([]byte { 1, 6, 9 })
+	if ok { test.Fatal("false positive:", n) }
+	ok, n = snake.Check([]byte { 1, 6, 1, 2, 3, 4, 5, 6, 9 })
+	if ok { test.Fatal("false positive:", n) }
+	ok, n = snake.Check([]byte { 1, 6, 0, 2, 3, 4, 5, 6, 9 })
+	if ok { test.Fatal("false positive:", n) }
+	ok, n = snake.Check([]byte { 1, 6, 7, 1, 4, 2, 5, 9 })
+	if ok { test.Fatal("false positive:", n) }
+	ok, n = snake.Check([]byte { 1, 6, 7, 3, 1, 4, 2, 5, 9 })
+	if ok { test.Fatal("false positive:", n) }
+	ok, n = snake.Check([]byte { 1, 6, 7, 3, 1, 4, 2, 5, 9 })
+	if ok { test.Fatal("false positive:", n) }
+	ok, n = snake.Check([]byte { 1, 6, 1, 2, 3, 4, 5, 9, 10})
+	if ok { test.Fatal("false positive:", n) }
+}
+
+func TestSnakeB(test *testing.T) {
+	snake := S(1, 6).AddVar(
+		[]byte { 1 },
+		[]byte { 2 },
+	).Add(9).AddVar(
+		[]byte { 3, 2 },
+		[]byte { 0 },
+		[]byte { 1, 1, 2, 3 },
+	)
+
+	test.Log(snake)
+
+	ok, n := snake.Check([]byte { 1, 6, 1, 2, 9, 3, 2, 0, 1, 1, 2, 3})
+	if !ok { test.Fatal("false negative:", n) }
+	ok, n = snake.Check([]byte { 1, 6, 2, 1, 9, 0, 1, 1, 2, 3, 3, 2})
+	if !ok { test.Fatal("false negative:", n) }
+	
+	ok, n = snake.Check([]byte { 1, 6, 9 })
+	if ok { test.Fatal("false positive:", n) }
+	ok, n = snake.Check([]byte { 1, 6, 1, 2, 9 })
+	if ok { test.Fatal("false positive:", n) }
+	ok, n = snake.Check([]byte { 1, 6, 9, 3, 2, 0, 1, 1, 2, 3})
+	if ok { test.Fatal("false positive:", n) }
+	ok, n = snake.Check([]byte { 1, 6, 2, 9, 0, 1, 1, 2, 3, 3, 2})
+	if ok { test.Fatal("false positive:", n) }
+	ok, n = snake.Check([]byte { 1, 6, 1, 2, 9, 3, 2, 1, 1, 2, 3})
+	if ok { test.Fatal("false positive:", n) }
+}

tape/decode.go

@@ -2,6 +2,7 @@ package tape
 
 import "io"
 import "math"
+import "bufio"
 
 // Decodable is any type that can decode itself from a decoder.
 type Decodable interface {
@@ -11,9 +12,16 @@ type Decodable interface {
 	Decode(decoder *Decoder) (n int, err error)
 }
 
-// Decoder wraps an [io.Reader] and decodes data from it.
+// Decoder decodes data from an [io.Reader].
 type Decoder struct {
-	io.Reader
+	bufio.Reader
+}
+
+// NewDecoder creates a new decoder that reads from reader.
+func NewDecoder(reader io.Reader) *Decoder {
+	decoder := &Decoder { }
+	decoder.Reader.Reset(reader)
+	return decoder
 }
 
 // ReadFull calls [io.ReadFull] on the reader.
@@ -21,12 +29,6 @@ func (this *Decoder) ReadFull(buffer []byte) (n int, err error) {
 	return io.ReadFull(this, buffer)
 }
 
-// ReadByte decodes a single byte from the input reader.
-func (this *Decoder) ReadByte() (value byte, n int, err error) {
-	uncasted, n, err := this.ReadUint8()
-	return byte(uncasted), n, err
-}
-
 // ReadInt8 decodes an 8-bit signed integer from the input reader.
 func (this *Decoder) ReadInt8() (value int8, n int, err error) {
 	uncasted, n, err := this.ReadUint8()
@@ -110,6 +112,13 @@ func (this *Decoder) ReadUintN(bytes int) (value uint64, n int, err error) {
 	return value, n, nil
 }
 
+// ReadFloat16 decodes a 16-bit floating point value from the input reader.
+func (this *Decoder) ReadFloat16() (value float32, n int, err error) {
+	bits, nn, err := this.ReadUint16()
+	n += nn; if err != nil { return 0, n, err }
+	return math.Float32frombits(f16bitsToF32bits(bits)), n, nil
+}
+
 // ReadFloat32 decldes a 32-bit floating point value from the input reader.
 func (this *Decoder) ReadFloat32() (value float32, n int, err error) {
 	bits, nn, err := this.ReadUint32()
@@ -130,3 +139,54 @@ func (this *Decoder) ReadTag() (value Tag, n int, err error) {
 	n += nn; if err != nil { return 0, n, err }
 	return Tag(uncasted), n, nil
 }
+
+// f16bitsToF32bits returns uint32 (float32 bits) converted from specified uint16.
+// Taken from https://github.com/x448/float16/blob/v0.8.4/float16
+//
+// MIT License
+// 
+// Copyright (c) 2019 Montgomery Edwards⁴⁴⁸ and Faye Amacker
+// 
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+// 
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+func f16bitsToF32bits(in uint16) uint32 {
+	// All 65536 conversions with this were confirmed to be correct
+	// by Montgomery Edwards⁴⁴⁸ (github.com/x448).
+	
+	sign := uint32(in&0x8000) << 16 // sign for 32-bit
+	exp := uint32(in&0x7c00) >> 10  // exponenent for 16-bit
+	coef := uint32(in&0x03ff) << 13 // significand for 32-bit
+
+	if exp == 0x1f {
+		if coef == 0 {
+			// infinity
+			return sign | 0x7f800000 | coef
+		}
+		// NaN
+		return sign | 0x7fc00000 | coef
+	}
+
+	if exp == 0 {
+		if coef == 0 {
+			// zero
+			return sign
+		}
+
+		// normalize subnormal numbers
+		exp++
+		for coef&0x7f800000 == 0 {
+			coef <<= 1
+			exp--
+		}
+		coef &= 0x007fffff
+	}
+
+	return sign | ((exp + (0x7f - 0xf)) << 23) | coef
+}

tape/dynamic.go

@@ -1,8 +1,19 @@
 package tape
 
+// dont smoke reflection, kids!!!!!!!!!
+// totally reflectric, reflectrified, etc. this is probably souper slow but
+// certainly no slower than the built in json encoder i'd imagine.
+// TODO: add support for struct tags: `tape:"0000"`, tape:"0001"` so they can get
+// transformed into tables with a defined schema
+
+// TODO: test all of these smaller functions individually
+
 import "fmt"
 import "reflect"
 
+var dummyMap map[uint16] any
+var dummyBuffer []byte
+
 // EncodeAny encodes an "any" value. Returns an error if the underlying type is
 // unsupported. Supported types are:
 //
@@ -15,25 +26,28 @@ import "reflect"
 //   - map[uint16]<supported type>
 func EncodeAny(encoder *Encoder, value any, tag Tag) (n int, err error) {
 	// primitives
-	switch value := value.(type) {
-	case int:    return encoder.WriteInt32(int32(value))
-	case uint:   return encoder.WriteUint32(uint32(value))
-	case int8:   return encoder.WriteInt8(value)
-	case uint8:  return encoder.WriteUint8(value)
-	case int16:  return encoder.WriteInt16(value)
-	case uint16: return encoder.WriteUint16(value)
-	case int32:  return encoder.WriteInt32(value)
-	case uint32: return encoder.WriteUint32(value)
-	case int64:  return encoder.WriteInt64(value)
-	case uint64: return encoder.WriteUint64(value)
-	case string: return EncodeAny(encoder, []byte(value), tag)
-	case []byte:
+	reflectValue := reflect.ValueOf(value)
+	switch reflectValue.Kind() {
+	case reflect.Int:    return encoder.WriteInt32(int32(reflectValue.Int()))
+	case reflect.Uint:   return encoder.WriteUint32(uint32(reflectValue.Uint()))
+	case reflect.Int8:   return encoder.WriteInt8(int8(reflectValue.Int()))
+	case reflect.Uint8:  return encoder.WriteUint8(uint8(reflectValue.Uint()))
+	case reflect.Int16:  return encoder.WriteInt16(int16(reflectValue.Int()))
+	case reflect.Uint16: return encoder.WriteUint16(uint16(reflectValue.Uint()))
+	case reflect.Int32:  return encoder.WriteInt32(int32(reflectValue.Int()))
+	case reflect.Uint32: return encoder.WriteUint32(uint32(reflectValue.Uint()))
+	case reflect.Int64:  return encoder.WriteInt64(int64(reflectValue.Int()))
+	case reflect.Uint64: return encoder.WriteUint64(uint64(reflectValue.Uint()))
+	case reflect.String: return EncodeAny(encoder, []byte(reflectValue.String()), tag)
+	}
+	if reflectValue.CanConvert(reflect.TypeOf(dummyBuffer)) {
 		if tag.Is(LBA) {
-			nn, err := encoder.WriteUintN(uint64(len(value)), tag.CN() + 1)
+			nn, err := encoder.WriteUintN(uint64(reflectValue.Len()), tag.CN() + 1)
 			n += nn; if err != nil { return n, err }
 		}
-		nn, err := encoder.Write(value)
+		nn, err := encoder.Write(reflectValue.Bytes())
 		n += nn; if err != nil { return n, err }
+		return n, nil
 	}
 
 	// aggregates
@@ -41,21 +55,120 @@ func EncodeAny(encoder *Encoder, value any, tag Tag) (n int, err error) {
 	switch reflectType.Kind() {
 	case reflect.Slice:
 		return encodeAnySlice(encoder, value, tag)
-	case reflect.Array:
-		return encodeAnySlice(encoder, reflect.ValueOf(value).Slice(0, reflectType.Len()).Interface(), tag)
+	// case reflect.Array:
+		// return encodeAnySlice(encoder, reflect.ValueOf(value).Slice(0, reflectType.Len()).Interface(), tag)
 	case reflect.Map:
 		if reflectType.Key() == reflect.TypeOf(uint16(0)) {
 			return encodeAnyMap(encoder, value, tag)
 		}
-		return 0, fmt.Errorf("cannot encode map key %T, key must be uint16", value)
+		return n, fmt.Errorf("cannot encode map key %T, key must be uint16", value)
 	}
-	return 0, fmt.Errorf("cannot encode type %T", value)
+	return n, fmt.Errorf("cannot encode type %T", value)
+}
+
+// DecodeAny decodes data and places it into destination, which must be a
+// pointer to a supported type. See [EncodeAny] for a list of supported types.
+func DecodeAny(decoder *Decoder, destination any, tag Tag) (n int, err error) {
+	reflectDestination := reflect.ValueOf(destination)
+	if reflectDestination.Kind() != reflect.Pointer {
+		return n, fmt.Errorf("expected pointer destination, not %v", destination)
+	}
+	return decodeAny(decoder, reflectDestination.Elem(), tag)
+}
+
+// unknownSlicePlaceholder is inserted by skeletonValue and informs the program
+// that the destination for the slice needs to be generated based on the item
+// tag in the OTA.
+type unknownSlicePlaceholder struct { }
+var unknownSlicePlaceholderType = reflect.TypeOf(unknownSlicePlaceholder { })
+
+// decodeAny is internal to [DecodeAny]. It takes in an addressable
+// [reflect.Value] as the destination.
+func decodeAny(decoder *Decoder, destination reflect.Value, tag Tag) (n int, err error) {
+	errWrongDestinationType := func(expected string) error {
+		panic(fmt.Errorf(
+		// return fmt.Errorf(
+			"expected %s destination, not %v",
+			expected, destination))
+	}
+	
+	switch tag.WithoutCN() {
+	case SI:
+		// SI: (none)
+		err = setInt(destination, uint64(tag.CN()))
+		if err != nil { return n, err }
+	case LI:
+		// LI: <value: IntN>
+		nn, err := decodeAndSetInt(decoder, destination, tag.CN() + 1)
+		n += nn; if err != nil { return n, err }
+	case FP:
+		// FP: <value: FloatN>
+		nn, err := decodeAndSetFloat(decoder, destination, tag.CN() + 1)
+		n += nn; if err != nil { return n, err }
+	case SBA:
+		// SBA: <data: U8>*
+		buffer := make([]byte, tag.CN())
+		nn, err := decoder.Read(buffer)
+		n += nn; if err != nil { return n, err }
+		err = setByteArray(destination, buffer)
+		if err != nil { return n, err }
+	case LBA:
+		// LBA: <length: UN> <data: U8>*
+		length, nn, err := decoder.ReadUintN(tag.CN() + 1)
+		n += nn; if err != nil { return n, err }
+		buffer := make([]byte, length)
+		nn, err = decoder.Read(buffer)
+		n += nn; if err != nil { return n, err }
+		err = setByteArray(destination, buffer)
+		if err != nil { return n, err }
+	case OTA:
+		// OTA: <length: UN> <elementTag: tape.Tag> <values>*
+		length, nn, err := decoder.ReadUintN(tag.CN() + 1)
+		n += nn; if err != nil { return n, err }
+		oneTag, nn, err := decoder.ReadTag()
+		n += nn; if err != nil { return n, err }
+		if destination.Kind() != reflect.Slice {
+			return n, errWrongDestinationType("slice")
+		}
+		if destination.Cap() < int(length) {
+			destination.Grow(int(length) - destination.Cap())
+		}
+		destination.SetLen(int(length))
+		for index := range length {
+			nn, err := decodeAny(decoder, destination.Index(int(index)), oneTag)
+			n += nn; if err != nil { return n, err }
+		}
+	case KTV:
+		// KTV: <length: UN> (<key: U16> <tag: Tag> <value>)*
+		table := destination
+		if table.Type() != reflect.TypeOf(dummyMap) {
+			return n, errWrongDestinationType("map[uint16] any")
+		}
+		length, nn, err := decoder.ReadUintN(tag.CN() + 1)
+		n += nn; if err != nil { return n, err }
+		table.Clear()
+		for _ = range length {
+			key, nn, err := decoder.ReadUint16()
+			n += nn; if err != nil { return n, err }
+			itemTag, nn, err := decoder.ReadTag()
+			n += nn; if err != nil { return n, err }
+			value, err := skeletonValue(decoder, itemTag)
+			if err != nil { return n, err }
+			nn, err = decodeAny(decoder, value.Elem(), itemTag)
+			n += nn; if err != nil { return n, err }
+			table.SetMapIndex(reflect.ValueOf(key), value.Elem())
+		}
+	default:
+		return n, fmt.Errorf("unknown TN %d", tag.TN())
+	}
+	return n, nil
 }
 
 // TagAny returns the correct tag for an "any" value. Returns an error if the
 // underlying type is unsupported. See [EncodeAny] for a list of supported
 // types.
 func TagAny(value any) (Tag, error) {
+	// TODO use reflection for all of this to ignore type names
 	// primitives
 	switch value := value.(type) {
 	case int,   uint:   return LI.WithCN(3), nil
@@ -70,11 +183,11 @@ func TagAny(value any) (Tag, error) {
 	// aggregates
 	reflectType := reflect.TypeOf(value)
 	switch reflectType.Kind() {
-	case reflect.Slice: return OTA.WithCN(reflect.ValueOf(value).Len()), nil
+	case reflect.Slice: return OTA.WithCN(IntBytes(uint64(reflect.ValueOf(value).Len())) - 1), nil
 	case reflect.Array: return OTA.WithCN(reflectType.Len()), nil
 	case reflect.Map:
 		if reflectType.Key() == reflect.TypeOf(uint16(0)) {
-			return OTA.WithCN(reflect.ValueOf(value).Len()), nil
+			return KTV.WithCN(IntBytes(uint64(reflect.ValueOf(value).Len())) - 1), nil
 		}
 		return 0, fmt.Errorf("cannot encode map key %T, key must be uint16", value)
 	}
@@ -89,7 +202,7 @@ func encodeAnySlice(encoder *Encoder, value any, tag Tag) (n int, err error) {
 	reflectType := reflect.TypeOf(value)
 	oneTag, err := TagAny(reflect.Zero(reflectType.Elem()).Interface())
 	if err != nil { return n, err }
-	for index := 0; index <= reflectValue.Len(); index += 1 {
+	for index := 0; index < reflectValue.Len(); index += 1 {
 		item := reflectValue.Index(index).Interface()
 		itemTag, err := TagAny(item)
 		if err != nil { return n, err }
@@ -98,7 +211,7 @@ func encodeAnySlice(encoder *Encoder, value any, tag Tag) (n int, err error) {
 	if oneTag.Is(SBA) { oneTag += 1 << 5 }
 	nn, err = encoder.WriteUint8(uint8(oneTag))
 	n += nn; if err != nil { return n, err }
-	for index := 0; index <= reflectValue.Len(); index += 1 {
+	for index := 0; index < reflectValue.Len(); index += 1 {
 		item := reflectValue.Index(index).Interface()
 		nn, err = EncodeAny(encoder, item, oneTag)
 		n += nn; if err != nil { return n, err }
@@ -118,7 +231,7 @@ func encodeAnyMap(encoder *Encoder, value any, tag Tag) (n int, err error) {
 		nn, err = encoder.WriteUint16(key)
 		n += nn; if err != nil { return n, err }
 		itemTag, err := TagAny(value)
-		n += nn; if err != nil { return n, err }
+		if err != nil { return n, err }
 		nn, err = encoder.WriteUint8(uint8(itemTag))
 		n += nn; if err != nil { return n, err }
 		nn, err = EncodeAny(encoder, value, itemTag)
@@ -126,3 +239,154 @@ func encodeAnyMap(encoder *Encoder, value any, tag Tag) (n int, err error) {
 	}
 	return n, nil
 }
+
+// setInt expects a settable destination.
+func setInt(destination reflect.Value, value uint64) error {
+	switch {
+	case destination.CanInt():
+		destination.Set(reflect.ValueOf(int64(value)).Convert(destination.Type()))
+	case destination.CanUint():
+		destination.Set(reflect.ValueOf(value).Convert(destination.Type()))
+	default:
+		return fmt.Errorf("cannot assign integer to %T", destination.Interface())
+	}
+	return nil
+}
+
+// setFloat expects a settable destination.
+func setFloat(destination reflect.Value, value float64) error {
+	if !destination.CanFloat() {
+		return fmt.Errorf("cannot assign float to %T", destination.Interface())
+	}
+	destination.Set(reflect.ValueOf(value).Convert(destination.Type()))
+	return nil
+}
+
+// setByteArrayexpects a settable destination.
+func setByteArray(destination reflect.Value, value []byte) error {
+	typ := destination.Type()
+	if typ.Kind() != reflect.Slice {
+		return fmt.Errorf("cannot assign %T to ", value)
+	}
+	if typ.Elem() != reflect.TypeOf(byte(0)) {
+		return fmt.Errorf("cannot convert %T to *[]byte", value)
+	}
+	destination.Set(reflect.ValueOf(value))
+	return nil
+}
+
+// decodeAndSetInt expects a settable destination.
+func decodeAndSetInt(decoder *Decoder, destination reflect.Value, bytes int) (n int, err error) {
+	value, nn, err := decoder.ReadUintN(bytes)
+	n += nn; if err != nil { return n, err }
+	return n, setInt(destination, value)
+}
+
+// decodeAndSetInt expects a settable destination.
+func decodeAndSetFloat(decoder *Decoder, destination reflect.Value, bytes int) (n int, err error) {
+	switch bytes {
+	case 8:
+		value, nn, err := decoder.ReadFloat64()
+		n += nn; if err != nil { return n, err }
+		return n, setFloat(destination, float64(value))
+	case 4:
+		value, nn, err := decoder.ReadFloat32()
+		n += nn; if err != nil { return n, err }
+		return n, setFloat(destination, float64(value))
+	}
+	return n, fmt.Errorf("cannot decode float%d", bytes * 8)
+}
+
+// skeletonValue returns a pointer value. In order for it to be set, it must be
+// dereferenced using Elem().
+func skeletonValue(decoder *Decoder, tag Tag) (reflect.Value, error) {
+	typ, err := typeOf(decoder, tag)
+	if err != nil { return reflect.Value { }, err }
+	return reflect.New(typ), nil
+}
+
+// typeOf returns the type of the current tag being decoded. It does not use up
+// the decoder, it only peeks.
+func typeOf(decoder *Decoder, tag Tag) (reflect.Type, error) {
+	switch tag.WithoutCN() {
+	case SI:
+		return reflect.TypeOf(uint8(0)), nil
+	case LI:
+		switch tag.CN() {
+		case 0: return reflect.TypeOf(uint8(0)), nil
+		case 1: return reflect.TypeOf(uint16(0)), nil
+		case 3: return reflect.TypeOf(uint32(0)), nil
+		case 7: return reflect.TypeOf(uint64(0)), nil
+		}
+		return nil, fmt.Errorf("unknown CN %d for LI", tag.CN())
+	case FP:
+		switch tag.CN() {
+		case 3: return reflect.TypeOf(float32(0)), nil
+		case 7: return reflect.TypeOf(float64(0)), nil
+		}
+		return nil, fmt.Errorf("unknown CN %d for FP", tag.CN())
+	case SBA: return reflect.SliceOf(reflect.TypeOf(byte(0))), nil
+	case LBA: return reflect.SliceOf(reflect.TypeOf(byte(0))), nil
+	case OTA:
+		elemTag, dimension, err := peekSlice(decoder, tag)
+		if err != nil { return nil, err }
+		if elemTag.Is(OTA) { panic("peekSlice cannot return OTA") }
+		typ, err := typeOf(decoder, elemTag)
+		if err != nil { return nil, err }
+		for _ = range dimension {
+			typ = reflect.SliceOf(typ)
+		}
+		return typ, nil
+	case KTV: return reflect.TypeOf(dummyMap), nil
+	}
+	return nil, fmt.Errorf("unknown TN %d", tag.TN())
+}
+
+// peekSlice returns the element tag and dimension count of the OTA currently
+// being decoded. It does not use up the decoder, it only peeks.
+func peekSlice(decoder *Decoder, tag Tag) (Tag, int, error) {
+	offset    := 0
+	dimension := 0
+	currentTag := tag
+	for {
+		elem, populated, n, err := peekSliceOnce(decoder, currentTag, offset)
+		if err != nil { return 0, 0, err }
+		currentTag = elem
+		offset     = n
+		dimension += 1
+		if elem.Is(OTA) {
+			if !populated {
+				// default to a large byte array, will be
+				// interpreted as a string.
+				return LBA, dimension + 1, nil
+			}
+		} else {
+			return elem, dimension, nil
+		}
+	}
+}
+
+// peekSliceOnce returns the element tag of the OTA located offset bytes ahead
+// of the current position. It does not use up the decoder, it only peeks. The n
+// return value denotes how far away from 0 it peeked. If the OTA has more than
+// zero items, populated will be set to true.
+func peekSliceOnce(decoder *Decoder, tag Tag, offset int) (elem Tag, populated bool, n int, err error) {
+	lengthStart  := offset
+	lengthEnd    := lengthStart + tag.CN() + 1
+	elemTagStart := lengthEnd
+	elemTagEnd   := elemTagStart + 1
+
+	headerBytes, err := decoder.Peek(elemTagEnd)
+	if err != nil { return 0, false, 0, err }
+
+	elem = Tag(headerBytes[len(headerBytes) - 1])
+	for index := lengthStart; index < lengthEnd; index += 1 {
+		if headerBytes[index] > 0 {
+			populated = true
+			break
+		}
+	}
+	n = elemTagEnd
+
+	return
+}

tape/dynamic_test.go

@@ -0,0 +1,195 @@
+package tape
+
+import "fmt"
+import "bytes"
+import "testing"
+import "reflect"
+import tu "git.tebibyte.media/sashakoshka/hopp/internal/testutil"
+
+func TestEncodeAnyInt(test *testing.T) {
+	err := testEncodeAny(test, uint8(0xCA), LI.WithCN(0), tu.S(0xCA))
+	if err != nil { test.Fatal(err) }
+	err = testEncodeAny(test, 400, LI.WithCN(3), tu.S(
+		0, 0, 0x1, 0x90,
+	))
+	if err != nil { test.Fatal(err) }
+}
+
+func TestEncodeAnyTable(test *testing.T) {
+	err := testEncodeAny(test, map[uint16] any {
+		0xF3B9: 1,
+		0x0102: 2,
+		0x0000: "hi!",
+		0xFFFF: []uint16 { 0xBEE5, 0x7777 },
+		0x1234: [][]uint16 { []uint16 { 0x5 }, []uint16 { 0x17, 0xAAAA} },
+	}, KTV.WithCN(0), tu.S(5).AddVar(
+		[]byte {
+			0xF3, 0xB9,
+			byte(LI.WithCN(3)),
+			0, 0, 0, 1,
+		},
+		[]byte {
+			0x01, 0x02,
+			byte(LI.WithCN(3)),
+			0, 0, 0, 2,
+		},
+		[]byte {
+			0, 0,
+			byte(SBA.WithCN(3)),
+			'h', 'i', '!',
+		},
+		[]byte {
+			0xFF, 0xFF,
+			byte(OTA.WithCN(0)), 2, byte(LI.WithCN(1)),
+			0xBE, 0xE5, 0x77, 0x77,
+		},
+		[]byte {
+			0x12, 0x34,
+			byte(OTA.WithCN(0)), 2, byte(OTA.WithCN(0)),
+			1, byte(LI.WithCN(1)),
+			0, 0x5,
+			2, byte(LI.WithCN(1)),
+			0,    0x17,
+			0xAA, 0xAA,
+		},
+	))
+	if err != nil { test.Fatal(err) }
+}
+
+func TestEncodeDecodeAnyTable(test *testing.T) {
+	err := testEncodeDecodeAny(test, map[uint16] any {
+		0xF3B9: uint32(1),
+		0x0102: uint32(2),
+		0x0000: []byte("hi!"),
+		0xFFFF: []uint16 { 0xBEE5, 0x7777 },
+		0x1234: [][]uint16 { []uint16 { 0x5 }, []uint16 { 0x17, 0xAAAA} },
+	}, nil)
+	if err != nil { test.Fatal(err) }
+}
+
+func TestPeekSlice(test *testing.T) {
+	buffer := bytes.NewBuffer([]byte {
+		2, byte(OTA.WithCN(3)),
+		0, 0, 0, 1, byte(LI.WithCN(1)),
+		0, 0x5,
+		2, byte(LI.WithCN(1)),
+		0,    0x17,
+		0xAA, 0xAA,
+	})
+	decoder := NewDecoder(buffer)
+
+	elem, dimension, err := peekSlice(decoder, OTA.WithCN(0))
+	if err != nil { test.Fatal(err) }
+	if elem != LI.WithCN(1) {
+		test.Fatalf("wrong element tag: %v %02X", elem, byte(elem))
+	}
+	if got, correct := dimension, 2; got != correct {
+		test.Fatalf("wrong dimension: %d != %d", got, correct)
+	}
+}
+
+func TestPeekSliceOnce(test *testing.T) {
+	buffer := bytes.NewBuffer([]byte {
+		2, byte(OTA.WithCN(3)),
+		0, 0, 0, 1, byte(LI.WithCN(1)),
+		0, 0x5,
+		2, byte(LI.WithCN(1)),
+		0,    0x17,
+		0xAA, 0xAA,
+	})
+	decoder := NewDecoder(buffer)
+
+	test.Log("--- stage 1")
+	elem, populated, n, err := peekSliceOnce(decoder, OTA.WithCN(0), 0)
+	if err != nil { test.Fatal(err) }
+	if elem != OTA.WithCN(3) {
+		test.Fatal("wrong element tag:", elem)
+	}
+	if !populated {
+		test.Fatal("wrong populated:", populated)
+	}
+	if got, correct := n, 2; got != correct {
+		test.Fatalf("wrong n: %d != %d", got, correct)
+	}
+	
+	test.Log("--- stage 2")
+	elem, populated, n, err = peekSliceOnce(decoder, elem, n)
+	if err != nil { test.Fatal(err) }
+	if elem != LI.WithCN(1) {
+		test.Fatal("wrong element tag:", elem)
+	}
+	if !populated {
+		test.Fatal("wrong populated:", populated)
+	}
+	if got, correct := n, 7; got != correct {
+		test.Fatalf("wrong n: %d != %d", got, correct)
+	}
+}
+
+func encAny(value any) ([]byte, Tag, int, error) {
+	tag, err := TagAny(value)
+	if err != nil { return nil, 0, 0, err }
+	buffer := bytes.Buffer { }
+	encoder := NewEncoder(&buffer)
+	n, err := EncodeAny(encoder, value, tag)
+	if err != nil { return nil, 0, n, err }
+	encoder.Flush()
+	return buffer.Bytes(), tag, n, nil
+}
+
+func decAny(data []byte) (Tag, any, int, error) {
+	destination := map[uint16] any { }
+	tag, err := TagAny(destination)
+	if err != nil { return 0, nil, 0, err }
+	n, err := DecodeAny(NewDecoder(bytes.NewBuffer(data)), &destination, tag)
+	if err != nil { return 0, nil, n, err }
+	return tag, destination, n, nil
+}
+
+func testEncodeAny(test *testing.T, value any, correctTag Tag, correctBytes tu.Snake) error {
+	bytes, tag, n, err := encAny(value)
+	if err != nil { return err }
+	test.Log("n:      ", n)
+	test.Log("tag:    ", tag)
+	test.Log("got:    ", tu.HexBytes(bytes))
+	test.Log("correct:", correctBytes)
+	if tag != correctTag {
+		return fmt.Errorf("tag not equal")
+	}
+	if ok, n := correctBytes.Check(bytes); !ok {
+		return fmt.Errorf("bytes not equal: %d", n)
+	}
+	if n != len(bytes) {
+		return fmt.Errorf("n not equal: %d != %d", n, len(bytes))
+	}
+	return nil
+}
+
+func testEncodeDecodeAny(test *testing.T, value, correctValue any) error {
+	if correctValue == nil { 
+		correctValue = value
+	}
+
+	test.Log("encoding...")
+	bytes, tag, n, err := encAny(value)
+	if err != nil { return err }
+	test.Log("n:  ", n)
+	test.Log("tag:", tag)
+	test.Log("got:", tu.HexBytes(bytes))
+	test.Log("decoding...", tag)
+	if n != len(bytes) {
+		return fmt.Errorf("n not equal: %d != %d", n, len(bytes))
+	}
+	
+	_, decoded, n, err := decAny(bytes)
+	if err != nil { return err }
+	test.Log("got:    ", tu.Describe(decoded))
+	test.Log("correct:", tu.Describe(correctValue))
+	if !reflect.DeepEqual(decoded, correctValue) {
+		return fmt.Errorf("values not equal")
+	}
+	if n != len(bytes) {
+		return fmt.Errorf("n not equal: %d != %d", n, len(bytes))
+	}
+	return nil
+}

tape/encode.go

@@ -2,6 +2,7 @@ package tape
 
 import "io"
 import "math"
+import "bufio"
 
 // Encodable is any type that can write itself to an encoder.
 type Encodable interface {
@@ -10,14 +11,16 @@ type Encodable interface {
 	Encode(encoder *Encoder) (n int, err error)
 }
 	
-// Encoder wraps an [io.Writer] and encodes data to it.
+// Encoder encodes data to an io.Writer.
 type Encoder struct {
-	io.Writer
+	bufio.Writer
 }
 
-// WriteByte encodes a single byte to the output writer.
-func (this *Encoder) WriteByte(value byte) (n int, err error) {
-	return this.WriteByte(uint8(value))
+// NewEncoder creates a new encoder that writes to writer.
+func NewEncoder(writer io.Writer) *Encoder {
+	encoder := &Encoder { }
+	encoder.Reset(writer)
+	return encoder
 }
 
 // WriteInt8 encodes an 8-bit signed integer to the output writer.
@@ -82,7 +85,7 @@ func (this *Encoder) WriteIntN(value int64, bytes int) (n int, err error) {
 	return this.WriteUintN(uint64(value), bytes)
 }
 
-// for below functions, increase buffers if go somehow gets support for over 64
+// for Write/ReadUintN, increase buffers if go somehow gets support for over 64
 // bit integers. we could also make an expanding int type in goutil to use here,
 // or maybe there is one in the stdlib. keep the int64 versions as well though
 // because its ergonomic.
@@ -92,13 +95,18 @@ func (this *Encoder) WriteUintN(value uint64, bytes int) (n int, err error) {
 	// TODO: don't make multiple write calls (without allocating)
 	buffer := [1]byte { }
 	for bytesLeft := bytes; bytesLeft > 0; bytesLeft -- {
-		buffer[0] = byte(buffer[0]) >> ((bytesLeft - 1) * 8)
+		buffer[0] = byte(value) >> ((bytesLeft - 1) * 8)
 		nn, err := this.Write(buffer[:])
 		n += nn; if err != nil { return n, err }
 	}
 	return n, nil
 }
 
+// WriteFloat16 encodes a 16-bit floating point value to the output writer.
+func (this *Encoder) WriteFloat16(value float32) (n int, err error) {
+	return this.WriteUint16(f32bitsToF16bits(math.Float32bits(value)))
+}
+
 // WriteFloat32 encodes a 32-bit floating point value to the output writer.
 func (this *Encoder) WriteFloat32(value float32) (n int, err error) {
 	return this.WriteUint32(math.Float32bits(value))
@@ -113,3 +121,69 @@ func (this *Encoder) WriteFloat64(value float64) (n int, err error) {
 func (this *Encoder) WriteTag(value Tag) (n int, err error) {
 	return this.WriteUint8(uint8(value))
 }
+
+// f32bitsToF16bits returns uint16 (Float16 bits) converted from the specified float32.
+// Conversion rounds to nearest integer with ties to even.
+// Taken from https://github.com/x448/float16/blob/v0.8.4/float16
+//
+// MIT License
+// 
+// Copyright (c) 2019 Montgomery Edwards⁴⁴⁸ and Faye Amacker
+// 
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+// 
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+func f32bitsToF16bits(u32 uint32) uint16 {
+	// Translated from Rust to Go by Montgomery Edwards⁴⁴⁸ (github.com/x448).
+	// All 4294967296 conversions with this were confirmed to be correct by x448.
+	// Original Rust implementation is by Kathryn Long (github.com/starkat99) with MIT license.
+
+	sign := u32 & 0x80000000
+	exp := u32 & 0x7f800000
+	coef := u32 & 0x007fffff
+
+	if exp == 0x7f800000 {
+		// NaN or Infinity
+		nanBit := uint32(0)
+		if coef != 0 {
+			nanBit = uint32(0x0200)
+		}
+		return uint16((sign >> 16) | uint32(0x7c00) | nanBit | (coef >> 13))
+	}
+
+	halfSign := sign >> 16
+
+	unbiasedExp := int32(exp>>23) - 127
+	halfExp := unbiasedExp + 15
+
+	if halfExp >= 0x1f {
+		return uint16(halfSign | uint32(0x7c00))
+	}
+
+	if halfExp <= 0 {
+		if 14-halfExp > 24 {
+			return uint16(halfSign)
+		}
+		coef := coef | uint32(0x00800000)
+		halfCoef := coef >> uint32(14-halfExp)
+		roundBit := uint32(1) << uint32(13-halfExp)
+		if (coef&roundBit) != 0 && (coef&(3*roundBit-1)) != 0 {
+			halfCoef++
+		}
+		return uint16(halfSign | halfCoef)
+	}
+
+	uHalfExp := uint32(halfExp) << 10
+	halfCoef := coef >> 13
+	roundBit := uint32(0x00001000)
+	if (coef&roundBit) != 0 && (coef&(3*roundBit-1)) != 0 {
+		return uint16((halfSign | uHalfExp | halfCoef) + 1)
+	}
+	return uint16(halfSign | uHalfExp | halfCoef)
+}

tape/tag.go

@@ -1,16 +1,18 @@
 package tape
 
+import "fmt"
+
 type Tag byte; const (
-	SI      Tag = 0 << 5  // Small integer
-	LI      Tag = 1 << 5  // Large integer
-	FP      Tag = 2 << 5  // Floating point
-	SBA     Tag = 3 << 5  // Small byte array
-	LBA     Tag = 4 << 5  // Large byte array
-	OTA     Tag = 5 << 5  // One-tag array
-	KTV     Tag = 6 << 5  // Key-tag-value table
-	TNMask  Tag = 0xE0    // The entire TN bitfield
-	CNMask  Tag = 0x20    // The entire CN bitfield
-	CNLimit Tag = 32      // All valid CNs are < CNLimit
+	SI      Tag = 0 << 5 // Small integer
+	LI      Tag = 1 << 5 // Large integer
+	FP      Tag = 2 << 5 // Floating point
+	SBA     Tag = 3 << 5 // Small byte array
+	LBA     Tag = 4 << 5 // Large byte array
+	OTA     Tag = 5 << 5 // One-tag array
+	KTV     Tag = 6 << 5 // Key-tag-value table
+	TNMask  Tag = 0xE0   // The entire TN bitfield
+	CNMask  Tag = 0x1F   // The entire CN bitfield
+	CNLimit Tag = 32     // All valid CNs are < CNLimit
 )
 
 func (tag Tag) TN() int {
@@ -33,11 +35,30 @@ func (tag Tag) Is(other Tag) bool {
 	return tag.TN() == other.TN()
 }
 
+func (tag Tag) String() string {
+	tn := fmt.Sprint(tag.TN())
+	switch tag.WithoutCN() {
+	case SI:  tn = "SI"
+	case LI:  tn = "LI"
+	case FP:  tn = "FP"
+	case SBA: tn = "SBA"
+	case LBA: tn = "LBA"
+	case OTA: tn = "OTA"
+	case KTV: tn = "KTV"
+	}
+	return fmt.Sprintf("%s:%d", tn, tag.CN())
+}
+
 // BufferTag returns the appropriate tag for a buffer.
 func BufferTag(value []byte) Tag {
 	return bufferLenTag(len(value))
 }
 
+// StringTag returns the appropriate tag for a string.
+func StringTag(value string) Tag {
+	return bufferLenTag(len(value))
+}
+
 func bufferLenTag(length int) Tag {
 	if length < int(CNLimit) {
 		return SBA.WithCN(length)