design/branched-generated-encoder.md

@@ -1,128 +0,0 @@
-# Branched Generated Decoder
-
-Pasted here because Tebitea is down
-
-## The problem
-
-TAPE is designed so that the decoder can gloss over data it does not understand.
-Technically the protocol allows for this, but I completely forgot to implement
-this in the generated decoder, oops. This would be trivial if TAPE messages were
-still flat tables, but they aren't, because those aren't useful enough. So,
-let's analyze the problem.
-
-## When it happens
-
-There are two reasons something might not match up with the expected data:
-
-The first and most obvious is unrecognized keys. If the key is not in the set of
-recognized keys for a KTV, it should leave the corresponding struct field blank.
-Once #6 has been implemented, throw an error if the data was not optional.
-
-The second is wrong types. If we are expecting KTV and get SBA, we should leave
-the data as empty. The aforementioned concern about #6 also applies here. We
-don't need to worry about special cases at the structure root, because it would
-be technically possible to make the structure root an option, so it really is
-just a normal value. Until #6, we will leave that blank too.
-
-## Preliminary ideas
-
-The first is going to be pretty simple. All we need to do is have a skimmer
-function that skims over TAPE data very, and then call that on the KTV value
-each time we run into a mystery key. It should only return an error if the
-structure of the data is malformed in such a way that it cannot continue to the
-next one. This should be stored in the tape package alongside the dynamic
-decoding functions, because they will essentially function the same way and
-could probably share lots of code.
-
-The second is a bit more complicated because of the existence of KTV and OTA
-because they are aggregate types. Go types work a bit differently, as if you
-have an array of an array of an array of ints, that information is represented
-in one place, whereas TAPE doesn't really do that. All of that information is
-sort of buried within the data structure, so we don't know what we will be
-decoding before we actually do it. Whenever we encounter a type we don't expect,
-we would need to abort decoding of the entire data structure, and then skim over
-whatever detritus is left, which would literally be in a half-decoded state. The
-fact that the code is generated flat and thus cannot use return or defer
-statements contributes to the complexity of this problem. We need to go up, but
-we can't. There is no up, only forward.
-
-Of course, the dynamic decoder does not have this problem in the first place
-because it doesn't expect anything, and constructs the destination to fit
-whatever it sees in the TAPE structure as it is decoding it. KTVs are completely
-dynamic because they are implemented as maps, so the only time it needs to
-completely comprehend a type is with OTAs. There is a function called typeOf
-that gets the type of the current tag and returns it as a reflect.Type, which
-necessitates recursion and peeking at OTAs and their elements.
-
-We could try to do the same thing in the generated decoder, comparing the
-determined type against the expected type to try to figure out whether we should
-decode an array or a table, etc. This is immediately problematic as it requires
-memory to be allocated, both for the peek buffer and the resulting tree of type
-information. If we end up with some crazy way to keep track of the types, that's
-only one half of the allocation problem and we would still be spending extra
-cycles going over all of that twice.
-
-## Performance constraints
-
-The generated decoder is supposed to blaze through data, and it can't do that if
-it does all the singing and dancing that the dynamic decoder does. It's time for
-some performance constraints:
-
-- No allocations, except as required to build the destination for the data
-- No redundant work
-- So, no freaking peeking
-- It should take well under 500 lines of generated code to decode one message of
-reasonable size (i.e. be careful not to bloat the binary)
-
-I'm not really going to do my usual thing here of making a slow version and
-speeding it up over time based on evidence and experimentation because these
-constraints inform the design so much it would be impossible to continue without
-them. I am 99% confident that these constraints will allow for an acceptable
-baseline of performance (for generated code) and we can still profile and
-micro-optimize later. This is good enough for me.
-Heavy solution
-
-There is a solution that might work very well which involves completely redoing
-the generated decoding code. We could create a function for every source type to
-destination type mapping that exists in protocol, and then compose them all
-together. The decoding methods for each message or type would be wrappers around
-the correct function for their root TAPE -> Go type mapping. The main benefit of
-this is it would make this problem a lot more manageable because the interface
-points between the data would be represented by function boundaries. This would
-allow the use of return and defer statements, and would allow more code sharing,
-producing a smaller binary. Go would probably inline these where needed.
-
-Would this work? Probably. More investigation is required to make sure. I want
-to stop re-writing things I don't need to. On the other hand, it is just the
-decoder.
-
-## Light solution
-
-TODO: find a solution that satisfies the performance constraints, keeps the same
-identical interface, and works off the same code. I am convinced this is doable,
-and it might even allow us to extract more data from an unexpected structure.
-However, continuing this way might introduce unmanageable complexity. It is
-already a little unmanageable and I am just one pony (kind of).
-
-## Implementation
-
-Heavy solution is going to work here, applied to only the points of
-`Generator.generateDecodeValue` where it decodes an aggregate data structure.
-That way, only minimal amounts of code need to be redone.
-
-Whenever a branch needs to happen, a call shall be generated, a deferred
-implementation request shall be added to a special FIFO queue within the
-generator. After generating data structures and their root decoding functions,
-the generator shall pick away at this queue until no requests remain. The
-generator shall accept new items during this process, so that recursion is
-possible. This is all to ensure it is only ever writing one function at a time
-
-The functions shall take a pointer to a type that accepts any type like (~) the
-destination's base type. We should also probably just call
-`Generator.generateDecodeValue` directly on user defined types this way, keeping
-their public `Decode` methods just for convenience.
-
-The tape package shall contain a skimming function that takes a decoder and a
-tag, and recursively consumes the decoder given the context of the tag. This
-shall be utilized by the decoder functions to skip over values if their tags
-or keys do not match up with what is expected.

generate/generate.go

@@ -6,7 +6,6 @@ import "maps"
 import "math"
 import "slices"
 import "strings"
-import "encoding/hex"
 import "git.tebibyte.media/sashakoshka/hopp/tape"
 
 const imports =
@@ -47,13 +46,6 @@ type Generator struct {
 	nestingLevel int
 	temporaryVar int
 	protocol     *Protocol
-
-	decodeBranchRequestQueue []decodeBranchRequest
-}
-
-type decodeBranchRequest struct {
-	hash [16]byte
-	typ  Type
 }
 
 func (this *Generator) Generate(protocol *Protocol) (n int, err error) {
@@ -87,14 +79,6 @@ func (this *Generator) Generate(protocol *Protocol) (n int, err error) {
 		n += nn; if err != nil { return n, err }
 	}
 
-	// request queue
-	for {
-		hash, typ, ok := this.pullDecodeBranchRequest()
-		if !ok { break }
-		nn, err := this.generateDecodeBranch(hash, typ)
-		n += nn; if err != nil { return n, err }
-	}
-
 	return n, nil
 }
 
@@ -162,7 +146,7 @@ func (this *Generator) generateTypedef(name string, typ Type) (n int, err error)
 	this.push()
 		nn, err = this.iprintf("var nn int\n")
 		n += nn; if err != nil { return n, err }
-		nn, err = this.generateDecodeValue(typ, "this", "tag")
+		nn, err = this.generateDecodeValue(typ, "this", "tag", "return n, nil")
 		n += nn; if err != nil { return n, err }
 		nn, err = this.iprintf("return n, nil\n")
 		n += nn; if err != nil { return n, err }
@@ -233,16 +217,20 @@ func (this *Generator) generateMessage(method uint16, message Message) (n int, e
 		n += nn; if err != nil { return n, err }
 		nn, err = this.generateErrorCheck()
 		n += nn; if err != nil { return n, err }
+		abort := "return n, nil" // TODO: skip value somehow
 		nn, err = this.iprintf("if !tag.Is(")
 		n += nn; if err != nil { return n, err }
 		nn, err = this.generateTN(message.Type)
 		n += nn; if err != nil { return n, err }
 		nn, err = this.printf(") {\n")
 		n += nn; if err != nil { return n, err }
-			// TODO skip value using the correct TAPE function and return
+		this.push()
+			nn, err = this.iprintf("%s\n", abort)
+			n += nn; if err != nil { return n, err }
+		this.pop()
 		nn, err = this.iprintf("}\n")
 		n += nn; if err != nil { return n, err }
-		nn, err = this.generateDecodeValue(message.Type, "this", "tag")
+		nn, err = this.generateDecodeValue(message.Type, "this", "tag", abort)
 		n += nn; if err != nil { return n, err }
 		nn, err = this.iprintf("return n, nil\n")
 		n += nn; if err != nil { return n, err }
@@ -431,7 +419,7 @@ func (this *Generator) generateEncodeValue(typ Type, valueSource, tagSource stri
 //   - n int
 //   - err error
 //   - nn int
-func (this *Generator) generateDecodeValue(typ Type, valueSource, tagSource string) (n int, err error) {
+func (this *Generator) generateDecodeValue(typ Type, valueSource, tagSource, abort string) (n int, err error) {
 	switch typ := typ.(type) {
 	case TypeInt:
 		// SI: (none)
@@ -497,7 +485,49 @@ func (this *Generator) generateDecodeValue(typ Type, valueSource, tagSource stri
 		}
 	case TypeArray:
 		// OTA: <length: UN> <elementTag: tape.Tag> <values>*
-		nn, err := this.generateDecodeBranchCall(typ, valueSource, tagSource)
+		lengthVar := this.newTemporaryVar("length")
+		nn, err := this.iprintf("var %s uint64\n", lengthVar)
+		n += nn; if err != nil { return n, err }
+		nn, err = this.iprintf(
+			"%s, nn, err = decoder.ReadUintN(int(%s.CN()))\n",
+			lengthVar, tagSource)
+		n += nn; if err != nil { return n, err }
+		nn, err = this.generateErrorCheck()
+		n += nn; if err != nil { return n, err }
+		nn, err = this.iprintf("*%s = make(", valueSource)
+		n += nn; if err != nil { return n, err }
+		nn, err = this.generateType(typ)
+		n += nn; if err != nil { return n, err }
+		nn, err = this.printf(", int(%s))\n", lengthVar)
+		n += nn; if err != nil { return n, err }
+		nn, err = this.iprintf("var itemTag tape.Tag\n")
+		n += nn; if err != nil { return n, err }
+		nn, err = this.iprintf("itemTag, nn, err = decoder.ReadTag()\n")
+		n += nn; if err != nil { return n, err }
+		nn, err = this.generateErrorCheck()
+		n += nn; if err != nil { return n, err }
+		nn, err = this.iprintf("if !itemTag.Is(")
+		n += nn; if err != nil { return n, err }
+		nn, err = this.generateTN(typ.Element)
+		n += nn; if err != nil { return n, err }
+		nn, err = this.iprintf(") {\n")
+		n += nn; if err != nil { return n, err }
+		this.push()
+			nn, err = this.iprintf("%s\n", abort)
+			n += nn; if err != nil { return n, err }
+		this.pop()
+		nn, err = this.iprintf("}\n")
+		n += nn; if err != nil { return n, err }
+		nn, err = this.iprintf("for index := range %s {\n", lengthVar)
+		n += nn; if err != nil { return n, err }
+		this.push()
+			nn, err = this.generateDecodeValue(
+				typ.Element,
+				fmt.Sprintf("(&(*%s)[index])", valueSource),
+				"itemTag", abort)
+			n += nn; if err != nil { return n, err }
+		this.pop()
+		nn, err = this.iprintf("}\n")
 		n += nn; if err != nil { return n, err }
 	case TypeTable:
 		// KTV: <length: UN> (<key: U16> <tag: Tag> <value>)*
@@ -509,7 +539,72 @@ func (this *Generator) generateDecodeValue(typ Type, valueSource, tagSource stri
 		n += nn; if err != nil { return n, err }
 	case TypeTableDefined:
 		// KTV: <length: UN> (<key: U16> <tag: Tag> <value>)*
-		nn, err := this.generateDecodeBranchCall(typ, valueSource, tagSource)
+		lengthVar := this.newTemporaryVar("length")
+		nn, err := this.iprintf("var %s uint64\n", lengthVar)
+		n += nn; if err != nil { return n, err }
+		nn, err = this.iprintf(
+			"%s, nn, err = decoder.ReadUintN(int(%s.CN()))\n",
+			lengthVar, tagSource)
+		n += nn; if err != nil { return n, err }
+		nn, err = this.generateErrorCheck()
+		n += nn; if err != nil { return n, err }
+		nn, err = this.iprintf("for _ = range %s {\n", lengthVar)
+		n += nn; if err != nil { return n, err }
+		this.push()
+			nn, err = this.iprintf("var key uint16\n")
+			n += nn; if err != nil { return n, err }
+			nn, err = this.iprintf("key, nn, err = decoder.ReadUint16()\n")
+			n += nn; if err != nil { return n, err }
+			nn, err = this.generateErrorCheck()
+			n += nn; if err != nil { return n, err }
+			nn, err = this.iprintf("var itemTag tape.Tag\n")
+			n += nn; if err != nil { return n, err }
+			nn, err = this.iprintf("itemTag, nn, err = decoder.ReadTag()\n")
+			n += nn; if err != nil { return n, err }
+			nn, err = this.generateErrorCheck()
+			n += nn; if err != nil { return n, err }
+			nn, err = this.iprintf("switch key {\n")
+			n += nn; if err != nil { return n, err }
+			keys := slices.Collect(maps.Keys(typ.Fields))
+			slices.Sort(keys)
+			for _, key := range keys {
+				field := typ.Fields[key]
+				nn, err = this.iprintf("case 0x%04X:\n", key)
+				n += nn; if err != nil { return n, err }
+				this.push()
+					labelVar := this.newTemporaryVar("label")
+					fieldAbort := fmt.Sprintf("goto %s", labelVar) // TODO: skip value somehow
+					nn, err = this.iprintf("if !itemTag.Is(")
+					n += nn; if err != nil { return n, err }
+					nn, err = this.generateTN(field.Type)
+					n += nn; if err != nil { return n, err }
+					nn, err = this.printf(") {\n")
+					n += nn; if err != nil { return n, err }
+					this.push()
+						nn, err = this.iprintf("%s\n", fieldAbort)
+						n += nn; if err != nil { return n, err }
+					this.pop()
+					nn, err = this.iprintf("}\n")
+					n += nn; if err != nil { return n, err }
+					nn, err = this.iprintf("{\n")
+					n += nn; if err != nil { return n, err }
+					this.push()
+						nn, err = this.generateDecodeValue(
+							field.Type,
+							fmt.Sprintf("(&%s.%s)", valueSource, field.Name),
+							"itemTag", fieldAbort)
+						n += nn; if err != nil { return n, err }
+					this.pop()
+					nn, err = this.iprintf("}\n")
+					n += nn; if err != nil { return n, err }
+					nn, err = this.iprintf("%s:;\n", labelVar)
+					n += nn; if err != nil { return n, err }
+				this.pop()
+			}
+			nn, err = this.iprintf("}\n")
+			n += nn; if err != nil { return n, err }
+		this.pop()
+		nn, err = this.iprintf("}\n")
 		n += nn; if err != nil { return n, err }
 	case TypeNamed:
 		// WHATEVER: [WHATEVER]
@@ -524,151 +619,6 @@ func (this *Generator) generateDecodeValue(typ Type, valueSource, tagSource stri
 	return n, nil
 }
 
-// generateDecodeBranchCall generates code to call an aggregate decoder function,
-// for a specified type. The definition of the function is deferred so no
-// duplicates are created. The function overwrites memory pointed to by the
-// variable (or parenthetical statement) specified by valueSource, and the value
-// will be encoded according to the tag stored in the variable (or parenthetical
-// statement) specified by tagSource. the code generated is a BLOCK and expects
-// these variables to be defined:
-//
-//   - decoder *tape.Decoder
-//   - n int
-//   - err error
-//   - nn int
-func (this *Generator) generateDecodeBranchCall(typ Type, valueSource, tagSource string) (n int, err error) {
-	hash := HashType(typ)
-	nn, err := this.iprintf("nn, err = %s(%s, %s)\n", this.decodeBranchName(hash), valueSource, tagSource)
-	n += nn; if err != nil { return n, err }
-	nn, err = this.generateErrorCheck()
-	n += nn; if err != nil { return n, err }
-	this.pushDecodeBranchRequest(hash, typ)
-	return n, nil
-}
-
-// generateDecodeBranch generates an aggregate decoder function definition for a
-// specified type. It assumes that hash == HashType(typ).
-func (this *Generator) generateDecodeBranch(hash [16]byte, typ Type) (n int, err error) {
-	nn, err := this.iprintf("func %s(this *", this.decodeBranchName(hash))
-	n += nn; if err != nil { return n, err }
-	nn, err = this.generateType(typ)
-	n += nn; if err != nil { return n, err }
-	nn, err = this.printf(" decoder *tape.Decoder, tag tape.Tag) (nn int, err error) \n{")
-	n += nn; if err != nil { return n, err }
-	this.push()
-	
-		nn, err = this.iprintf("var nn int\n")
-		n += nn; if err != nil { return n, err }
-		
-		switch typ := typ.(type) {
-		case TypeArray:
-			// OTA: <length: UN> <elementTag: tape.Tag> <values>*
-			// read header
-			lengthVar := this.newTemporaryVar("length")
-			nn, err := this.iprintf("var %s uint64\n", lengthVar)
-			n += nn; if err != nil { return n, err }
-			nn, err = this.iprintf("%s, nn, err = decoder.ReadUintN(int(tag.CN()))\n", lengthVar)
-			n += nn; if err != nil { return n, err }
-			nn, err = this.generateErrorCheck()
-			n += nn; if err != nil { return n, err }
-			elementTagVar := this.newTemporaryVar("elementTag")
-			nn, err = this.iprintf("var %s uint64\n", lengthVar)
-			n += nn; if err != nil { return n, err }
-			nn, err = this.iprintf("%s, nn, err = decoder.ReadTag()\n", elementTagVar)
-			n += nn; if err != nil { return n, err }
-			nn, err = this.generateErrorCheck()
-			n += nn; if err != nil { return n, err }
-
-			// abort macro
-			abort := func() (n int, err error) {
-				// skim entire array
-				nn, err = this.iprintf("for _ = range %s {\n", lengthVar)
-				n += nn; if err != nil { return n, err }
-				this.push()
-					nn, err = this.iprintf("nn, err = tape.Skim(decoder, tag)")
-					n += nn; if err != nil { return n, err }
-					nn, err = this.generateErrorCheck()
-					n += nn; if err != nil { return n, err }
-				this.pop()
-				nn, err = this.iprintf("}\n")
-				n += nn; if err != nil { return n, err }
-				nn, err = this.iprintf("return n, nil")
-				n += nn; if err != nil { return n, err }
-				return n, nil
-			}
-
-			// validate header
-			// TODO: here, validate that length is less than the
-			// max, whatever that is configured to be. the reason we
-			// want to read it here is that we would have to skip
-			// the tag anyway so why not.
-			nn, err = this.iprintf("if %s != ", elementTagVar)
-			n += nn; if err != nil { return n, err }
-			nn, err = this.generateTag(typ.Element, "(*this)")
-			n += nn; if err != nil { return n, err }
-			nn, err = this.printf(" {\n")
-			n += nn; if err != nil { return n, err }
-			this.push()
-				nn, err = abort()
-				n += nn; if err != nil { return n, err }
-			this.pop()
-			nn, err = this.iprintf("}\n")
-			n += nn; if err != nil { return n, err }
-
-			// decode payloads
-			nn, err = this.iprintf("*this = make(")
-			n += nn; if err != nil { return n, err }
-			nn, err = this.generateType(typ)
-			n += nn; if err != nil { return n, err }
-			nn, err = this.printf(", %s)\n", lengthVar)
-			n += nn; if err != nil { return n, err }
-			nn, err = this.iprintf("for index := range %s {\n", lengthVar)
-			n += nn; if err != nil { return n, err }
-			this.push()
-				nn, err = this.generateDecodeValue(typ.Element, "(*this)[index]", elementTagVar)
-				n += nn; if err != nil { return n, err }
-			this.pop()
-			nn, err = this.iprintf("}\n")
-			n += nn; if err != nil { return n, err }
-		case TypeTableDefined:
-			// KTV: <length: UN> (<key: U16> <tag: Tag> <value>)*
-			// TODO
-		default: return n, fmt.Errorf("unexpected type: %T", typ)
-		}
-
-	this.pop()
-	nn, err = this.iprintf("}")
-	n += nn; if err != nil { return n, err }
-	return n, nil
-}
-
-func (this *Generator) decodeBranchName(hash [16]byte) string {
-	return fmt.Sprintf("decodeBranch_%s", hex.EncodeToString(hash[:]))
-}
-
-// pushDecodeBranchRequest pushes a new branch decode function request to the
-// back of the queue, if it is not already in the queue.
-func (this *Generator) pushDecodeBranchRequest(hash [16]byte, typ Type) {
-	for _, item := range this.decodeBranchRequestQueue {
-		if item.hash == hash { return }
-	}
-	this.decodeBranchRequestQueue = append(this.decodeBranchRequestQueue, decodeBranchRequest {
-		hash: hash,
-		typ:  typ,
-	})
-}
-
-// pullDecodeBranchRequest pulls a branch decode function request from the front
-// of the queue.
-func (this *Generator) pullDecodeBranchRequest() (hash [16]byte, typ Type, ok bool) {
-	if len(this.decodeBranchRequestQueue) < 1 {
-		return [16]byte { }, nil, false
-	}
-	request := this.decodeBranchRequestQueue[0]
-	this.decodeBranchRequestQueue = this.decodeBranchRequestQueue[1:]
-	return request.hash, request.typ, true
-}
-
 func (this *Generator) generateErrorCheck() (n int, err error) {
 	return this.iprintf("n += nn; if err != nil { return n, err }\n")
 }

generate/protocol.go

@@ -1,10 +1,5 @@
 package generate
 
-import "fmt"
-import "maps"
-import "slices"
-import "crypto/md5"
-
 type Protocol struct {
 	Messages map[uint16] Message
 	Types    map[string] Type
@@ -16,7 +11,7 @@ type Message struct {
 }
 
 type Type interface {
-	fmt.Stringer
+	
 }
 
 type TypeInt struct {
@@ -24,84 +19,29 @@ type TypeInt struct {
 	Signed bool
 }
 
-func (typ TypeInt) String() string {
-	output := ""
-	if typ.Signed {
-		output += "I"
-	} else {
-		output += "U"
-	}
-	output += fmt.Sprint(typ.Bits)
-	return output
-}
-
 type TypeFloat struct {
 	Bits int
 }
 
-func (typ TypeFloat) String() string {
-	return fmt.Sprintf("F%d", typ.Bits)
-}
-
 type TypeString struct { }
 
-func (TypeString) String() string {
-	return "String"
-}
-
 type TypeBuffer struct { }
 
-func (TypeBuffer) String() string {
-	return "Buffer"
-}
-
 type TypeArray struct {
 	Element Type
 }
 
-func (typ TypeArray) String() string {
-	return fmt.Sprintf("[]%v", typ.Element)
-}
-
 type TypeTable struct { }
 
-func (TypeTable) String() string {
-	return "Table"
-}
-
 type TypeTableDefined struct {
 	Fields map[uint16] Field
 }
 
-func (typ TypeTableDefined) String() string {
-	output := "{"
-	for _, key := range slices.Sorted(maps.Keys(typ.Fields)) {
-		output += fmt.Sprintf("%04X %v", key, typ.Fields[key])
-	}
-	output += "}"
-	return output
-}
-
 type Field struct {
 	Name string
 	Type Type
 }
 
-func (field Field) String() string {
-	return fmt.Sprintf("%s %v", field.Name, field.Type)
-}
-
 type TypeNamed struct {
 	Name string
 }
-
-func (typ TypeNamed) String() string {
-	return typ.Name
-}
-
-func HashType(typ Type) [16]byte {
-	// TODO: if we ever want to make the compiler more efficient, this would
-	// be a good place to start, complex string concatenation in a hot path
-	// (sorta)
-	return md5.Sum([]byte(typ.String()))
-}