Update documentation

flow/README.md

@@ -297,7 +297,21 @@ you are holding the corresponding future.
 
 ### Flatbuffers/ObjectSerializer
 
-1. Motivation and Goals
+1. Introduction
+
+    The rough goal is to be able to introduce a field to a network message
+    without requiring a protocol-incompatible upgrade. In order for this to work,
+    correctness must not depend on that field always being present. This can be
+    tested in simulation by randomly (use buggify) default-initializing that
+    field when deserializing. Once you make a protocol-incompatible upgrade you
+    can rely on the field always being present in the new protocol, just like
+    before. Currently we are using a custom flatbuffers implementation so to that
+    we can present (roughly) the same serialization api as before. Currently the
+    ObjectSerializer is only used for network messages, but that may change.
+    Flatbuffers was selected because it is (relatively) simple among protocols
+    providing forwards/backwards compatibility, and its binary format is [well
+    documented](https://github.com/dvidelabs/flatcc/blob/master/doc/binary-format.md)
+
 1. Correspondence to flatbuffers IDL
     - Tables
     ```
@@ -319,7 +333,7 @@ you are holding the corresponding future.
     - Unions
     ```
     // Flow type
-    using T = std::variant<A, B, C>;
+    using T = boost::variant<A, B, C>;
 
     // IDL equivalent
     union T { A, B, C}
@@ -341,18 +355,52 @@ you are holding the corresponding future.
     [T]
     ```
 
-TODO finish documenting/implementing the following.
-1. Vtables collected from default-constructed instances
-1. Requirements (serialize must be cheap for a default-constructed instance, must have a serialize method or implement a trait.)
-1. Traits/Concepts: vector_like, union_like, dynamic_size, scalar
-1. isDeserializing idiom
-1. Gotchas (serialize gets called more than once on save path, maybe more)
+1. Flatbuffers Traits
+
+    In order to serialize a type as a flatbuffers vector, struct, or union, you can implement the appropriate trait for your type.
+    - `scalar_traits` corresponds to a flatbuffers struct. See `UID` for an example.
+    - `vector_like_traits` corresponds to a flatbuffers vector. See `VectorRef` for an example.
+    - `dynamic_size_traits` corresponds to a flatbuffers vector of uint8_t. See `StringRef` for an example.
+    - `union_like_traits` corresponds to a flatbuffers union. See `boost::variant` for an example.
+
+1. Potential Gotchas
+    - Flatbuffers 'vtables' are collected from default-constructed instances of
+      each type. Consequently types serialized by flatbuffers should have cheap
+      default constructors. Future work: we may be able to collect vtables
+      without an instance of a type using `declval`.
+
+    - `T::serialize` may get called multiple times when serializing `T`. It is
+      guaranteed to be called only once for deserialization though, and thus
+      the `Ar::isDeserializing` idiom is appropriate. Future work: in theory we
+      don't need to call `T::serialize` multiple times when serializing, but
+      this would complicate the implementation.
+
 1. File identifiers
+
+    [File identifiers](https://google.github.io/flatbuffers/md__schemas.html)
+    are used to sanity check that the message you're deserializing is of the
+    schema you expect. You can give a type `T` a file identifier by making
+    `T::file_identifier` a static member of type `FileIdentifier`. You don't
+    need to change the file identifier for a type when evolving its schema.
+
 1. Schema evolution
-1. Testing plan: have buggify sometimes default initialize fields that are introduced without changing the protocol version.
-1. (Future work) Allow ObjectSerializer to take the usual version specifications, `IncludeVersion`, `AssumeVersion`, or `Unversioned`.
-1. (Future work) Smaller messages for deprecated fields
-1. (Future work) `Deprecated<...>` template that knows whether or not the field was present? Automatically buggifies the field being absent?
+
+    Two schemas are forward/backward compatible if they meet the following
+    requirements. (Future work) Any fields that are not common to both schemas should be
+    default-initialized in deserialized messages. Currently they will be
+    uninitialized if their default constructor doesn't initialize.
+
+    - Two tables are compatible if one table's fields are all compatible with a prefix of the other table's fields.
+    - Two vectors are compatible if their element types are compatible.
+    - Two unions are compatible if one unions's fields are all compatible with a prefix of the other unions's fields.
+    - Two scalar types are only compatible if they are equal.
+
+1. Deprecation
+
+    Flatbuffers allows fields to be deprecated, and a deprecated field consumes
+    only two bytes on the wire. (Future work) Introduce `Deprecated<...>`
+    template or something similar so that we can write smaller messages for
+    deprecated fields.
 
 ### ACTOR return values