Update design doc, and make AUTOMATIC_IDEMPOTENCY visible

design/idempotency_ids.md

@@ -6,12 +6,22 @@ As an intermediate goal, I plan to introduce this feature disabled by default. T
 
 # API
 
-Introduce a new transaction option `IDEMPOTENCY_ID`, which will be validated to be at most 255 bytes.
+Introduce a new transaction option `AUTOMATIC_IDEMPOTENCY`, which sets the transaction's idempotency id to 16 random bytes if there's no idempotency id yet. Setting this also instructs the fdb client to manage the full lifecycle of the idempotency id (expiring the id automatically after a commit acknowledgement).
+
+Introduce a new transaction option `IDEMPOTENCY_ID`, which will be validated to be at most 255 bytes. This option sets the idempotency id for the transaction, and also instructs the fdb client _not to_ expire this id automatically. The user is responsible for calling `fdb_database_expire_idempotency_id` to expire it.
+
 Add 
 ```
-FDBFuture* fdb_transaction_commit_result(FDBTransaction* tr, uint8_t const* idempotency_id, int idempotency_id_length)
+FDBFuture* fdb_database_commit_result(FDBDatabase* db, uint8_t const* idempotency_id, int idempotency_id_length, int64_t read_snapshot)
 ```
-, which can be used to determine the result of a commit that failed with `transaction_timed_out`.
+, which can be used to determine the result of a commit with the given read snapshot and idempotency id.
+
+Add
+```
+void fdb_database_expire_idempotency_id(FDBDatabase* db, uint8_t const* idempotency_id, int idempotency_id_length)
+```
+
+This lets the client know that the caller is done with this idempotency id and that the cluster may purge it.
 
 Commits for transactions with idempotency ids would not fail with `commit_unknown_result`, but in (extremely) rare cases could fail with a new error that clients are expected to handle by restarting the process.
 # Background
@@ -49,10 +59,9 @@ in that id and the cluster can reclaim the space used to store the idempotency
 id. The commit proxy that committed a batch is responsible for cleaning all
 idempotency kv pairs from that batch, so clients must tell that specific proxy
 that they're done with the id. The first proxy will also periodically clean up
-the oldest idempotency ids, based on a policy determined by two knobs.  One knob
+the oldest idempotency ids, based on a policy determined by knobs.  One knob
 will control the minimum lifetime of an idempotency id (i.e. don't delete
-anything younger than 1 day), and the other will control the target byte size of
-the idempotency keys (e.g. keep 100 MB of idempotency keys around).
+anything younger than 1 day). More knobs may be considered in the future.
 
 # Commit protocol
 
@@ -76,13 +85,29 @@ If a transaction learns that it has been in-flight so long that its idempotency
 
 # Considerations
 
-- Additional storage space on the cluster. This can be controlled directly via an idempotency id target bytes knob/config.
+- Additional storage space on the cluster.
 - Potential write hot spot.
 
 # Multi-version client
 
 The multi-version client will generate its own idempotency id for a transaction and manage its lifecycle. It will duplicate the logic in NativeApi to achieve the same guarantees. As part of this change we will also ensure that the previous commit attempt is no longer in-flight before allowing the commit future to become ready. This will fix a potential "causal-write-risky" issue if a commit attempt fails with `cluster_version_changed`.
 
+# Implementation
+
+```
+FDBFuture* fdb_database_commit_result(FDBDatabase* db, uint8_t const* idempotency_id, int idempotency_id_length, int64_t read_snapshot)
+```
+
+The implementation would first commit a transaction that conflicts with any transaction with the same idempotency id (recall that the idempotency id is added to the conflict ranges), to ensure that none are in flight.
+
+The implementation would then search storage servers for the idempotency id to determine the commit result. The range of keys that need to be read is bounded by the range of possible commit versions for a transaction with `read_snapshot`.
+
+```
+void fdb_database_expire_idempotency_id(FDBDatabase* db, uint8_t const* idempotency_id, int idempotency_id_length)
+```
+
+The fdb client would need to keep track of which commit proxies are responsible for recently committed transactions with idempotency ids, so it knows where to direct the expire requests. Proactively expiring idempotency ids is done on a best-effort basis, but this needs to work well enough that the storage used by the cluster for idempotency ids is acceptable.
+
 # Experiments
 
 - Initial experiments show that this is about 1% overhead for the worst case workload which is transactions that only update a single key.

fdbclient/vexillographer/fdb.options

@@ -290,8 +290,7 @@ description is not currently required but encouraged.
             description="Associate this transaction with this ID for the purpose of checking whether or not this transaction has already committed. Must be at least 16 bytes and less than 256 bytes. This feature is in development and not ready for general use. Unless the automatic_idempotency option is set after this option, the client will not automatically attempt to remove this id from the cluster after a successful commit."
             hidden="true" />
     <Option name="automatic_idempotency" code="505"
-            description="Automatically assign a random 16 byte idempotency id for this transaction. Prevents commits from failing with ``commit_unknown_result``. WARNING: If you are also using the multiversion client or transaction timeouts, if either cluster_version_changed or transaction_timed_out was thrown during a commit, then that commit may have already succeeded or may succeed in the future. This feature is in development and not ready for general use."
-            hidden="true" />
+            description="Automatically assign a random 16 byte idempotency id for this transaction. Prevents commits from failing with ``commit_unknown_result``. WARNING: If you are also using the multiversion client or transaction timeouts, if either cluster_version_changed or transaction_timed_out was thrown during a commit, then that commit may have already succeeded or may succeed in the future. This feature is in development and not ready for general use." />
     <Option name="snapshot_ryw_enable" code="600"
             description="Snapshot read operations will see the results of writes done in the same transaction. This is the default behavior." />
     <Option name="snapshot_ryw_disable" code="601"