Encoding methods used:
- Tag localities: Run length encoding
- Tag ids: Compact representation
- Commit versions: delta encoding.
If "n" is the number of entries in the version vector, with the tags
spread over "m" data centers, these techniques will reduce the number
of bytes to represent the version vector from "(11 * n)" bytes to
"(3 * m + 2 * n)" / "(3 * m + 3 * n)" bytes (depending on the max tag
id value, and ignoring some constants) in the best case.
* EncryptKeyProxy server APIs for simulation runs.
Description
diff-2: FlowSingleton util class
Bug fixes
diff-1: Expected errors returned to the caller
Major changes proposed are:
1. EncryptKeyProxy server APIs:
1.1. Lookup Cipher details via BaseCipherId
1.2. Lookup latest Cipher details via encryption domainId.
2. EncyrptKeyProxy implements caches indexed by: baseCipherId &
encyrptDomainId
3. Periodic task to refresh domainId indexed cache to support
'limiting cipher lifetime' abilities if supported by
external KMS solutions.
Testing
EncyrptKeyProxyTest workload to validate the newly added code.
* Fixing leaked stream with explicit notify failed before destructor
* better logic to prevent races in change feed fetching
* Found new race that makes assert incorrect
* handle server overloaded in initial read from fdb
* Handling more blob error types in granule retry
* Fixing rollback metadata problem, added better debugging
* Fixing version race when fetching change feed metadata
* Better racing split request handling
* fixing assert
* Handle change feed popped check in the blob worker
* fix: do not use a RYW transaction for a versionstamp because of randomize API version (#6768)
* more merge conflict issues
* Change feed destroy fixes
* Fixing change feed destroy and move race
* Check error condition in BG file req
* Using relative endpoints for blob worker interface
* Fixing bug in previous fix
* More destroy and move race fixes
* Don't update empty version on destroy in case it gets rolled back. moved() and removing will take care of ensuring it is not read
* Bug fix (#6796)
* fix: do not use a RYW transaction for a versionstamp because of randomize API version
* fix: if the initialSnapshotVersion was pruned, granule history was incorrect
* added a way to compress null bytes in printable()
* Fixing durability issue with moving and destroying change feeds
* Adding fix for not fully deleting files for a granule that child granules need to re-snapshot
* More destroy and move races
* Fixing change feed destroy and pop races
* Renaming bg prune to purge, and adding a C api and unit test for it
* more cleanup
* review comments
* Observability for granule purging
* better handling for change feed not registered
* Fixed purging bugs (#6815)
* fix: do not use a RYW transaction for a versionstamp because of randomize API version
* fix: if the initialSnapshotVersion was pruned, granule history was incorrect
* added a way to compress null bytes in printable()
* fixed a few purging bugs
Co-authored-by: Evan Tschannen <evan.tschannen@snowflake.com>
* Initialize cluster version at wall-clock time
Previously, new clusters would begin at version 0. After this change,
clusters will initialize at a version matching wall-clock time. Instead
of using the Unix epoch (or Windows epoch), FDB clusters will use a new
epoch, defaulting to January 1, 2010, 01:00:00+00:00. In the future,
this base epoch will be modifiable through fdbcli, allowing
administrators to advance the cluster version.
Basing the version off of time allows different FDB clusters to share
data without running into version issues.
* Send version epoch to master
* Cleanup
* Update fdbserver/storageserver.actor.cpp
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
* Jump directly to expected version if possible
* Fix initial version issue on storage servers
* Add random recovery offset to start version in simulation
* Type fixes
* Disable reference time by default
Enable on a cluster using the fdbcli command `versionepoch add 0`.
* Use correct recoveryTransactionVersion when recovering
* Allow version epoch to be adjusted forwards (to decrease the version)
* Set version epoch in simulation
* Add quiet database check to ensure small version offset
* Fix initial version issue on storage servers
* Disable reference time by default
Enable on a cluster using the fdbcli command `versionepoch add 0`.
* Add fdbcli command to read/write version epoch
* Cause recovery when version epoch is set
* Handle optional version epoch key
* Add ability to clear the version epoch
This causes version advancement to revert to the old methodology whereas
versions attempt to advance by about a million versions per second,
instead of trying to match the clock.
* Update transaction access
* Modify version epoch to use microseconds instead of seconds
* Modify fdbcli version target API
Move commands from `versionepoch` to `targetversion` top level command.
* Add fdbcli tests for
* Temporarily disable targetversion cli tests
* Fix version epoch fetch issue
* Fix Arena issue
* Reduce max version jump in simulation to 1,000,000
* Rework fdbcli API
It now requires two commands to fully switch a cluster to using the
version epoch. First, enable the version epoch with `versionepoch
enable` or `versionepoch set <versionepoch>`. At this point, versions
will be given out at a faster or slower rate in an attempt to reach the
expected version. Then, run `versionepoch commit` to perform a one time
jump to the expected version. This is essentially irreversible.
* Temporarily disable old targetversion tests
* Cleanup
* Move version epoch buggify to sequencer
This will cause some issues with the QuietDatabase check for the version
offset - namely, it won't do anything, since the version epoch is not
being written to the txnStateStore in simulation. This will get fixed in
the future.
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
* Change DNS cache to use std::map.
Revert commit 90c259d84e95dd35e01149c0a86bd18e82e33930, because if we use unordered_map, toString() can be inconsistent.
* Move ClientKnob::COORDINATOR_HOSTNAME_RESOLVE_DELAY to FlowKnob::HOSTNAME_RESOLVE_DELAY.
* Move Hostname to its own files.
Also, add resolve-related variables and functions in Hostname.
* OTEL Span Implementation.
* Addi trace logging, refactor constructors, unit tests.
* Unit tests for creating OTELSpans
* refactor flag names
* Additional comments.
* Formatting.
* Add back Arena.h include
* cleanup header includes
* Remove include cstddef.
* Remove memory include.
* Remove trailing commas on enums.
* Enum formatting.
* Changing SpanStatus enum from ERROR to ERR to see if it is clashing with Windows.h.
* Move OTELEvents to SmallVectorRef<KeyValueRef>.
* Clean up unused includes.
* Unit tests
* Const reference arguments for OTEL constructors and additional addAttribute
unit tests. Adding return of OTELSpan reference on addAttribute.
* Formatting.
* Begin messagepack encoding tests.
* Formatting.
* MessagePack encoding unit tests.
* Formatting.
* Remove swapBinary.
* remove ambiguous helper methods
* Formatting fixes
* Fix ambiguous calls in AddEvents unit tests.
* Include AddAttributes unit test.
* descope windows for UDP encoding test
* Move ifndef WIN32 around MPEncoding unit test.
* Fix AddEvents Attributes size assertion.
* Formatting.
* Enable AddLinks unit test.
* Full MP encoding testing.
* Fix for encoding longer strings with MessagePack and unit test.
* Remove unnecessary header includes and serialize_string_ref function.
* Fix typos
* Update flow/Tracing.actor.cpp
Co-authored-by: Lukas Joswiak <lukas.joswiak@snowflake.com>
* Update flow/Tracing.actor.cpp
Co-authored-by: Lukas Joswiak <lukas.joswiak@snowflake.com>
* Use ASSERT_WE_THINK and add logging.
We don't want people creating incredibly large traces, so we are only
supporting a subset of MessagePack collection and string sizes. Assert
and log when we hit these unsupported sizes.
* Remove TODOs no longer applicable.
* Refactor OTELEvent to OTELEventRef.
* Remove unnecessary public declaration in struct.
* fix OTELEventRef attribute size assertion
* Formatting
Co-authored-by: Lukas Joswiak <lukas.joswiak@snowflake.com>
