`ReadYourWritesTransaction` has memory allocated before being passed to
the main thread. This allows both threads to continue to access the
transaction object. Currently, the transaction gets allocated and
initialized on the foreign thread, and then re-initialized on the main
thread. This causes a bunch of extra, unnecessary work for each
`ReadYourWritesTransaction` where the temporary object gets destructed.
The fix is to only allocate memory for the `ReadYourWritesTransaction`
on the foreign thread, and then initialize it once on the main thread.
* Avoid using structured bindings in doctest assertions
clang doesn't allow this with the latest releases of doctest
This will unblock #7319
* Add constructor to MappedKV
Adding GetEncryptCipherKeys and GetLatestCipherKeys helper actors, which encapsulate cipher key fetch logic: getting cipher keys from local BlobCipherKeyCache, and on cache miss fetch from EKP (encrypt key proxy). These helper actors also handles the case if EKP get shutdown in the middle, they listen on ServerDBInfo to wait for new EKP start and send new request there instead.
The PR also have other misc changes:
* EKP is by default started in simulation regardless of. ENABLE_ENCRYPTION knob, so that in restart tests, if ENABLE_ENCRYPTION is switch from on to off after restart, encrypted data will still be able to be read.
* API tweaks for BlobCipher
* Adding a ENABLE_TLOG_ENCRYPTION knob which will be used in later PRs. The knob should normally be consistent with ENABLE_ENCRYPTION knob, but could be used to disable TLog encryption alone.
This PR is split out from #6942.
Since memory is now limited with RSS size, add RSS size in status json for
reporting. Also change how available_bytes is calculated from:
(available + virtual memory) * process_limit / machine_limit
to:
(available memory) * process_limit / machine_limit
* Fix a heap-use-after-free in a unit test
The data passed to IAsyncFile::write must remain valid until the future
is ready.
* Use holdWhile instead of a new state variable
* Add logic for DDSketch in mako
* Return double from percentile() and fix crash in deserialize()
* make sure to serialize and print result from mergeSketchReport()
* clean up comments
* move ddsketch into its own file
* remove LatencySampleBin and add DDSketch to ThreadStatistics
* Update DDSketch implementation
* remove assertions that cause circular references
* add DDSketchMako as a subsclass from DDSketch
* Merge branch 'ddsketch_mako' of github.com:sfc-gh-khoxha/foundationdb into ddsketch_mako
* Revert "Merge branch 'ddsketch_mako' of github.com:sfc-gh-khoxha/foundationdb into ddsketch_mako"
This reverts commit cc29a68aefd1b385b563bfbaa09a32e399c0d233.
* add ddsketch mako class and rename export flag
* remove redundant decimal roundings
* print max/min/avg from ddsketch
* remove latency sample bin completly
* Make ThreadStatistics dump latency to a file and read from file in printReport()
* make sure to add latency data from file to final stats
* change mergeSketchReport to use new ThreadStatistics serialization (1)
* use C-style string arrays in Arguments instead of std::string
* remove unused header
* only serialize non-empty sketches
* fix CentOS build error
* Update report file count properly
* avoid deserializing empty sketches
* fix segmentation fault when getting file name for export_sketch_path
* make sure to properly add file to report_files list
* fix printing bugs when running in report mode
* fix incorrect insertion of report files
* don't use range based loop for char array
* don't reset args.num_report_files
* Update the usage info for new options
* switch to using std::vector for sketches instead of std::array
* make sure to use true/false instead of 1/0 for booleans
* remove op_name if not being used
* remove fp code in dumpThreadSamples
* replace lambda with function in printReport
* merge and print stats in seperate functions
* make sure to exit after printing report
* address review feedback
* make defaultMin, defaultMax static and move setBucketSize to protected
* switch to reverse iterators when moving backwards along bucket array
* Add simulation test for 1 data hall + 1 machine failure case.
* Disable BUGGIFY for DEGRADED_RESET_INTERVAL.
A simulation test discovered a situation where machines attempting to connect
to a dead coordinator (with a well-known endpoint) were getting themselves
marked degraded. This flapping of the degraded state prevented recovery from
completing, as it started over any time it noticed that tlogs on degraded
hosts could be relocated to non-degraded ones.
bin/fdbserver -r simulation -f tests/rare/CycleWithDeadHall.toml -b on -s 276841956
Description
Major changes proposed include:
1. Update EncryptKeyServer APIs to be tenant aware.
2. Update KmsConnector APIs to be tenant aware
Client of above APIs such as: CP, SS and BlobWorker need to supply
encryption domain info that includes: tenantId and tenantName
Testing
1. Update EncryptKeyProxyTest
2. Update RESTKmsConnectorTest
3. Update SimKmsConnectorTest
