* ApiTester: test with invalid cluster files
* More asserts in monitorProxies
* ApiTester: Test tampering the cluster file
* Fix update of connection string from the cluster file to use the new connection string only if it valid
* ApiTester: add linker dependency on std++fs
* upgrade_test: no-cleanup-on-error option
* ApiTester: use atomic operations to change and access the transaction handle
Adding the following metrics:
* BlobCipherKeyCache hit/miss
* EKP: KMS requests latencies
* For each component that using encryption, they now need to pass a UsageType enum to the encryption helper methods (GetEncryptCipherKeys/GetLatestEncryptCipherKey/encrypt/decrypt) and those methods will help to log get cipher key latency samples and encryption/decryption cpu times accordingly.
* Assert that arena's appear last in serializer calls
* Fix all occurrences of Arena's not appearing last in serializer call
* Work around issue from Standalone inheriting from Arena privately
* Attempt to fix windows build
Use fb_ prefix instead of detail namespace to scope implementation
details in headers
* Remove API 720 guards for tenants (experimental feature) and the cluster ID special keys (no need to guard)
* Enable the relaxed special key access in transactions that need to use special key-space APIs introduced in 7.2
* ApiTester: enable access to database instances in workloads
* ApiTester: Inject database create errors
* Fix getClusterProtocol and database shared object initialization in case of database create errors
* Check deferred error in all Thread-Safe API calls returning a future
* ApiTester: disable injection of database create errors for tests with old versions
* Save thread indexes for client instances in MVC
* An option to inlude client thread identifiers into trace file names
* Avoid using new trace file options in client versions not supporting them
* Remove redundant include generated by IDE
* Use different base trace file names for client threads by default. Provide an option to revert to using a shared base file name
* Fixing typos in the description of trace_share_among_client_threads
* Encryption data at-rest db-config
Description
diff-1: Handle 'force' updates to encryption_at_rest db-config
Major changes proposed:
1. Introduce 'encryption_data_at_rest_mode" 'configure new'
option to enable Encryption data at-rest. The feature is disabled
by default.
2. The configuration is meant to be set at the time of database
creation, addition checks will be done to avoid updating the config
in subsequent PR.
3. DatabaseConfiguration validity check to account for "tenant_mode"
set to `required` if Encryption data at-rest is selected given
EncryptionDomain matches Tenant boundaries.
Testing
devCorrectness - 100K
* flow: add ApiVersion to replace hard coding api version
Instead of hard coding api value, let's rely on feature versions akin to
ProtocolVersion.
* ApiVersion: remove use of -1 for latest and use LATEST_VERSION
* Move arena members to the end of serializer calls
See
https://github.com/apple/foundationdb/tree/main/flow#flatbuffersobjectserializer
for why this is necessary.
* Fix a heap-use-after-free
Previously memory owned by
EncryptKeyProxyData::baseCipherDomainIdKeyIdCache was borrowed by a call
to EncryptKeyProxyData::insertIntoBaseDomainIdCache where it was
invalidated and then used. Now
EncryptKeyProxyData::insertIntoBaseDomainIdCache takes shared ownership
by taking a Standalone.
And also rename some types to end in Ref to follow the flow conventions
described here: https://github.com/apple/foundationdb/tree/main/flow#arenas
A new knob `ENABLE_STORAGE_SERVER_ENCRYPTION` is added, which despite its name, currently only Redwood supports it. The knob is mean to be only used in tests to test encryption in individual components, and otherwise enabling encryption should be done through the general `ENABLE_ENCRYPTION` knob.
Under the hood, a new `Encryption` encoding type is added to `IPager`, which use AES-256 to encrypt a page. With this encoding, `BlobCipherEncryptHeader` is inserted into page header for encryption metadata. Moreover, since we compute and store an SHA-256 auth token with the encryption header, we rely on it to checksum the data (and the encryption header), and skip the standard xxhash checksum.
`EncryptionKeyProvider` implements the `IEncryptionKeyProvider` interface to provide encryption keys, which utilizes the existing `getLatestEncryptCipherKey` and `getEncryptCipherKey` actors to fetch encryption keys from either local cache or EKP server. If multi-tenancy is used, for writing a new page, `EncryptionKeyProvider` checks if a page contain only data for a single tenant, if so, fetches tenant specific encryption key; otherwise system encryption key is used. The tenant check is done by extracting tenant id from page bound key prefixes. `EncryptionKeyProvider` also holds a reference of the `tenantPrefixIndex` map maintained by storage server, which is used to check if a tenant do exists, and getting the tenant name in order to get the encryption key.
This allows us to return:
[firstKey, transition points, endKey]
where firstKey <= keys.begin and endKey >= keys.end.
This also fixes a prior issue where we were ending the range with a
larger key's value rather than carrying the largest value within the
range to the end.
limits are being modified inplace for subsequent requests. However,
the updated minRows is being checked against the total result instead of
the original minRows.
