The CTest item
* update_bindings_go_src_fdb_generated_go
will compare the generated go api with hardcoded go api to test the
generation. New FDB option will cause hardcoded go code outdated.
This change attempts to address the SIGSEGV happening when network routine is started
multiple times concurrently.
It changes the network mutex to be a RW mutex, to optimize the case
of calls when network is already started.
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x7f4600000011 pc=0x7f46a405678e]
runtime stack:
runtime.throw({0x1169773?, 0x7f46a479f96c?})
/usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0x7f465affb790 sp=0x7f465affb760 pc=0x44e45d
runtime.sigpanic()
/usr/local/go/src/runtime/signal_unix.go:821 +0x3e9 fp=0x7f465affb7f0 sp=0x7f465affb790 pc=0x466e49
goroutine 60 [syscall]:
runtime.cgocall(0xf4f640, 0xc0001f7f80)
/usr/local/go/src/runtime/cgocall.go:157 +0x6e fp=0xc0001f7f58 sp=0xc0001f7f20 pc=0x41840e
github.com/apple/foundationdb/bindings/go/src/fdb._Cfunc_fdb_run_network()
_cgo_gotypes.go:378 +0x85 fp=0xc0001f7f80 sp=0xc0001f7f58 pc=0xd19d25
github.com/apple/foundationdb/bindings/go/src/fdb.startNetwork.func1()
/home/user/go/pkg/mod/github.com/apple/foundationdb/bindings/go@v0.0.0-20221026173525-97cc643cef69/src/fdb/fdb.go:209 +0x2a fp=0xc0001f7fe0 sp=0xc0001f7f80 pc=0xd2704a
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0001f7fe8 sp=0xc0001f7fe0 pc=0x487741
created by github.com/apple/foundationdb/bindings/go/src/fdb.startNetwork
/home/user/go/pkg/mod/github.com/apple/foundationdb/bindings/go@v0.0.0-20221026173525-97cc643cef69/src/fdb/fdb.go:208 +0x72
goroutine 1 [running]:
goroutine running on other thread; stack unavailable
* throttle hot shards
* expire throttled shards over time
* add backoff
* Parallelize messaging from RK to CP
* Obtain shards from a single SS
* handle expired transactions
* bump transaction_throttled_hot_shard
* Change SevError to SevWarn for CannotMonitorHotShardForSS
* Add log per request
* Update main branch to 7.4
* Update API version to 740
* Makes fdb_c_client_config_tests.py passing after API version update
* Remove from_7.3.0_until_7.4.0 and add from_7.3.0
* Update tests in fdb_c_client_config_tests.py
* Test watch cleanup on cancel
* Fix clearing the database in Java integration tests
* Always cancel the futures wrapped by MVC abortable futures
* More tests for watch cleanup
* Fix clear database database in some Java integration tests
* Add networkoption to disable non-TLS connections
* add disable plaintext connection to fdbserver
* python doc
* Formatting
* Add tls disable plaintext connection to client api test
* review
* fix negative test
* formatting
* add TLS support to c client config tests
Adds support for TLS in the client and server separately
* add tests for disable_plaintext_connections
Test TLS and Plaintext Clusters and Clients
* Fix documentation
* Rename option to indicate it is client-only
* clearer formatting
* default to allowing plaintext connections
* add SetTLSDisablePlaintextConnection to go bindings
Fix `RangeResult.readThrough` misuses:
1. KeyValueStores do not need to set readThrough, as it will not be
serialized and return. Also setting it to the last key of the result
is not right, it should at least be the keyAfter of the last key;
2. Fix NativeAPI doesn't set `RangeResult.more` in a few places;
3. Avoid `tryGetRange()` setting `readThrough` when `more` is false,
which was a workaround for the above item 2;
4. `tryGetRangeFromBlob()` doesn't set `more` but set `readThrough` to
indicate it is end, which was following the same above workaround I
think. Fixed that.
5. `getRangeStream()` is going to set `more` to true and then let the
`readThrough` be it's boundary.
Also added readThrough getter/setter function to validate it's usage.
* EaR: Implement Key Check Value semantics
Description
Key Check Value (KCV) is a checksum of cryptographic encryption key
used to validate encryption keys's integrity. FDB Encryption at-rest
relies on external KMS to supply encryption keys.
Patch proposes following major changes:
1. Implement Sha256 based KCV implementation to protect against
'baseCipher' corruption in two possible scenarios:
a) potential corruption external to FDB
b) potential corruption within FDB processes.
2. Scheme persists computed KCV token in block encryption header,
which then gets validated as part of header validation during
decryption.
3. FDB Encryption key derivation uses HMAC_SHA256 digest generation
scheme, which allows max 64 bytes of 'cipher buffer', patch add
required check to ensure 'baseCipher' length are within bounds.
OpenSSL HMAC underlying call ignores extra length if supplied, however,
it weakens the security guarantees, hence, disallowed.
Testing
devRunCorrectness - multiple 500K runs
Valgrind & Asan - BlobCipherUnit, RESTKMSUnit, BlobGranuleCorrectness*,
EncryptionOps, EncryptKeyProxyTest
* Api Tester: Specify knobs in the toml file; Test loop profiler
* Gracefully stop the loop profiler thread
* Protect loop profiler thread by mutex
* Create loop profiler thread only if is not stopped
