2890 Commits

Author SHA1 Message Date
Ata E Husain Bohra
333aadb903
Interface to enable clients to send/receive REST requests/responses (#6866)
* Interface to enable clients to send/receive REST requests/responses

Description

Major changes:
1. Add RESTClient interface enabling client to send/receive REST HTTP
   requests. Support REST APIs are: get, head, put, post, delete, trace
2. Add RESTUtil file introducing below interfaces:
 2.1. RESTUrl - Extract URI information: host, service, request-parameters.
 2.2. RESTConnectionPool-
      Connection establishment, life-cycle management, connection-pool (TTL)
 2.3. RESTClientKnobs - supports REST Knob parameter management and updates

Testing

Unit test - fdbrpc/RESTClient, fdbrpc/RESTUtils
2022-04-27 12:17:52 -07:00
Markus Pilman
cbe4a873d2 Merge remote-tracking branch 'origin/main' into features/validate-trace-events-in-simulation 2022-04-25 17:39:29 -06:00
Ray Jenkins
1c5bf135d5
Revert "Migrate to OpenTelemetry tracing. (#6855)" (#6941)
This reverts commit 5df3bac110d9b5b88931b008b852433688bb7eb0.
2022-04-25 09:29:56 -05:00
Bharadwaj V.R
08323de905 fix formatting 2022-04-22 15:10:24 -07:00
Bharadwaj V.R
588b2fa509
Merge branch 'main' into block-down 2022-04-22 14:53:09 -07:00
Bharadwaj V.R
988a70f064
Merge pull request #6858 from sfc-gh-bvr/dbcorever
Track newest and lowest compatible protocol versions in DBCoreState
2022-04-22 14:46:21 -07:00
Bharadwaj V.R
4a5c2268da
Merge branch 'apple:main' into block-down 2022-04-22 14:45:54 -07:00
Markus Pilman
9e65e15b45 Merge remote-tracking branch 'origin/main' into features/validate-trace-events-in-simulation 2022-04-22 15:39:55 -06:00
Ata E Husain Bohra
6c9030408e Fix Build: use boost::hash to compute hash for std::pair
Description

Fix Build: use boost::hash to compute hash for std::pair

Testing

1. Build - gcc/clang
2. Simulation test: EncryptKeyProxyTest, EncryptionOps
3. Unit test: flow/BlobCipher
4. Running 10k correctness Joshua run
2022-04-22 13:16:30 -07:00
Ata E Husain Bohra
670d40ef79
FDB native KMS Connector Framework (#6846)
* FDB native KMS Connector Framework

Description

Major changes includes:
1. Framework code to enable FDB native KMS connector implementation.
2. SERVER_KNOBS->KMS_CONNECTOR_TYPE controls the connector type selection.
3. KmsConnectorInterface endpoint definitions, every KMSConnector
   implementation needs to support defined endpoints.
4. Update EncryptKeyProxy to leverage KmsConnectorInterface endpoints
   to fetch encryption keys on-demand and/or periodic refreshes.
   Integrate SimKmsConnector implementation.
5. Implement SimKmsConnector by leveraging existing SimKeyProxy
   implementation.

Testing

Unit test: fdbserver/SimKmsConnector
Simulation: EncryptKeyProxy
2022-04-22 08:53:39 -07:00
Bharadwaj V.R
822eb9ec26
Merge branch 'apple:main' into dbcorever 2022-04-22 08:08:34 -07:00
Bharadwaj V.R
ed08cfbf52
Merge branch 'apple:main' into block-down 2022-04-22 06:19:38 -07:00
Ata E Husain Bohra
04ecd8e08f
Revert "Revert "Update 'salt' details for EncryptHeader AuthToken details (#6881)" (#6902)" (#6922)
Description

Major changes proposed:
1. This reverts commit f38b2e82098f432bc4437f823c5c1dafd52e1319.
2. Also add fix for Valgrind failure due to unintialized variables.
3. Improve checks to catch is cipherKey details cached in BlobCipherKeyCache
   isn't as expected

Testing

Overall correctness: 10K (20220421-193911-ahusain-foundationdb-a730e5cb38541e20)
EncyrptionOps correctness: 100K (20220421-194315-ahusain-foundationdb-29c598a8b9420430)
EncryptionOps Valgrind: 100 (20220421-194434-ahusain-foundationdb-7fc5f98eddc0921a)
2022-04-21 18:57:56 -07:00
Bharadwaj V.R
449a315c06
Merge branch 'apple:main' into block-down 2022-04-21 09:37:42 -07:00
Bharadwaj V.R
c20fb6ef6d
Merge branch 'apple:main' into dbcorever 2022-04-21 09:37:29 -07:00
Markus Pilman
f38b2e8209
Revert "Update 'salt' details for EncryptHeader AuthToken details (#6881)" (#6902)
This reverts commit a38318a6acc1ef71ebcc5ea4af6bb2ea0edf0887.
2022-04-21 09:04:40 -07:00
Markus Pilman
bbb1392aad Merge remote-tracking branch 'origin/main' into features/validate-trace-events-in-simulation 2022-04-21 08:24:18 -06:00
Markus Pilman
85757eb47c
Update flow/Trace.cpp 2022-04-20 15:30:42 -06:00
Renxuan Wang
e40cc8722c
A few hostname improvements. (#6825)
* Add tryResolveHostnames() in connection string.

* Add missing hostname to related interfaces.

* Do not pass RequestStream into *GetReplyFromHostname() functions.

Because we are using new RequestStream for each request anyways. Also, the passed in pointer could be nullptr, which results in seg faults.

* Add dynamic hostname resolve and reconnect intervals.

* Address comments.
2022-04-20 13:42:46 -07:00
Markus Pilman
3335b2686e
Merge branch 'main' into features/validate-trace-events-in-simulation 2022-04-20 12:03:33 -06:00
Markus Pilman
f7a8ebf818
Update flow/Trace.cpp
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2022-04-20 11:55:01 -06:00
Bharadwaj V.R
8a0ce5bfc7 Rename isInvalidMagic and fix formatting 2022-04-20 09:17:28 -07:00
Bharadwaj V.R
a2449041ea Fix formatting of ProtocolVersion.h 2022-04-20 08:45:53 -07:00
Bharadwaj V.R
4d6f4ecd9c
Merge branch 'main' into dbcorever 2022-04-20 08:23:34 -07:00
Ray Jenkins
5df3bac110
Migrate to OpenTelemetry tracing. (#6855) 2022-04-20 09:26:37 -05:00
Bharadwaj V.R
a711c55061
Merge branch 'apple:main' into dbcorever 2022-04-20 06:16:27 -07:00
Bharadwaj V.R
89af5561f1
Merge branch 'apple:main' into block-down 2022-04-20 06:13:01 -07:00
Markus Pilman
d4ee7be1d7 Reduce excessive tracing and fail after 1M traces 2022-04-19 21:11:51 -06:00
Andrew Noyes
297d831192
Put guard pages next to fast alloc memory (#6885)
* Put guard pages next to fast alloc memory

I verified that we can now detect #6753 without creating tons of
threads.

* Use pageSize instead of 4096

* Don't include mmapInternal for windows
2022-04-19 11:22:35 -07:00
Bharadwaj V.R
51ef860612
Merge branch 'apple:main' into block-down 2022-04-19 10:16:56 -07:00
Ata E Husain Bohra
a38318a6ac
Update 'salt' details for EncryptHeader AuthToken details (#6881)
* Update 'salt' details for EncryptHeader AuthToken details

Description

Major changes:
1. Add 'salt' to BlobCipherEncryptHeader::cipherHeaderDetails.
2. During decryption it is possible that BlobKeyCacheId doesn't
    contain required baseCipherDetails. Add API to KeyCache to
    allowing re-populating of CipherDetails with a given 'salt'
3. Update BaseCipherKeyIdCache indexing using {BaseCipherKeyId, salt}
    tuple. FDB processes leverage BlobCipherKeyCache to implement
    in-memory caching of cipherKeys, given EncryptKeyProxy supplies
    BaseCipher details, each encryption participant service would
    generate its derived key by using different 'salt'. Further,
    it is possible to cache multiple {baseCipherKeyId, salt} tuples;
    for instance: CP encrypted mutations being deciphered by
    StorageServer etc.

Testing

1. Update EncyrptionOps simulation test to simulate KeyCache miss
2. Update BlobCipher unit tests to validate above mentioned changes
2022-04-18 22:01:56 -07:00
Bharadwaj V.R
b10ba334de
Merge branch 'apple:main' into block-down 2022-04-18 08:56:04 -07:00
Bharadwaj V.R
47a1a282e9
Merge branch 'apple:main' into dbcorever 2022-04-15 15:47:14 -07:00
Andrew Noyes
29cf5f1fbf
Fix an ASSERT when an fdbcli command times out (#6857)
* Re-throw operation_cancelled

There's a few places in fdbcli where we don't rethrow operation
cancelled but wait on a future. It's very unusual that you don't want to
rethrow operation_cancelled.

* Update ASSERT

It's possible to get error_code_broken_promise here if the network has
already shutdown.
2022-04-14 12:09:25 -07:00
Bharadwaj V.R
78c4771f9d Fix ProtocolVersion.h formatting diffs 2022-04-14 11:10:39 -07:00
Bharadwaj V.R
3c080c6382
Merge branch 'apple:main' into dbcorever 2022-04-14 10:39:32 -07:00
Bharadwaj V.R
85904ec739
Merge branch 'apple:main' into block-down 2022-04-14 07:23:16 -07:00
Junhyun Shim
edc659d339 Use camelCase & move error code to 6xxx 2022-04-13 21:11:52 +02:00
Junhyun Shim
b6a0c0f942 Merge remote-tracking branch 'upstream/main' into tenant-token-sign 2022-04-13 19:55:37 +02:00
Bharadwaj V.R
3cbe7f7d63 Update min compatible version to 7.1 2022-04-13 09:47:12 -07:00
Bharadwaj V.R
2f2ece073c Add sw version tracking to DBCoreState 2022-04-13 08:05:22 -07:00
Bharadwaj V.R
3e50dda79f
Merge branch 'apple:main' into block-down 2022-04-13 06:58:59 -07:00
Lukas Joswiak
0783b044fe Add protocol feature 2022-04-12 14:35:09 -07:00
Bharadwaj V.R
e4f22e5394
Merge branch 'apple:main' into block-down 2022-04-12 10:50:02 -07:00
Bharadwaj V.R
46b22b79ea Revert "Add SW versions to DBCore state"
This reverts commit 50b45f1bf953c500194d45c5025b177a6f4eb1dd.
2022-04-12 10:47:49 -07:00
Aaron Molitor
6b2d0e5d6b update version to 7.2.0 -- protocol version 2022-04-11 23:23:27 -05:00
Aaron Molitor
c440365779 update version to 7.2.0 -- pr comment protocol version 2022-04-11 23:23:27 -05:00
Aaron Molitor
8db8545db2 update version to 7.2.0 -- protocol version changes 2022-04-11 23:23:27 -05:00
Bharadwaj V.R
50b45f1bf9 Add SW versions to DBCore state 2022-04-11 12:01:12 -07:00
Vaidas Gasiunas
ca563466a6
Merge pull request #6401 from sfc-gh-mpilman/features/private-request-streams
Features/private request streams
2022-04-11 18:29:06 +02:00