1758 Commits

Author SHA1 Message Date
Markus Pilman
9d16260822 Merge remote-tracking branch 'origin/main' into features/authz 2022-06-28 13:13:51 -06:00
Markus Pilman
1bdbd89839 Merge remote-tracking branch 'origin/main' into features/cmake-refactor-dirs 2022-06-28 08:52:03 -06:00
Markus Pilman
20e7371036 some Linux fixes 2022-06-28 08:51:13 -06:00
Markus Pilman
3448bc6f61 fix fdbmonitor 2022-06-27 19:11:24 -06:00
Markus Pilman
88af6b403f fdbrpc compiling 2022-06-27 18:11:55 -06:00
Markus Pilman
03d913a1de Flow compiling 2022-06-27 17:05:55 -06:00
Lukas Joswiak
c451d47a7e Initialize uninitialized field 2022-06-27 17:05:19 -04:00
Markus Pilman
624430b6ae addess review comments 2022-06-27 14:22:23 -06:00
Markus Pilman
3aaae9c521 Merge remote-tracking branch 'origin/main' into features/authz 2022-06-27 11:07:14 -06:00
Markus Pilman
a47ed89018 Linux fixes and addressed review comments 2022-06-23 20:52:13 -06:00
Markus Pilman
38e100ebc5 flow bindings are compiling 2022-06-23 19:06:05 -06:00
Markus Pilman
de48e90276 fdbserver is now compiling 2022-06-23 18:45:26 -06:00
Markus Pilman
8af056e7b0 fdbclient now compiling 2022-06-23 18:05:36 -06:00
Markus Pilman
9d80ee0cb6 fdbrpc is compiling 2022-06-23 17:15:49 -06:00
Markus Pilman
ffaf15c12a moved wellknownendpoints and fixed some includes 2022-06-23 17:03:53 -06:00
Markus Pilman
10e478dfc3 Flow is compiling 2022-06-23 16:35:19 -06:00
Markus Pilman
d35445a868 enforce include modularization in cmake 2022-06-23 14:37:35 -06:00
Lukas Joswiak
88557d9169 Simplify function call when transaction is null 2022-06-22 14:50:17 -07:00
Renxuan Wang
839af5701e
Fix bug in resolveTCPEndpoint() when hostname resolving fails. (#7375)
* Close trace file when error happens in runNetwork().

* Improve the bestCount algorithm in getLeader().

In the current implementation, if the nominees are [0,1], the chosen leader will be 1, which is an exception to other cases and our expectation that if 2 nominees have the same frequency, the one with lower id will be the leader.

* Remove unnecessary new statement.

stream will never be a nullptr.

* Move self->dnsCache out of lambda capture.

Member variables are not capture by default, thus, `host` and `service` are not captured. This somehow successfully compile, but throws std::bad_alloc or basic_string::_S_create exceptions when we call `host+":"+service` in dnsCache.remove().

* Revert unintended change.

* Address comments.
2022-06-13 20:24:30 -07:00
Markus Pilman
13d8b13722 Migrated Authz code to use JWT 2022-06-13 18:20:27 -06:00
Markus Pilman
799fe32346 Merge remote-tracking branch 'origin/main' into features/authz 2022-06-13 18:02:11 -06:00
Trevor Clinkenbeard
942d687506
Clean up includes in actor header files (#7331)
* Remove unnecessary actorcompiler.h includes (from non-actor files)

* Make AsyncFileChaos a non-actor header file

* Add unactorcompiler.h include to the end of actor header files

* Add missing actorcompiler.h includes to actor header files
2022-06-13 13:26:51 -07:00
A.J. Beamon
986dd67278 Add some basic support for running multiple extra clusters in simulation. Use this to simulate a metacluster in some tests. 2022-06-10 10:08:18 -07:00
Josh Slocum
d6920cde28 Implemented blob granule merging 2022-06-09 10:50:53 -05:00
Dan Adkins
bd47f390bd
Add simulation test for three_data_hall configuration (#7305)
* Add simulation test for 1 data hall + 1 machine failure case.

* Disable BUGGIFY for DEGRADED_RESET_INTERVAL.

A simulation test discovered a situation where machines attempting to connect
to a dead coordinator (with a well-known endpoint) were getting themselves
marked degraded. This flapping of the degraded state prevented recovery from
completing, as it started over any time it noticed that tlogs on degraded
hosts could be relocated to non-degraded ones.

bin/fdbserver -r simulation -f tests/rare/CycleWithDeadHall.toml -b on -s 276841956
2022-06-06 13:14:49 -07:00
Junhyun Shim
3e79735b2f
Authz JWT support (#7279)
* Add JWT support to TokenSign

* Encapsulate OpenSSL public/private key type

Type-safe passing around of keys without having to DER/PEM-serialize
(OpenSSL doesn't have distinct types for public and private key)

* Apply Clang format

* Add verify benchmark for JWT and FlatBuffers token

* Unit test base64url::{encode, decode}

* Make all payload fields optional

Let user code validate non-signature fields

* Make all payload fields optional

Completely defer field check to user code

* Move rapidjson from fdbclient to contrib

* Make fdbrpc's rapidjson linkage private

Currently only sources include them.

* Modify rapidjson path in apiversioner.py

* Algorithm::Unknown > Algorithm::UNKNOWN
2022-06-02 13:22:50 +02:00
Renxuan Wang
0500bcb3a7 Use unique pointer to avoid memory leak. 2022-05-27 14:15:21 -07:00
Renxuan Wang
d502c47bfe Avoid creating new RequestStream when retry. 2022-05-27 14:15:21 -07:00
Renxuan Wang
cd2a575e02
Move the resolve of coordinator hostname from getCoordinatorProtocol() to getClusterProtocolImpl(). (#7245)
* Move the resolve of coordinator hostname from getCoordinatorProtocol() to getClusterProtocolImpl().

* Guard DNS Cache behind a knob.
2022-05-26 09:45:54 -07:00
Josh Slocum
5621fcff40 formatting fix 2022-05-25 17:16:56 -05:00
Josh Slocum
fd7b662767 cleaning up 2022-05-25 17:16:56 -05:00
Josh Slocum
e3a570d50d Blob worker tenant integration mostly works 2022-05-25 17:16:56 -05:00
Zhanwei Wang
da346d718f Fix compatibility issue of s3 backup
1. For v4 signature, http request path should be encoded, currently only parameters are encoded.
2. For v2 signature, http request date header should be in form of either the http date header or x-amz-date, currently in form of ISO 8601
3. Empty response header value cause http_bad_response error.
4. Skip verifying MD5 for range get request and if http response code 206 is returned.
2022-05-25 09:48:54 -05:00
Xiaoxi Wang
5fdf9f99d4 Merge branch 'fix/remove-reference-count' of github.com:sfc-gh-xwang/foundationdb into fix/remove-reference-count 2022-05-23 13:00:36 -07:00
Xiaoxi Wang
15219677aa remove extra reference count inheritance 2022-05-23 13:00:31 -07:00
A.J. Beamon
993a79b42c
Restart protocol monitoring logic when we cycle the coordinator, even if its the same coordinator. Add some defensive code in case we fetch the protocol version for an absent peer. (#6397) 2022-05-19 10:00:23 +02:00
Josh Slocum
a43c98519d
Switching char* to std::string for ProcessInfo to have it own memory (valgrind errors) (#7176) 2022-05-17 12:41:04 -07:00
Vaidas Gasiunas
b55863d958
Merge pull request #7132 from sfc-gh-vgasiunas/vgasiunas-fix-version-check-for-downgrades
Fixing the problem with client getting stuck on server downgrades
2022-05-13 10:53:18 +02:00
Andrew Noyes
a92ef37d44 Log a backtrace before throwing serialization_failed 2022-05-12 20:08:36 -07:00
Renxuan Wang
30e124c09b Remove HostnameStatus and resolve trigger.
They are no longer needed since we have coordinators DNS cache; and they are introducing complex crashes.
2022-05-12 10:13:55 -07:00
Junhyun Shim
0dbbadfd77
Merge pull request #7088 from sfc-gh-jshim/mtls-test-helpers
mTLS test helpers
2022-05-12 14:35:23 +02:00
Vaidas Gasiunas
2f67637701 Fixing the problem with client getting stuck on server downgrades:
- Restoring the original check for strict protocol compatibility before sending packets
 - Resetting the compatibility flag when connection is closed, so that the protocol compatibility is checked again for a new connection
2022-05-11 19:12:38 +02:00
Junhyun Shim
6459e840dc Merge remote-tracking branch 'remotes/origin/main' into mtls-test-helpers 2022-05-11 16:00:17 +02:00
Junhyun Shim
8789232df4 Add ScopeExit to flow and remove scattered impls 2022-05-11 11:51:11 +02:00
A.J. Beamon
0597ae7513 When an incompatible connection is closed, clear the state that prevents us from sending messages to it 2022-05-10 15:29:36 -07:00
Sam Gwydir
b1ce3fc15a WolfSSL fix for TokenSign 2022-05-09 13:57:03 -07:00
Ata E Husain Bohra
33ae398268
REST KmsConnector implementation (#6994)
* REST KmsConnector implementation

Description
  diff-1: Address review comments.
          Add utility interface to Platform namespace to
          create and operate on tmpfile
 diff-2: Address review comments
         Link Boost::filesystem to CMake build process

Major changes includes:
1. Implement REST based KmsConnector implementation.
2. Salient features of the connector:
 2.1. Two required configuration are:
   a. Discovery KMS URLs - enable KMS discovery on bootstrap
   b. Endpoint path configuration to construct URI to fetch/refresh
      encryption keys
   c. Configuration to provide "validationTokens" to connect with
      external KMS. Patch implements file-based token validation scheme.
 2.2. On startup, RESTKmsConnector discovers KMS Urls and caches
      them in-memory. Extracts "validationTokens" based on input config.
 2.3. Expose endpoints to allow fetch/refresh of encryption keys.
 2.4. Defines JSON format to interact with external KMS - request &
      response payload format.
3. Extend Platform namespace with an interface to create and operate on
   tmp files.
4. Update Platform 'readFileBytes' and 'writeFileBytes' to leverage
   fstream supported implementation.

NOTE: KMS URLs fetched after initial discovery will be persisted using
      DynamicKnobs. It is TODO at the moment and shall be completed
      once DynamicKnobs is feature complete

Testing

Unit test to validation following:
1. Parsing on "validation tokens" logic.
2. Construction and parsing of REST JSON request and response strings.
2022-05-07 13:18:35 -07:00
sfc-gh-tclinkenbeard
8ea68154bf Remove WITH_TLS CMake variable 2022-05-02 22:45:00 -07:00
sfc-gh-tclinkenbeard
475d66084d Remove ENCRYPTION_ENABLED macro 2022-05-02 22:26:31 -07:00
Ray Jenkins
dc9e782ccc
OpenTelemetry Tracing Perf Fixes (#6990) 2022-05-02 14:56:51 -05:00