apple/foundationdb
1
0
Fork 0
mirror of https://github.com/apple/foundationdb.git synced 2025-05-14 18:02:31 +08:00
Code Issues Packages Projects Releases Wiki Activity
24,317 Commits 35 Branches 318 Tags
Commit Graph

430 Commits

Author SHA1 Message Date
Xiaoxi Wang
bbcb3cc018 extract KeyBackedConfig, StorageWiggleData class; solve template resolution problem; solve MV txn and native api conflict by splitting RunTransaction file 2023-01-02 23:34:39 -08:00
A.J. Beamon
b1fb8e8c7b Resolve merge issue by renaming namespace (combining with existing TenantAPI namespace) 2022-12-05 15:47:58 -08:00
A.J. Beamon
ce4a2a55b2 Remove the usage of tenant names (mostly) from the storage server 2022-12-05 15:44:41 -08:00
Hui Liu
c1772486db Incremental restore - preload ApplyMutationsKeyVersionMap 2022-11-30 19:29:24 -08:00
Ata E Husain Bohra
91fc3fef4a
[EAR]: Remove usage of EncryptDomainName for Encryption at-rest operations (#8715) 
* [EAR]: Remove usage of EncryptDomainName for Encryption at-rest operations

Description

 diff-1: Address review comments

EncryptDomainName is an auxillary information, given EAR encryption domain
matches with Tenants, EncryptDomainName maps to TenantName in the current
code. However, this mapping adds EAR depedency has multiple drawbacks:
1. In some scenarios obtaning consistent mapping of TenantId <-> TenantName
   is difficult to maintain. For instance: StorageServer (SS)  TLog mutation
   pop loop, it is possible that same commit batch contains: TenantMap update
   mutation as well as a Tenant user mutation. SS would parse TenantMap update
   mutation (FDB System Keyspace encryption domain), process the mutation, but,
   doesn't apply it to the process local TenantMap. SS then attempts to process,
   Tenant user mutation and fails to decrypt the mutation given TenantMetadaMap
   isn't updated yet.
2. FDB codebase uses EncryptDomainId matching TenantId, TenantName is used as
   an auxillary information source and feels better to be handled by an
   external KMS.

Major changes include:
1. EAR to remove TenantName dependency across all participating processes
   such as: CommitProxy, Redwood, BlobGranule and Backup agent.
2. Update EKP and KmsConnector APIs to avoid relying on "domainName"
   information being passed around to external KMS EAR endpoints.

Testing

devRunCorrectness - 100K
EncryptKeyProxyTest - 100K
EncryptionOps Test - 100K
 2022-11-16 10:26:39 -08:00
Nim Wijetunga
a21f657429
Add Encryption Code Probes for BlobGranule (#8779) 
* add code probes

* address pr comments

* address pr comments

* Trigger Build

* fix
 2022-11-16 06:13:55 -08:00
Nim Wijetunga
a1a2839cb3
Revert "Extend Transaction Window during Restore in Simulation" (#8806) 
* fix resolver bug

* trigger build
 2022-11-12 15:24:30 -08:00
Nim Wijetunga
8bd5f362a5
Extend Transaction Window during Restore in Simulation (#8729) 
* fix bug

* fix formatting
 2022-11-08 11:53:36 -08:00
Jingyu Zhou
6c0f890f78 Fix restarting restore test failure 
Old fdbserver may not set the "enableSnapshotBackupEncryption" key, thus we
should allow the key to be not present.
 2022-10-27 08:43:55 -07:00
Nim Wijetunga
6f37f55917
Restore System Keys First in Backup/Restore Workloads (#8475) 
* system key restore ordering

* restore system keys before regular data

* atomic restore backup fix

* change testing

* fix compile error

* fix compile issue

* fix compile issues

* Trigger Build

* only split restore if encryption is enabled

* revert knob changes

* Update fdbserver/workloads/AtomicSwitchover.actor.cpp

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>

* Update fdbserver/workloads/AtomicSwitchover.actor.cpp

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>

* Update fdbserver/workloads/BackupCorrectness.actor.cpp

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>

* Update fdbserver/workloads/AtomicRestore.actor.cpp

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>

* add todo

* strengthen check

* seperate system restore for atomic restore

* address pr comments

* address pr comments

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
 2022-10-26 09:38:27 -07:00
Steve Atherton
344290fdf1 IO buffer lifetime fix, buf must stay alive until the file read completes. 2022-10-23 16:28:31 -07:00
Jingyu Zhou
57fc39d5b7 Make decodeKVPairs a non-actor function 2022-10-20 19:34:44 -07:00
Nim Wijetunga
d439bc1e6e
TenantEntryCache Watch Based Refresh (#8399) 
* tenant modification changekey

* address pr comments

* change backup to use watch based tenant cache

* format

* address pr comments

* trigger build

* add todo

* check tenants disabled

* trigger build

* trigger build

* address pr comments

* address pr comments

* trigger build
 2022-10-18 19:05:07 -07:00
Nim Wijetunga
8ca6397e97
Enable Backup Tests for Encrypted Snapshots (#8419) 
* enable encrypted snapshot backup tests

* address pr comments

* remove tenant cache when decoding

* remove client knob
 2022-10-12 10:45:30 -07:00
Markus Pilman
ea1325a552
Merge pull request #8319 from sfc-gh-tclinkenbeard/add-rare-code-probe-annotation 
Add `rare` code probe decoration
 2022-10-07 09:39:00 -06:00
Markus Pilman
550488b020 Merge remote-tracking branch 'origin/main' into bugfixes/open-for-ide 
# Conflicts:
#	bindings/c/CMakeLists.txt
#	fdbclient/include/fdbclient/GetEncryptCipherKeys.actor.h
#	fdbserver/BackupWorker.actor.cpp
#	fdbserver/BlobWorker.actor.cpp
#	fdbserver/CommitProxyServer.actor.cpp
#	fdbserver/KeyValueStoreMemory.actor.cpp
#	fdbserver/StorageCache.actor.cpp
#	fdbserver/include/fdbserver/GetEncryptCipherKeys.actor.h
#	fdbserver/storageserver.actor.cpp
#	fdbserver/workloads/PhysicalShardMove.actor.cpp
#	flow/CMakeLists.txt
 2022-10-04 18:27:48 -06:00
A.J. Beamon
55b880432e
Merge pull request #8379 from sfc-gh-ajbeamon/literal-string-ref-removal 
LiteralStringRef removal
 2022-10-04 09:41:42 -07:00
Nim Wijetunga
778cb63f71
Ensure nullptr is not passed to memcpy (#8376) 
* fix

* fix

* Trigger Build
 2022-09-30 17:28:43 -07:00
A.J. Beamon
111bf9e34d Add macros for literal string ref versions of __FUNCTION__ and __FILE__ 2022-09-30 16:05:47 -07:00
Nim Wijetunga
232e23c87e
Snapshot Backup Encryption (#8095) 
* inital commit

* add encryption support

* refactor passing encryption keys

* disable encryption codepath

* pass enable encryption cli arg for fdbbackup

* change encryption key passing

* only enable encryption for backup if tenant mode is required

* revert server knobs

* ignore json trace files

* --amend

* integrate ekp

* add comments

* comments

* comments

* revert knobs

* fix bug

* modify encrypted backup logic

* use common prefix

* check boundaries between begin and end key

* return system encrypt domain

* modify

* add tenant entry cache to backup

* slight revert

* format

* Trigger Build

* format

* use tenant name to fetch keys

* Trigger Build

* Trigger Build

* merge

* fix getting ekp

* fix bug

* address pr comments

* Trigger Build

* Trigger Build

* remove faulty check
 2022-09-29 14:45:47 -07:00
A.J. Beamon
3353103d9d Fix filtering of potential backup mutations in commit proxy and backup worker; add code probe to ensure we are testing default backup sharing and add some tests to hit it 2022-09-27 15:25:30 -07:00
sfc-gh-tclinkenbeard
985958c260 Add rare code probe decoration 2022-09-25 15:28:32 -07:00
A.J. Beamon
a7c200b16c Add support for sharing mutation streams between a default backup and DR 2022-09-22 11:36:46 -07:00
A.J. Beamon
fda0d7223d Update backup to include system key ranges needed for tenants. Run simulated backup tests with tenants. 2022-09-22 10:00:13 -07:00
Nim Wijetunga
eadb769cfa
Encrypt Backup Mutation Log (#8159) 
* encrypt backup mutation log

* format

* address pr comments

* format

* fix bug

* revert knobs

* address pr comments
 2022-09-20 15:43:39 -07:00
A.J. Beamon
4fd64630e8 Convert literal string ref instances to use _sr suffix 2022-09-19 11:35:58 -07:00
Nim Wijetunga
91fb7c72c8 remove code probes 2022-09-14 10:40:35 -07:00
A.J. Beamon
410f27412b
Merge pull request #7620 from sfc-gh-ajbeamon/make-tuple 
Add a Tuple::makeTuple function to easily construct a tuple
 2022-07-20 17:09:10 -07:00
A.J. Beamon
fee131f24d
Merge pull request #7591 from sfc-gh-ajbeamon/key-backed-types-range-read-more 
Add support for returning a more flag from key backed types' range reads
 2022-07-20 15:57:50 -07:00
A.J. Beamon
190ad8c7e9 Convert existing tuple usages to use Tuple::makeTuple() 2022-07-19 13:45:59 -07:00
Markus Pilman
1de37afd52
Make TEST macros C++ only (#7558) 
* proof of concept

* use code-probe instead of test

* code probe working on gcc

* code probe implemented

* renamed TestProbe to CodeProbe

* fixed refactoring typo

* support filtered output

* print probes at end of simulation

* fix missed probes print

* fix deduplication

* Fix refactoring issues

* revert bad refactor

* make sure file paths are relative

* fix more wrong refactor changes
 2022-07-19 13:15:51 -07:00
A.J. Beamon
1b81e72604 Add a Tuple::makeTuple function to easily construct a tuple. Update Tuple to allow all types to be passed via .append() so they can be used with makeTuple. 2022-07-19 11:50:58 -07:00
A.J. Beamon
91949439d2 Add support for returning whether a key-backed type range read has more results or not. 2022-07-13 15:46:10 -07:00
A.J. Beamon
62683b4140 Update key backed types to support a templated Codec. This required updating the existing Codec to convert from objects into byte strings through tuples. 2022-07-01 10:11:02 -07:00
A.J. Beamon
1dc78a3b3f Add support for different transaction types (Reference<ITransaction>, Reference<ReadYourWritesTransaction>) to key backed types 2022-07-01 10:02:39 -07:00
Markus Pilman
ffaf15c12a moved wellknownendpoints and fixed some includes 2022-06-23 17:03:53 -06:00
Markus Pilman
d35445a868 enforce include modularization in cmake 2022-06-23 14:37:35 -06:00
Renxuan Wang
0a332ee1c1 Add proxy option to backup and restore params. 2022-03-28 17:10:49 -07:00
sfc-gh-tclinkenbeard
a71099471b Update copyright header dates 2022-03-21 13:36:23 -07:00
Jingyu Zhou
1a5bf25b5c Update code base to use fmt 8.1.1 2022-03-04 15:52:06 -08:00
Renxuan Wang
f7eb66441d Try eliminating warnings in macOS and Windows CI builds. 
MacOS warnings are format warnings, e.g., `format specifies type 'long' but the argument has type 'Version' (aka 'long long')`.
Windows warnings are `ACTOR does not contain a wait() statement`.
 2022-02-25 19:06:57 -08:00
A.J. Beamon
250a88e682 Enforce that trace event suppression calls happen first when using trace event call chaining. Fix various instances where we weren't following this requirement. 2022-02-24 12:25:52 -08:00
sfc-gh-tclinkenbeard
ec64890ac1 Remove some usages of PRId64 by using fmt library 2021-11-30 23:35:36 -08:00
sfc-gh-tclinkenbeard
ebcc023b6f Enable missing-field-initializers clang warning 2021-11-01 14:18:31 -07:00
Chaoguang Lin
65956ae6b7 Refactor configure command; refactor changeConfig to template code to reuse existing tests 2021-09-21 10:06:04 -07:00
FDB Formatster
2c788c233d apply clang-format to *.c, *.cpp, *.h, *.hpp files 2021-08-27 17:07:47 -07:00
Jingyu Zhou
426e906a87 Remove duplicated code 2021-08-01 22:39:28 -07:00
Steve Atherton
507c1f11e3 Add .log() to bare TraceEvent() invocations without any .detail()s to avoid clang-tidy warning about immediate destruction of object without use. 2021-07-26 19:55:10 -07:00
Steve Atherton
f596a81073 Rename ::TRUE and ::FALSE in BooleanParams to ::True and ::False so as to not conflict with the TRUE and FALSE macros provided by the Windows and MacOS SDKs. 2021-07-17 00:11:40 -07:00
sfc-gh-tclinkenbeard
382533c4e1 Add FDB_ prefix to BOOLEAN_PARAM macros 2021-07-09 05:42:14 -07:00