21210 Commits

Author SHA1 Message Date
Markus Pilman
1de37afd52
Make TEST macros C++ only (#7558)
* proof of concept

* use code-probe instead of test

* code probe working on gcc

* code probe implemented

* renamed TestProbe to CodeProbe

* fixed refactoring typo

* support filtered output

* print probes at end of simulation

* fix missed probes print

* fix deduplication

* Fix refactoring issues

* revert bad refactor

* make sure file paths are relative

* fix more wrong refactor changes
2022-07-19 13:15:51 -07:00
A.J. Beamon
0d20cd2068
Merge pull request #7618 from sfc-gh-ajbeamon/fix-compile-warnings
Fix a few compile warnings about unused variables
2022-07-19 11:10:42 -07:00
A.J. Beamon
c4cd72bea1 Fix a few compile warnings about unused variables 2022-07-19 10:01:41 -07:00
A.J. Beamon
0e03fbd9e7
Merge pull request #7605 from sfc-gh-ajbeamon/fix-trace-log-detail-formatting
Remove some instances of using printable or toString in a trace event detail
2022-07-19 08:24:58 -07:00
Dan Lambright
775ab3323e
update tag client cache on getKeyLocation (#7613) 2022-07-19 11:24:17 -04:00
Trevor Clinkenbeard
1eec6f993d
Merge pull request #7572 from sfc-gh-akejriwal/tenantquota
Introduce storage quotas per tenant
2022-07-18 20:00:22 -07:00
A.J. Beamon
38855fd786 Remove some instances of using printable or toString in a trace event detail. 2022-07-18 16:24:33 -07:00
Ankita Kejriwal
2757262ec9
Simplify storageQuotaKey() function implementation
Co-authored-by: Trevor Clinkenbeard <trevor.clinkenbeard@snowflake.com>
2022-07-18 15:56:11 -07:00
Renxuan Wang
570db7a7d2
Fix processDiskReadSeconds and processDiskWriteSeconds. (#7608) 2022-07-18 13:47:45 -07:00
Jingyu Zhou
d28fb2b809
Merge pull request #7607 from sfc-gh-jshim/dlopen-nodelete-for-sanitizer
Apply RTLD_NODELETE for USE_SANITIZER builds
2022-07-18 13:25:33 -07:00
Trevor Clinkenbeard
18a2c3f8a8
Merge pull request #7489 from sfc-gh-tclinkenbeard/cstate-pimpl
Mark `CoordinatedState::getConflict` const
2022-07-18 13:22:18 -07:00
Junhyun Shim
8b70c7050d Apply RTLD_NODELETE for USE_SANITIZER builds
This allows their stack trace to be symbolized in CI/Jenkins XSAN runs
2022-07-18 19:25:12 +02:00
Andrew Noyes
989e323991
Attempt to fix windows build (#7600)
* Attempt to fix windows build

The windows build is complaining that some symbols in flow are
duplicate, which seems fair since we're currently compiling all the flow
sources and _also_ linking to flow for flowlinktest. This change makes
it so that we only link flow, and don't compile the flow source files
again.

* Remove source files from fdbclient and fdbrpc link tests
2022-07-18 08:59:03 -07:00
Ankita Kejriwal
882df774c0 Storage quota: use transaction options and minor fixes in the test 2022-07-15 16:34:04 -07:00
A.J. Beamon
02ab3b05ab
Merge pull request #7590 from sfc-gh-ajbeamon/update-prefix-encoding-in-tenant-metadata-json
Use a more standard encoding in the JSON metadata reported for tenants
2022-07-15 14:32:53 -07:00
Josh Slocum
fb4fc8af9d
Merge pull request #7587 from sfc-gh-ahusain/ahusain-hybrid_encryption_active
BlobGranule encryption and compression support
2022-07-15 15:57:51 -05:00
Ata E Husain Bohra
05e7f34c80 BlobFile Encryption and compression support
Fix windows CI build issue.

Description

Testing
2022-07-15 11:04:02 -07:00
He Liu
4dc742493e
Add cleanUpChangeFeeds(). (#7598)
Co-authored-by: He Liu <heliu@apple.com>
2022-07-15 10:29:09 -07:00
Ata E Husain Bohra
1da88d5d8c BlobFile Encryption and compression support
Fix formatting issues

Description

Testing
2022-07-14 22:46:14 -07:00
Ata E Husain Bohra
f288abebc2 BlobFile Encryption and compression support
Fix formatting issues and rename KNOB

Description

Testing
2022-07-14 17:22:00 -07:00
Ankita Kejriwal
6f130b4a69 Change quota type to unit64_t to avoid macOS compilation issue 2022-07-14 17:09:03 -07:00
Ata E Husain Bohra
f6f117592d BlobFile Encryption and compression support
- Limit verbose logging under DEBUG_MACRO
 - Update/Add code documentation

Description

Testing
2022-07-14 17:04:14 -07:00
Ata E Husain Bohra
3a3811d6cc BlobFile Encryption and compression support
Few fixes
 - Update BlobGranuleCipherKeysMeta to use 'std::string' to
   persist 'Initialization Vector'
 - Update WriteSnapshot to persist 'BlobGranuleCipherKeysMeta' as part
   of BlobFileIndex

Description

Testing
2022-07-14 17:04:14 -07:00
Ata E Husain Bohra
24b2de8de8 BlobFile Encryption and compression support
Description

Testing
2022-07-14 17:04:14 -07:00
Ankita Kejriwal
bb05321d24 Introduce storage quotas per tenant.
This change adds:
* ability to store the mapping from tenants to quota in the system keyspace,
* a setter and getter function
* a new workload to test this functionality

FDBCORE-2437
2022-07-14 16:35:12 -07:00
Xiaoxi Wang
4cbe5dd683
Merge pull request #7595 from sfc-gh-xwang/feature/dd-refactor-incremental
[DD testability] add resumeRelocation method
2022-07-14 15:24:56 -07:00
Yi Wu
7d7ce0909f
Restart tests carry forward encryption knobs value (#7497)
Previously to get around the issue that EKP is not present when restart test switching encryption from on to off and read encrypted data, EKP was made to start in simulation regardless of encryption knob. This PR revert that change, and instead force restart test not to change encryption knob, by passing previous encryption knob through restartInfo.ini file. Also since we don't allow downgrading an encrypted cluster to previous version, disable encryption in downgrade tests.

Also adding an assert to allow reading encrypted mutations only if encryption knob is on. We may reconsider allowing switching encryption on/off for existing cluster, but for now we don't allow it.
2022-07-14 14:45:17 -07:00
neethuhaneesha
598457b872
Merge pull request #7551 from neethuhaneesha/rocksCompressionStats
Adding rocksdb compression and data size stats.
2022-07-14 12:58:10 -07:00
Xiaoxi Wang
f53c108e4f
Update fdbserver/DataDistribution.actor.cpp 2022-07-14 12:17:34 -07:00
Xiaoxi Wang
42bef3b265
Update fdbserver/DataDistribution.actor.cpp 2022-07-14 12:17:29 -07:00
Xiaoxi Wang
7ee091d1da add test namespace 2022-07-14 12:13:23 -07:00
Xiaoxi Wang
015026f146 add comment; add resumeRelocation 2022-07-14 11:32:27 -07:00
neethuhaneesha
3cbb51de21 Adding rocksdb compression and data size stats. 2022-07-14 11:28:20 -07:00
Marian Dvorsky
ce4dc0fb0f
Update ctest_to_joshua to work with new ctests (#7584)
* Package inferred directories to the resulting tarball in ctest_to_joshua (except for the root source and build directories). This allows the API tests (which specify the workloads as a directory containing the workload specification files) to now work.
* Add bin/mkcert which is used by some tests.
* Rewrite fdb_c_shim_tests.py to specify its dependencies on command line.
2022-07-14 18:34:48 +02:00
Xiaoxi Wang
87f833469b Merge branch 'main' of https://github.com/apple/foundationdb into feature/dd-refactor-incremental 2022-07-14 09:15:51 -07:00
Zhongxing Zhang
9c20b15f5a
Use enum variables to invoke Priority checking (#7514)
* Use enum variables to invoke Priority checking

* add an explicit isDataMovementForReadBalancing function
2022-07-14 09:06:56 -07:00
Lukas Joswiak
407300bfa6 Disable testing of the remote key value store in simulation 2022-07-13 18:32:50 -07:00
Ray Jenkins
bdef35390d Conditional on context.isSampled in Span operator=().
In operator= overload we need to check if the current span which is being
overwritten is eligble for serialization. If so we must send to the tracer
via the g_tracer->trace(*this) call. Previously we were incorrectly checking
the function argument o's context for sampling. This resulted in loss traces
for the NAPI:readVersionBatcher spans.
2022-07-13 17:50:51 -07:00
Xiaoxi Wang
7e3e79c53e
Merge pull request #7585 from sfc-gh-xwang/feature/dd-refactor-incremental
[DD Testability] create DataDistributor::init() method
2022-07-13 17:03:12 -07:00
Josh Slocum
b85fbaef52
Merge pull request #7395 from sfc-gh-jslocum/bg_file_chunking
Chunked Snapshot Files
2022-07-13 17:22:34 -05:00
A.J. Beamon
91ccdbcb03 Use a more standard encoding in the JSON metadata reported for tenants. 2022-07-13 15:22:19 -07:00
Xiaoxi Wang
c3aca3f28c add comment 2022-07-13 15:08:02 -07:00
Josh Slocum
e32e726b9f
Merge pull request #7583 from sfc-gh-jslocum/granule_merging_improvements
Granule merging improvements
2022-07-13 16:04:22 -05:00
Xiaoxi Wang
0e488ee16b Merge branch 'main' of https://github.com/apple/foundationdb into feature/dd-refactor-incremental 2022-07-13 12:53:21 -07:00
Xiaoxi Wang
1592f6ad34 create DataDistributor::init() method 2022-07-13 12:53:04 -07:00
Xiaoxi Wang
a2062df220
Merge pull request #7573 from sfc-gh-xwang/feature/dd-refactor-incremental
move getInitialDataDistribution to DDTxnProcessor
2022-07-13 10:00:38 -07:00
Xiaoxi Wang
e0984091f3
Merge pull request #7494 from sfc-gh-xwang/bug/cycle-issue7423
Extend transaction window in Cycle workload if in simulation the transaction_too_old keep appearing
2022-07-13 09:25:16 -07:00
Josh Slocum
b9437e9a8d BW re-sending merge candidate if manager changes 2022-07-13 10:39:55 -05:00
Josh Slocum
0720b358ff comments and bug fix 2022-07-13 08:01:26 -05:00
Josh Slocum
b2a96b64e6 refactoring one function into two 2022-07-13 07:49:19 -05:00