5997 Commits

Author SHA1 Message Date
Steve Atherton
a627df88c6 IO buffer lifetime fix, source buffer must stay alive until the file write completes. 2022-10-23 16:38:08 -07:00
Steve Atherton
344290fdf1 IO buffer lifetime fix, buf must stay alive until the file read completes. 2022-10-23 16:28:31 -07:00
Nim Wijetunga
7dacaed983
add domain name asserts (#8534) 2022-10-21 17:24:27 -07:00
neethuhaneesha
a1eb1d4a48
Rocksdb storage using single_key_deletes instead of deleterange on clearrange operation. (#8452) 2022-10-21 15:47:19 -07:00
Josh Slocum
9987da5a4f
Add blob granule benchmark to track stats by varying amount of deltas, and several simple perf improvements (#8533) 2022-10-21 17:05:51 -05:00
Josh Slocum
d7b88f203c
added repeatFromFiles that supports new file format (#8502) 2022-10-21 10:29:35 -05:00
Jon Fu
ddd9021469
Merge pull request #8495 from sfc-gh-jfu/tenant-restarting-tests
Add support for maintaining tenant state and default tenants in restarting tests
2022-10-20 21:11:33 -07:00
Jingyu Zhou
57fc39d5b7 Make decodeKVPairs a non-actor function 2022-10-20 19:34:44 -07:00
Jon Fu
60e76ef4a7 Merge branch 'main' of github.com:apple/foundationdb into tenant-restarting-tests 2022-10-20 17:57:22 -07:00
Jingyu Zhou
a8391caf23 Revert "Data loss protection v2" 2022-10-20 18:09:58 -05:00
Nim Wijetunga
2745168d72
Add Testing for LastTenantModification Field (#8509)
* add testing

* address pr comments
2022-10-20 13:38:38 -07:00
Jon Fu
3533cff94d process '_experimental' suffix in fromString helper 2022-10-20 13:38:13 -07:00
Jon Fu
c6998a7185 change tenantModes option to accept array of string 2022-10-20 11:44:03 -07:00
Jingyu Zhou
c357a054a7
Merge pull request #8472 from sfc-gh-ljoswiak/fixes/cluster-id-v2
Data loss protection v2
2022-10-19 12:49:05 -07:00
Josh Slocum
4ad6a8030b
Fixing type in constructor (#8486) 2022-10-19 12:21:23 -07:00
Jingyu Zhou
0872cbfb2f
Fix a test timeout (#8488)
* Fix a test timeout due to buggified knob MAX_WRITE_TRANSACTION_LIFE_VERSIONS

The buggified knob MAX_WRITE_TRANSACTION_LIFE_VERSIONS can be only 1M. In some
tests, this transaction always end up commitVersion - readVersion is a little
above 1M, thus always getting transaction_too_old error.

* Change MAX_COMMIT_BATCH_INTERVAL instead

So that the master may give out versions fast enough.

* Fix an assertion failure in a unit test

48125>>8 = 187, 48125 = 0xbbfd
48128>>8 = 188, 48128 = 0xbc00
So if 48125 is chosen as the index, 48128 changes the higher order byte.

48125 & 0xff7f = 47997 = 0xbb7d. Thus +5 won't change the higher order byte.
2022-10-19 09:45:56 -07:00
Ankita Kejriwal
52cd71f87d
Merge pull request #7917 from sfc-gh-akejriwal/monitorusage
Monitor storage bytes used by tenants
2022-10-18 23:15:48 -07:00
Lukas Joswiak
9c847a20e8 Rename cluster ID key
In FDB 7.1, this key was stored in the txnStateStore. In 7.2, it has
been moved to the database. This was causing protocol compatibility
issues during upgrades, so we need to rename the key.
2022-10-18 21:37:42 -07:00
Lukas Joswiak
7342672c11 Move cluster ID from txnStateStore to the database
The cluster ID is now stored in the database instead of in the
txnStateStore. The cluster controller will read it on boot and send it
to all processes to persist.
2022-10-18 21:37:42 -07:00
Lukas Joswiak
7f889c87e3 Fix simulation issue where process switch was ignored
The simulator tracks only active processes. Rebooted or killed processes
are removed from the list of processes, and only get added back when the
process is rebooted and starts up again. This causes a problem for the
`RebootProcessAndSwitch` kill type, which wants to simultaneously reboot
all machines in a cluster and change their cluster file. If a machine is
currently being rebooted, it will miss the reboot process and switch
command.

The fix is to add a check when a process is being started in simulation.
If the process has had its cluster file changed and the cluster is in a
state where all processes should have had their cluster files reverted
to the original value, the simulator will now send a
`RebootProcessAndSwitch` signal right when the process is started. This
will cause an extra reboot, but should correctly switch the process back
to its original, correct cluster file, allowing the cluster to fully
recover all clusters.

Note that the above issue should only affect simulation, due to how the
simulator tracks processes and handles kill signals.

This commit also adds a field to each process struct to determine
whether the process is being run in a DR cluster in the simulation run.
This is needed because simulation does not differentiate between
processes in different clusters (other than by the IP), and some
processes needed to switch clusters and some simply needed to be
rebooted.
2022-10-18 21:37:42 -07:00
Nim Wijetunga
d439bc1e6e
TenantEntryCache Watch Based Refresh (#8399)
* tenant modification changekey

* address pr comments

* change backup to use watch based tenant cache

* format

* address pr comments

* trigger build

* add todo

* check tenants disabled

* trigger build

* trigger build

* address pr comments

* address pr comments

* trigger build
2022-10-18 19:05:07 -07:00
Ankita Kejriwal
3d890ced9a Code cleanup 2022-10-18 15:53:22 -07:00
Jingyu Zhou
14d13475db
Merge pull request #8437 from sfc-gh-tclinkenbeard/add-quota-clear-command
Add fdbcli "quota clear" command
2022-10-18 15:11:05 -07:00
Ankita Kejriwal
fb014a4834 Merge branch 'main' of github.com:apple/foundationdb into monitorusage 2022-10-18 13:14:47 -07:00
Josh Slocum
89519343a7
adding new bg read metrics to client and refactoring them into a new trace event (#8493) 2022-10-18 14:13:35 -05:00
Jingyu Zhou
d45f7a088b
Merge pull request #8489 from halfprice/zhewu/debug-shard-size
Use consistencyScanInfoFetcher function and add a timeout
2022-10-17 20:28:18 -07:00
Jingyu Zhou
1b2fcdd4f6
Merge pull request #8053 from sfc-gh-akejriwal/getsizetenant
Make the storage metrics functions tenant aware
2022-10-17 20:02:52 -07:00
sfc-gh-tclinkenbeard
6bfb5c00fb Merge remote-tracking branch 'origin/main' into add-quota-clear-command 2022-10-17 17:02:42 -07:00
sfc-gh-tclinkenbeard
07cc152bf5 Merge remote-tracking branch 'origin/main' into add-quota-clear-command 2022-10-17 16:28:22 -07:00
sfc-gh-tclinkenbeard
7e4f380423 Add tests for fdbcli quota commands 2022-10-17 16:27:45 -07:00
Ankita Kejriwal
3eb41e1598 Make the WaitMetricRequest::TenantInfo an Optional. 2022-10-17 16:03:21 -07:00
Zhe Wu
5096cabc59 Fix schema error 2022-10-17 13:28:00 -07:00
Trevor Clinkenbeard
cf935a1915
Merge pull request #8444 from sfc-gh-tclinkenbeard/limit-gtt-size
Limit number of tags tracked by `GlobalTagThrottler`
2022-10-17 12:23:46 -07:00
Dennis Zhou
77e2e4a381 blob/c: tenant verifyBlobRange() c api 2022-10-17 11:42:26 -05:00
Dennis Zhou
1c90fbbc61 blob: tenant verifyBlobRange() native api 2022-10-17 11:42:08 -05:00
Dennis Zhou
9d38ddfabc blob/c: tenant listBlobbifiedRanges() c api 2022-10-17 11:42:07 -05:00
Dennis Zhou
4a33de31e9 blob: tenant listBlobbifiedRanges() native api 2022-10-17 11:41:45 -05:00
Dennis Zhou
a0246caebb blob/c: tenant (un)blobbify() c api 2022-10-17 11:41:44 -05:00
Dennis Zhou
120b38a926 blob: tenant (un)blobbify() native api 2022-10-17 11:41:12 -05:00
Dennis Zhou
0ee0ce62f0 blob: blobGranuleGetTenantEntry() for tenant and database calls
Currently, blobGranuleGetTenantEntry() expects to get the TenantName
through the transaction. Database calls cannot abide by this contract.
So enable the call to take in a TenantName and expect at least 1 to be
set.
2022-10-17 11:41:12 -05:00
Dennis Zhou
b2ce2bb72c blob: fix purge & wait tenant apis to use the ITenant interface 2022-10-17 11:41:12 -05:00
Dennis Zhou
622f2a79db api: add TenantBlobRangeApi feature flag and use for purge tenant api 2022-10-17 11:41:02 -05:00
sfc-gh-tclinkenbeard
a556f21ed2 Merge remote-tracking branch 'origin/main' into limit-gtt-size 2022-10-15 10:26:16 -07:00
Ankita Kejriwal
0f9da9d1ad Merge branch 'main' of github.com:apple/foundationdb into getsizetenant 2022-10-14 16:50:51 -07:00
sfc-gh-tclinkenbeard
4ab947a9b3 Merge remote-tracking branch 'origin/main' into limit-gtt-size 2022-10-14 15:59:57 -07:00
Josh Slocum
914dfd7438
REST Blob metadata kms connector (#8474)
* Changing RESTKMSConnector request handling to not be synchronous, as that would be a huge perf bottleneck

* Implementation of blob kms fetch

* cleanup

* review comments
2022-10-14 17:49:00 -05:00
Ankita Kejriwal
a24ab2d4fa Incorporate code review comments + workload code cleanup 2022-10-14 13:59:36 -07:00
sfc-gh-tclinkenbeard
3827b1ee62 Merge remote-tracking branch 'origin/main' into add-quota-clear-command 2022-10-14 11:55:45 -07:00
Josh Slocum
7cec0a5249
Blob metadata refresh (#8456)
* Adding EKP refresh of blob metadata

* Adding re-fetching blob metadata from BGTenantMap

* adding buggifies from code review comments
2022-10-14 08:17:50 -05:00
Ankita Kejriwal
854212fe94 Incorportate code review suggestions 2022-10-13 17:41:31 -07:00