511 Commits

Author SHA1 Message Date
Xiaoge Su
0a60142160 Extract ProcessInfo, MachineInfo, KillType out from ISimulator 2023-01-24 14:48:42 -08:00
Xiaoge Su
50de69c897 Extract IConnection and NetworkAddress out from network.h 2023-01-24 14:48:31 -08:00
Yi Wu
845cc62a39
Redwood: fix tree height overgrowth with per-tenant encryption (#9020)
* Fix Redwood tree height overgrowth when EaR and tenant page split are enabled, by removing the buildNewSubtree() logic.
* Fixing incorrect page upper bound for the last page created by writePages() without the buildNewSubtree() logic.
* Enable tenant page split if encryption mode is domain-aware encryption.
* Related test fixes:
  - In simulation, pass encryption mode to storage/Redwood via knobs. This is a workaround to enable testing with Redwood encryption before we correctly pass the encryption mode via db config. Also temporarily disable tenant page split for restart tests.
  - Disable raw access in FuzzApiCorrectness test if domain-aware encryption is enabled, to avoid test timeout
  - Disable encryption for DrUpgradeRestart test, which is likely to fail due to a rare EKP deadlock issue blocking recovery. Will re-enable after the deadlock issue is fixed.
2023-01-06 15:56:37 -08:00
Nim Wijetunga
3a238d638a
Commit Proxy uses Encryption DB Config (#8911)
* add encryption db config

* address pr comments

* address pr comments

* add comments

* cp uses db config

* remove includes

* fix tests

* fix tests

* modify comment

* add encryption enabled method

* modify simulation

* Trigger Build

* change commit proxy encrypt mode fetching

* address pr comments

* address pr comments

* Trigger Build

* Trigger Build

* Trigger Build

* Trigger Build
2022-12-12 21:23:55 -08:00
sfc-gh-tclinkenbeard
68f14f017c Fix clang 15 compiler warnings 2022-12-08 13:59:37 -08:00
neethuhaneesha
f3b0cfb2cb Enabling backup tests with rocksdb storage engine. 2022-11-29 16:05:28 -08:00
Nim Wijetunga
97713cadff
Update Encryption Mode DB Config Values (#8839)
* add encryption db config

* address pr comments

* address pr comments

* add comments

* add comment

* modify check

* change condition

* address pr comments

* simplify check

* address pr comments
2022-11-22 16:33:59 -08:00
Markus Pilman
f105cb1809 Merge remote-tracking branch 'origin/main' into bugfixes/machines-attrition-debugging 2022-11-14 10:11:52 -07:00
Chaoguang Lin
9a643286cb
Make the hardcoded timeouts as parameters in test files; Also longRunningTest para to disable Simulation timeout restriction (#8765) 2022-11-09 16:57:51 -08:00
He Liu
d28187ca90
Do not set timeout for RocksDB reads in simulation. (#8716)
* Do not set timeout for RocksDB reads in simulation.

* Cleanup.

* Ignore validateStorage test.
2022-11-07 15:59:52 -08:00
Markus Pilman
f1fea14255 Merge remote-tracking branch 'origin/main' into bugfixes/machines-attrition-debugging 2022-11-01 13:51:35 -06:00
Lukas Joswiak
5ca2b89bdf Fix simulation issue where process switch was ignored
The simulator tracks only active processes. Rebooted or killed processes
are removed from the list of processes, and only get added back when the
process is rebooted and starts up again. This causes a problem for the
`RebootProcessAndSwitch` kill type, which wants to simultaneously reboot
all machines in a cluster and change their cluster file. If a machine is
currently being rebooted, it will miss the reboot process and switch
command.

The fix is to add a check when a process is being started in simulation.
If the process has had its cluster file changed and the cluster is in a
state where all processes should have had their cluster files reverted
to the original value, the simulator will now send a
`RebootProcessAndSwitch` signal right when the process is started. This
will cause an extra reboot, but should correctly switch the process back
to its original, correct cluster file, allowing the cluster to fully
recover all clusters.

Note that the above issue should only affect simulation, due to how the
simulator tracks processes and handles kill signals.

This commit also adds a field to each process struct to determine
whether the process is being run in a DR cluster in the simulation run.
This is needed because simulation does not differentiate between
processes in different clusters (other than by the IP), and some
processes needed to switch clusters and some simply needed to be
rebooted.
2022-10-27 13:56:13 -07:00
Lukas Joswiak
a72066be33 Add simulation support for changing the cluster file 2022-10-27 13:56:13 -07:00
Markus Pilman
e7b5b870a3 Merge remote-tracking branch 'origin/main' into bugfixes/machines-attrition-debugging 2022-10-24 15:24:36 -06:00
Jon Fu
60e76ef4a7 Merge branch 'main' of github.com:apple/foundationdb into tenant-restarting-tests 2022-10-20 17:57:22 -07:00
Jingyu Zhou
a8391caf23 Revert "Data loss protection v2" 2022-10-20 18:09:58 -05:00
Jon Fu
3533cff94d process '_experimental' suffix in fromString helper 2022-10-20 13:38:13 -07:00
Jon Fu
c6998a7185 change tenantModes option to accept array of string 2022-10-20 11:44:03 -07:00
Lukas Joswiak
c1e5dd662c Protect availability of DR coordinators in simulation
Otherwise, simulation may reboot and delete a majority of coordinator
processes in the DR cluster, causing `configuration_never_created`
errors.
2022-10-19 16:56:08 -07:00
Jon Fu
04f709c7cb Merge branch 'main' of github.com:apple/foundationdb into tenant-restarting-tests 2022-10-19 14:30:55 -07:00
Jon Fu
702fcd1274 remove allowDisablingTenants flag and add new downgrade test 2022-10-19 14:15:03 -07:00
Lukas Joswiak
7f889c87e3 Fix simulation issue where process switch was ignored
The simulator tracks only active processes. Rebooted or killed processes
are removed from the list of processes, and only get added back when the
process is rebooted and starts up again. This causes a problem for the
`RebootProcessAndSwitch` kill type, which wants to simultaneously reboot
all machines in a cluster and change their cluster file. If a machine is
currently being rebooted, it will miss the reboot process and switch
command.

The fix is to add a check when a process is being started in simulation.
If the process has had its cluster file changed and the cluster is in a
state where all processes should have had their cluster files reverted
to the original value, the simulator will now send a
`RebootProcessAndSwitch` signal right when the process is started. This
will cause an extra reboot, but should correctly switch the process back
to its original, correct cluster file, allowing the cluster to fully
recover all clusters.

Note that the above issue should only affect simulation, due to how the
simulator tracks processes and handles kill signals.

This commit also adds a field to each process struct to determine
whether the process is being run in a DR cluster in the simulation run.
This is needed because simulation does not differentiate between
processes in different clusters (other than by the IP), and some
processes needed to switch clusters and some simply needed to be
rebooted.
2022-10-18 21:37:42 -07:00
Lukas Joswiak
43854c5ac8 Add simulation support for changing the cluster file 2022-10-18 21:37:16 -07:00
Jon Fu
930801b483 allow tenantmodes option to specify an array of possible choices 2022-10-18 10:48:27 -07:00
Jon Fu
7969b08f52 change tenantmode type to int and rewrite some reboot logic 2022-10-17 09:38:11 -07:00
Jon Fu
ee0027ba76 remove tenantModeRequired flag and disable defaulttenant for cyclerestart test 2022-10-12 12:32:43 -07:00
Jon Fu
a3d42c6073 Merge branch 'main' of github.com:apple/foundationdb into tenant-restarting-tests 2022-10-12 11:59:27 -07:00
Jon Fu
056f938991 add tenantmode to testconfig and change precedence rules for some tenant options 2022-10-12 11:51:50 -07:00
Nim Wijetunga
8ca6397e97
Enable Backup Tests for Encrypted Snapshots (#8419)
* enable encrypted snapshot backup tests

* address pr comments

* remove tenant cache when decoding

* remove client knob
2022-10-12 10:45:30 -07:00
Jon Fu
2c0f936740 Merge branch 'main' of github.com:apple/foundationdb into tenant-restarting-tests 2022-10-11 12:41:03 -07:00
Jon Fu
3c7d17e2bb work in progress commit for defaulttenant in restarting tests 2022-10-11 12:40:36 -07:00
Markus Pilman
ea1325a552
Merge pull request #8319 from sfc-gh-tclinkenbeard/add-rare-code-probe-annotation
Add `rare` code probe decoration
2022-10-07 09:39:00 -06:00
Josh Slocum
dc917453c1
Targeted blob granules fault injection (#8231) 2022-10-05 13:44:38 -05:00
Hui Liu
1f87286424
Merge pull request #8366 from sfc-gh-huliu/fix7
fix killMachine - make sure we have at least 1 blob worker in a dc
2022-09-30 13:56:32 -07:00
Hui Liu
9db48eb10c fix killMachine - make sure we have at least 1 blob worker in a dc 2022-09-30 11:18:27 -07:00
Lukas Joswiak
020e28a63f Disable RocksDB storage engine in unit test TestConfig 2022-09-30 08:59:47 -07:00
Xiaoxi Wang
3578832c69 merge upstream/main 2022-09-28 14:01:50 -07:00
A.J. Beamon
c8bb15e8ee
Merge pull request #8216 from sfc-gh-ajbeamon/backup-support-for-tenants
Include tenant and metacluster metadata in backups
2022-09-28 12:00:24 -07:00
Nim Wijetunga
251afa3610
Set g_simulator fields for restart tests (#8318)
* fix tests

* fix tenant concurrency

* Set g_simulator variables for restart tests

* fix machine attrition for restarting
2022-09-28 10:09:26 -07:00
Xiaoxi Wang
3428fe467a Merge branch 'main' of https://github.com/apple/foundationdb into feature/dd-refactor-simple 2022-09-27 11:55:35 -07:00
Xiaoxi Wang
e476e814b2 merge upstream/main 2022-09-27 10:15:26 -07:00
Markus Pilman
437efc60f0
Merge pull request #8199 from sfc-gh-tclinkenbeard/improve-code-coverage
Add `probe::assert::RocksDB` code probe annotation
2022-09-26 16:09:08 -06:00
Markus Pilman
c503b2dc53 fix probe::assert::RocksDB 2022-09-26 15:03:35 -06:00
sfc-gh-tclinkenbeard
985958c260 Add rare code probe decoration 2022-09-25 15:28:32 -07:00
Dan Adkins
48e1b06d29 Add comments about the use of protectedAddresses. 2022-09-22 17:01:01 -07:00
A.J. Beamon
fda0d7223d Update backup to include system key ranges needed for tenants. Run simulated backup tests with tenants. 2022-09-22 10:00:13 -07:00
Xiaoxi Wang
7169a8b132 merge upstream/main 2022-09-15 21:27:26 -07:00
sfc-gh-tclinkenbeard
c0fd93869c Add probe::assert::rocksDB code probe annotation 2022-09-15 11:54:02 -07:00
sfc-gh-tclinkenbeard
82adc1e856 Make g_simulator a pointer 2022-09-15 09:00:33 -07:00
Xiaoxi Wang
1f1a66be39 merge upstream/main 2022-09-14 12:32:28 -07:00