Zhe Wang 0f6fa090ce
Bulkload Engine Support General Storage Engine and Fix BulkLoad Bugs (#11898)
* bulkload support general engine and fix bugs

* add comments

* improve test coverage and fix bug

* nits and address comments

* nit

* nits

* fix data inconsistency bug due to bulkload metadata

* fix ss bulkload task metadata bugs

* nit and fix CI issue

* fix bugs of restore ss bulkload metadata

* use ssBulkLoadMetadata for fetchKey and general kv engine

* cleanup bulkload file for fetchkey

* fix CI issue

* fix simulation stuck due to repeated re-recruitment of unfit dd

* randomly do available space check when finding the dest team for bulkload in simulation

* address conflict

* code clean up

* update BulkDumping.toml same to BulkLoading.toml

* consolidate ss fetchkey and fetchshard failed to read bulkload task metadata

* fix DD bulkload job busy loop bug which causes segfault and test terminate unexpectedly in joshua test

* nit

* fix ss busy loop for bulkload in fetchkey

* use sqlite for bulkload ctest

* fix bulkload ctest stuck issue due to merge and change storage engine to ssd

* fix comments for CC recruit DD

* address comments

* address comments

* add comments

* fix ci format issue

* address comments

* add comments
2025-02-06 12:04:13 -08:00
..