191 Commits

Author SHA1 Message Date
Josh Slocum
48bcf6eaba
parameterizing min split size instead of hardcoding it to min_shard_bytes (#7404) 2022-06-30 12:28:22 -07:00
A.J. Beamon
26b35c07cd Refactor how tenant map entries are encoded and decoded. Add a specific version to the encoding that matches the version used when this feature was introduced (and the only version in which it was used). 2022-06-29 10:58:58 -07:00
A.J. Beamon
4bafe77889 Some refactoring of tenant code:
* extract tenant management into its own file and namespace
* rename the tenant management workload source file
* extract tenant special keys functions to a separate file
* extract some helper functions to GenericTransactionHelper.h
* convert StringRef -> TenantNameRef
* move some TenantMapEntry implementation into the cpp file
* add some helper functions to decode/encode a tenant mode
2022-06-27 12:32:49 -07:00
Markus Pilman
d35445a868 enforce include modularization in cmake 2022-06-23 14:37:35 -06:00
Josh Slocum
ae865027d6 fixes for blob metadata memory from valgrind 2022-06-07 13:50:11 -05:00
Josh Slocum
fcd20c479d addressing review comments 2022-06-03 08:36:07 -05:00
Josh Slocum
567b1d35f7 Full integration with BlobConnectionProvider for blob granules 2022-06-01 12:39:51 -05:00
Josh Slocum
85af0a25b2 Enabling BM to understand tenant boundaries, and changing BlobGranuleCorrectness to use tenants 2022-05-25 17:16:56 -05:00
Josh Slocum
6276cebad9
Blob integration (#6808)
* Fixing leaked stream with explicit notify failed before destructor

* better logic to prevent races in change feed fetching

* Found new race that makes assert incorrect

* handle server overloaded in initial read from fdb

* Handling more blob error types in granule retry

* Fixing rollback metadata problem, added better debugging

* Fixing version race when fetching change feed metadata

* Better racing split request handling

* fixing assert

* Handle change feed popped check in the blob worker

* fix: do not use a RYW transaction for a versionstamp because of randomize API version (#6768)

* more merge conflict issues

* Change feed destroy fixes

* Fixing change feed destroy and move race

* Check error condition in BG file req

* Using relative endpoints for blob worker interface

* Fixing bug in previous fix

* More destroy and move race fixes

* Don't update empty version on destroy in case it gets rolled back. moved() and removing will take care of ensuring it is not read

* Bug fix (#6796)

* fix: do not use a RYW transaction for a versionstamp because of randomize API version

* fix: if the initialSnapshotVersion was pruned, granule history was incorrect

* added a way to compress null bytes in printable()

* Fixing durability issue with moving and destroying change feeds

* Adding fix for not fully deleting files for a granule that child granules need to re-snapshot

* More destroy and move races

* Fixing change feed destroy and pop races

* Renaming bg prune to purge, and adding a C api and unit test for it

* more cleanup

* review comments

* Observability for granule purging

* better handling for change feed not registered

* Fixed purging bugs (#6815)

* fix: do not use a RYW transaction for a versionstamp because of randomize API version

* fix: if the initialSnapshotVersion was pruned, granule history was incorrect

* added a way to compress null bytes in printable()

* fixed a few purging bugs

Co-authored-by: Evan Tschannen <evan.tschannen@snowflake.com>
2022-04-08 14:15:25 -07:00
Josh Slocum
aaaf42525a misc bg operational fixes and improvements 2022-04-05 12:26:00 -05:00
Josh Slocum
cb918b9cef Added basic blob granule consistency check 2022-04-04 11:38:42 -05:00
Josh Slocum
377e252fcf
Better split sizing in blob manager (#6725) 2022-04-01 16:09:46 -07:00
Josh Slocum
7fc6dfa6c5 Adding useful debugging trace events 2022-03-29 14:48:28 -05:00
Jingyu Zhou
da0673ccce
Merge pull request #6705 from RenxuanW/another
Add proxy option to backup and restore params.
2022-03-29 11:36:13 -07:00
Josh Slocum
2f8e9d9de0 misc bg fixes 2022-03-29 08:05:52 -05:00
Renxuan Wang
0a332ee1c1 Add proxy option to backup and restore params. 2022-03-28 17:10:49 -07:00
Josh Slocum
1b1182f414 Blob Worker side of beginVersion done, added unit test 2022-03-24 17:15:11 -05:00
Josh Slocum
0f9e88572a Cleaning up debugging and fixing race in blob manager recruitment 2022-03-17 14:57:43 -05:00
Josh Slocum
a85b578d2b Added simulation validation that multiple blob maangers don't have same epoch 2022-03-17 14:57:43 -05:00
Evan Tschannen
2c88a189a9 more pruning bug fixes 2022-03-15 13:34:59 -07:00
Evan Tschannen
d46e551f11 merge 2022-03-15 13:09:10 -07:00
Josh Slocum
ce5b567971 Cleanup the cleanup 2022-03-15 15:02:28 -05:00
Josh Slocum
9d9b70abd4 Fixing another BM recovery and granule splitting race 2022-03-14 18:56:44 -05:00
Josh Slocum
8c35ceb684 switching bg file keys to sort on version then type, and using that to fix race where 2 blob workers try to split the same granule at different versions 2022-03-14 17:38:31 -05:00
Josh Slocum
1c14d4674d Fixing more BM races with granule splitting 2022-03-14 16:11:17 -05:00
Josh Slocum
3e278e8580 asserts for assign is empty is false if BM dies and rangeAssigner actor is dead 2022-03-14 15:53:09 -05:00
Josh Slocum
e7aa534cae Forgot to update worker mapping after picking worker 2022-03-14 15:04:43 -05:00
Josh Slocum
8d88b7ca41 making range assigner boundary changes fully synchronous to ensure no races with things reading assignments 2022-03-14 14:39:00 -05:00
Josh Slocum
67dd3d071e Fixing incorrect assert in BM split handling 2022-03-11 10:50:46 -06:00
Josh Slocum
479ac313ca Fixing conflict handling in BM recovery 2022-03-10 16:35:08 -06:00
Josh Slocum
903f7adbc4 more cleanup 2022-03-10 13:53:06 -06:00
Josh Slocum
c35e020da7 SS and BM test macros 2022-03-10 12:38:23 -06:00
Josh Slocum
ba35e7908a off by one 2022-03-10 12:11:44 -06:00
Josh Slocum
4b254d259c Ensuring BM split retry is idempotent 2022-03-10 11:54:57 -06:00
Josh Slocum
c8c97e0256 Blob Worker focused cleanup 2022-03-10 09:55:23 -06:00
Josh Slocum
1f964ac085 BM focused cleanup 2022-03-09 15:04:17 -06:00
Josh Slocum
9dbb6d216e SS-focused cleanup 2022-03-09 10:39:04 -06:00
Josh Slocum
b21d0943b9 client-focused cleanup 2022-03-09 10:01:25 -06:00
Josh Slocum
f230ac0e41 Fixing more merge issues 2022-03-09 09:13:03 -06:00
Josh Slocum
e71b3533f9 Merge branch 'main' into blob_integration 2022-03-09 08:59:56 -06:00
Jingyu Zhou
1a5bf25b5c Update code base to use fmt 8.1.1 2022-03-04 15:52:06 -08:00
Evan Tschannen
976375466f fixed more pruning bugs 2022-03-02 17:03:37 -08:00
Josh Slocum
e0578f24d2 Fixes for range streaming BG snapshot 2022-03-01 12:45:58 -06:00
Evan Tschannen
6f1fb71949 Fixed a number of different pruning problems 2022-03-01 10:13:20 -08:00
Evan Tschannen
8f8987ad67 Merge branch 'blob_integration' of https://github.com/apple/foundationdb into blob_integration 2022-03-01 10:12:36 -08:00
Josh Slocum
06f58f848d Improving change feed validation, and another popped while reading race fixed 2022-03-01 12:05:55 -06:00
Josh Slocum
c27204d401 more BM split race fixes 2022-03-01 07:18:43 -06:00
Josh Slocum
839edbf5ea moved granule split evaluation to not be specific to any individual blob worker, to simplify it and handle additional cases 2022-02-28 16:49:58 -06:00
Josh Slocum
642aab2b8b blob manager range assignment race fixes 2022-02-28 16:12:27 -06:00
Josh Slocum
a556582041 Fixes for new BM reecovery 2022-02-28 10:36:28 -06:00