201 Commits

Author SHA1 Message Date
Josh Slocum
6276cebad9
Blob integration (#6808)
* Fixing leaked stream with explicit notify failed before destructor

* better logic to prevent races in change feed fetching

* Found new race that makes assert incorrect

* handle server overloaded in initial read from fdb

* Handling more blob error types in granule retry

* Fixing rollback metadata problem, added better debugging

* Fixing version race when fetching change feed metadata

* Better racing split request handling

* fixing assert

* Handle change feed popped check in the blob worker

* fix: do not use a RYW transaction for a versionstamp because of randomize API version (#6768)

* more merge conflict issues

* Change feed destroy fixes

* Fixing change feed destroy and move race

* Check error condition in BG file req

* Using relative endpoints for blob worker interface

* Fixing bug in previous fix

* More destroy and move race fixes

* Don't update empty version on destroy in case it gets rolled back. moved() and removing will take care of ensuring it is not read

* Bug fix (#6796)

* fix: do not use a RYW transaction for a versionstamp because of randomize API version

* fix: if the initialSnapshotVersion was pruned, granule history was incorrect

* added a way to compress null bytes in printable()

* Fixing durability issue with moving and destroying change feeds

* Adding fix for not fully deleting files for a granule that child granules need to re-snapshot

* More destroy and move races

* Fixing change feed destroy and pop races

* Renaming bg prune to purge, and adding a C api and unit test for it

* more cleanup

* review comments

* Observability for granule purging

* better handling for change feed not registered

* Fixed purging bugs (#6815)

* fix: do not use a RYW transaction for a versionstamp because of randomize API version

* fix: if the initialSnapshotVersion was pruned, granule history was incorrect

* added a way to compress null bytes in printable()

* fixed a few purging bugs

Co-authored-by: Evan Tschannen <evan.tschannen@snowflake.com>
2022-04-08 14:15:25 -07:00
Josh Slocum
aaaf42525a misc bg operational fixes and improvements 2022-04-05 12:26:00 -05:00
Evan Tschannen
c168840b54 blob workers properly destroy change feeds when they are no longer needed 2022-04-05 11:02:32 -05:00
Josh Slocum
7fc6dfa6c5 Adding useful debugging trace events 2022-03-29 14:48:28 -05:00
Jingyu Zhou
da0673ccce
Merge pull request #6705 from RenxuanW/another
Add proxy option to backup and restore params.
2022-03-29 11:36:13 -07:00
Josh Slocum
61474d5d54 Future-proof blob granules with full file size 2022-03-29 08:06:07 -05:00
Renxuan Wang
0a332ee1c1 Add proxy option to backup and restore params. 2022-03-28 17:10:49 -07:00
Josh Slocum
3cd1e5599e Cleanup 2022-03-24 17:15:11 -05:00
Josh Slocum
989dd8d7eb Finished implementing beginVersion 2022-03-24 17:15:11 -05:00
Josh Slocum
1b1182f414 Blob Worker side of beginVersion done, added unit test 2022-03-24 17:15:11 -05:00
Josh Slocum
c72197122d bump 2022-03-23 08:07:49 -05:00
Josh Slocum
4e20da76fb bump 2022-03-22 15:37:16 -05:00
Josh Slocum
917c7acca5 Using desiredOldest for when change feed metadata is safe 2022-03-21 22:21:40 -05:00
Josh Slocum
f4c2a789c6 Explicitly cancelling file updater future 2022-03-21 11:47:40 -05:00
Josh Slocum
f7f2c9e979 Fixing BW erroneously asking BM to split even after it recieved a reply to continue without splitting 2022-03-20 08:13:38 -05:00
Josh Slocum
327ae95819 fix warnings 2022-03-18 11:19:00 -05:00
Josh Slocum
37e7c80f26 Merge branch 'main' into blob_integration 2022-03-17 18:45:42 -05:00
Josh Slocum
051d09159f Changing pops to asynchronous to remove performance bottleneck 2022-03-17 14:39:52 -05:00
Evan Tschannen
2c88a189a9 more pruning bug fixes 2022-03-15 13:34:59 -07:00
Evan Tschannen
d46e551f11 merge 2022-03-15 13:09:10 -07:00
Josh Slocum
ce5b567971 Cleanup the cleanup 2022-03-15 15:02:28 -05:00
Josh Slocum
b27d6bbfab Have BW read manager lock on startup to avoid races with old BM 2022-03-15 10:10:29 -05:00
Josh Slocum
8c35ceb684 switching bg file keys to sort on version then type, and using that to fix race where 2 blob workers try to split the same granule at different versions 2022-03-14 17:38:31 -05:00
sfc-gh-tclinkenbeard
8dcac2f76d Fix typos 2022-03-13 10:02:11 -03:00
Josh Slocum
479ac313ca Fixing conflict handling in BM recovery 2022-03-10 16:35:08 -06:00
Josh Slocum
4f88e00404 Adding change feed pop to set of expected granule errors 2022-03-10 14:37:32 -06:00
Josh Slocum
903f7adbc4 more cleanup 2022-03-10 13:53:06 -06:00
Josh Slocum
4b254d259c Ensuring BM split retry is idempotent 2022-03-10 11:54:57 -06:00
Josh Slocum
c8c97e0256 Blob Worker focused cleanup 2022-03-10 09:55:23 -06:00
Josh Slocum
1f964ac085 BM focused cleanup 2022-03-09 15:04:17 -06:00
Josh Slocum
9dbb6d216e SS-focused cleanup 2022-03-09 10:39:04 -06:00
Josh Slocum
b21d0943b9 client-focused cleanup 2022-03-09 10:01:25 -06:00
Josh Slocum
f230ac0e41 Fixing more merge issues 2022-03-09 09:13:03 -06:00
Josh Slocum
e71b3533f9 Merge branch 'main' into blob_integration 2022-03-09 08:59:56 -06:00
Jingyu Zhou
1a5bf25b5c Update code base to use fmt 8.1.1 2022-03-04 15:52:06 -08:00
Evan Tschannen
976375466f fixed more pruning bugs 2022-03-02 17:03:37 -08:00
Josh Slocum
e0578f24d2 Fixes for range streaming BG snapshot 2022-03-01 12:45:58 -06:00
Evan Tschannen
6f1fb71949 Fixed a number of different pruning problems 2022-03-01 10:13:20 -08:00
Evan Tschannen
8f8987ad67 Merge branch 'blob_integration' of https://github.com/apple/foundationdb into blob_integration 2022-03-01 10:12:36 -08:00
Josh Slocum
ce176e846e Switching initial granule snapshot to use range stream 2022-03-01 11:20:44 -06:00
Josh Slocum
3090fb174c Tweaks to granule initial snapshot 2022-03-01 08:52:38 -06:00
Josh Slocum
c27204d401 more BM split race fixes 2022-03-01 07:18:43 -06:00
Josh Slocum
ed9f95b299 simplifying blob manager recovery by always treating the DB granule mapping as the source of truth, and persisting all boundary changes there BEFORE updating workers 2022-02-25 14:46:14 -06:00
Evan Tschannen
940f26cb0c Merge branch 'blob_integration' of https://github.com/apple/foundationdb into blob_integration 2022-02-25 11:00:13 -08:00
Josh Slocum
e5b4fb3d80 Adding explicit popVersion to change feed reply, like tlogs have 2022-02-24 15:20:18 -06:00
A.J. Beamon
250a88e682 Enforce that trace event suppression calls happen first when using trace event call chaining. Fix various instances where we weren't following this requirement. 2022-02-24 12:25:52 -08:00
Josh Slocum
2701e44564 Fixing error handling and race for blob worker status stream to manager 2022-02-23 15:55:16 -06:00
Evan Tschannen
330b2b48ec improved file cleanup execution and testing 2022-02-22 12:00:09 -08:00
Josh Slocum
bfb1902a7a Fixing logic for when granule split is valid 2022-02-16 11:34:22 -06:00
Josh Slocum
abdbc5aafb Fixing granule opening after not deleting granule locks 2022-02-14 13:20:54 -06:00