334 Commits

Author SHA1 Message Date
Chaoguang Lin
879eb98dec Rename registerSpecialKeySpaceModule to registerSpecialKeysImpl 2022-04-21 12:54:34 -07:00
Chaoguang Lin
c0264a8522 Remove the client profiling special keys and update related documentations 2022-04-18 17:54:50 -07:00
Dan Lambright
1b3b4166c6
Merge branch 'main' into vv 2022-04-08 17:18:13 -04:00
Josh Slocum
6276cebad9
Blob integration (#6808)
* Fixing leaked stream with explicit notify failed before destructor

* better logic to prevent races in change feed fetching

* Found new race that makes assert incorrect

* handle server overloaded in initial read from fdb

* Handling more blob error types in granule retry

* Fixing rollback metadata problem, added better debugging

* Fixing version race when fetching change feed metadata

* Better racing split request handling

* fixing assert

* Handle change feed popped check in the blob worker

* fix: do not use a RYW transaction for a versionstamp because of randomize API version (#6768)

* more merge conflict issues

* Change feed destroy fixes

* Fixing change feed destroy and move race

* Check error condition in BG file req

* Using relative endpoints for blob worker interface

* Fixing bug in previous fix

* More destroy and move race fixes

* Don't update empty version on destroy in case it gets rolled back. moved() and removing will take care of ensuring it is not read

* Bug fix (#6796)

* fix: do not use a RYW transaction for a versionstamp because of randomize API version

* fix: if the initialSnapshotVersion was pruned, granule history was incorrect

* added a way to compress null bytes in printable()

* Fixing durability issue with moving and destroying change feeds

* Adding fix for not fully deleting files for a granule that child granules need to re-snapshot

* More destroy and move races

* Fixing change feed destroy and pop races

* Renaming bg prune to purge, and adding a C api and unit test for it

* more cleanup

* review comments

* Observability for granule purging

* better handling for change feed not registered

* Fixed purging bugs (#6815)

* fix: do not use a RYW transaction for a versionstamp because of randomize API version

* fix: if the initialSnapshotVersion was pruned, granule history was incorrect

* added a way to compress null bytes in printable()

* fixed a few purging bugs

Co-authored-by: Evan Tschannen <evan.tschannen@snowflake.com>
2022-04-08 14:15:25 -07:00
Dan Lambright
5bdc525353
Merge branch 'main' into vv 2022-04-08 13:16:04 -04:00
Jingyu Zhou
4fd414a8ed Merge branch 'main' into vv
Fix Conflicts:
	fdbclient/NativeAPI.actor.cpp
2022-03-31 09:38:36 -07:00
Jingyu Zhou
cfcf0f152c Merge branch 'main-4a085fc84' into vv
Fix Conflicts:
	fdbclient/NativeAPI.actor.cpp
	fdbserver/ClusterRecovery.actor.cpp
	fdbserver/MasterInterface.h
	fdbserver/masterserver.actor.cpp
	flow/error_definitions.h
2022-03-30 22:28:06 -07:00
Jingyu Zhou
b34f4052cd Merge branch 'main-f28dfc12b' into vv
Fix Conflicts:
	fdbclient/MultiVersionTransaction.actor.cpp
	fdbclient/NativeAPI.actor.cpp
	fdbclient/NativeAPI.actor.h
	fdbserver/storageserver.actor.cpp
2022-03-30 21:01:25 -07:00
Jingyu Zhou
00b57d4cce Merge branch 'main-67eba5ec7' into vv
Fix Conflicts:
	fdbclient/CommitProxyInterface.h
	fdbclient/NativeAPI.actor.cpp
	fdbclient/StorageServerInterface.h
	fdbserver/CommitProxyServer.actor.cpp
	fdbserver/storageserver.actor.cpp
2022-03-30 20:05:55 -07:00
Jingyu Zhou
e9659b5dd4 Merge branch 'master-PR-6500' into vv
Fix Conflicts:
	fdbclient/CommitProxyInterface.h
	fdbclient/NativeAPI.actor.cpp
	fdbserver/masterserver.actor.cpp
2022-03-30 14:53:49 -07:00
Jon Fu
f266695a76 clear shared state map entries on version upgrade and use currentProtocolVersion instead of piping version through function parameters 2022-03-30 15:28:55 -04:00
Jon Fu
9d0a520fe0 add ProtocolVersion to shared state and state cluster map 2022-03-30 13:28:06 -04:00
Jon Fu
53d252fda0 Merge branch 'main' of github.com:apple/foundationdb into jfu-grv-cache-multi-threaded 2022-03-29 17:23:08 -04:00
Jon Fu
bdbe78e48f address code review comments 2022-03-29 17:20:22 -04:00
Josh Slocum
b92c6ea92c granule parallelism, metrics, and other improvements to bg client bindings 2022-03-24 13:44:09 -05:00
Jon Fu
7c3151142e Merge branch 'main' of github.com:apple/foundationdb into jfu-grv-cache-multi-threaded 2022-03-23 16:41:33 -04:00
Jon Fu
4466e00716 Fix multithreaded GRV cache code and enable grv caching in multithreaded ctest suite 2022-03-23 16:09:12 -04:00
Josh Slocum
f27475e2f4 Merge branch 'main' into blob_integration 2022-03-22 11:41:58 -05:00
sfc-gh-tclinkenbeard
a71099471b Update copyright header dates 2022-03-21 13:36:23 -07:00
Josh Slocum
37e7c80f26 Merge branch 'main' into blob_integration 2022-03-17 18:45:42 -05:00
A.J. Beamon
05495908b8 Implement some tenant tests 2022-03-17 12:10:18 -07:00
Trevor Clinkenbeard
765e018afb
Merge pull request #6580 from sfc-gh-tclinkenbeard/const-smoother
Mark `Smoother::smooth*` methods `const`
2022-03-16 13:43:25 -07:00
Jon Fu
9731d96797 Merge branch 'main' of github.com:apple/foundationdb into jfu-grv-cache-multi-threaded 2022-03-16 14:32:33 -04:00
Jon Fu
7ab205391b Revert "remove all multi-version client code related to grv cache"
This reverts commit 6d05ecffc21d8497b4f753d11b17beb6102f32ab.
2022-03-16 14:26:36 -04:00
Jingyu Zhou
2acc222d9a Detect stale GRV proxy replies
Each GetReadVersionReply now has the GRV proxy ID that sent it so that the
client can detect old GRV proxy's replies and avoid apply stale version vector
changes.
2022-03-15 14:23:07 -07:00
Josh Slocum
ce5b567971 Cleanup the cleanup 2022-03-15 15:02:28 -05:00
A.J. Beamon
0536e56725 The tenant cache uses a simpler eviction policy. Some various improvement around the use of futures. Add documentation for C API and fix inaccurate statement in special keys documentation. 2022-03-15 09:23:30 -07:00
A.J. Beamon
68069c9784 Undo some unused changes. Fix tenant cache eviction. 2022-03-15 09:23:30 -07:00
A.J. Beamon
ecccfd0868 Add cache invalidation to tenant cache. Send tenant ID along with tenant name in requests to validate that the tenant hasn't changed. Fix a few bugs. 2022-03-15 09:23:30 -07:00
A.J. Beamon
2a21126028 Don't apply read prefixes on the client. Cache tenant data locally. 2022-03-15 09:23:30 -07:00
A.J. Beamon
c635dcd3ad Add tenant support in the FDB native client 2022-03-15 09:21:27 -07:00
Jingyu Zhou
800478ad5e Add a new client error code stale_version_vector 2022-03-14 20:50:12 -07:00
Josh Slocum
98eeaac1b7 More debugging and fixing a stuck merge cursor 2022-03-14 11:57:56 -05:00
sfc-gh-tclinkenbeard
9aca5f8011 Mark several methods const 2022-03-11 00:47:35 -04:00
Tao Lin
e2c7c30faf
GetMappedRange support serializable & check RYW & continuation (#6181) 2022-03-10 10:05:44 -08:00
Josh Slocum
b21d0943b9 client-focused cleanup 2022-03-09 10:01:25 -06:00
Josh Slocum
e71b3533f9 Merge branch 'main' into blob_integration 2022-03-09 08:59:56 -06:00
A.J. Beamon
cdebda35ab
Merge pull request #5725 from sfc-gh-jfu/jfu-grv-cache
Add transaction option for clients to use cached read versions
2022-03-04 09:17:27 -08:00
Zhe Wu
9c99c59b0c Fix get key location overloading proxies 2022-02-24 16:35:30 -08:00
Josh Slocum
e5b4fb3d80 Adding explicit popVersion to change feed reply, like tlogs have 2022-02-24 15:20:18 -06:00
Dan Lambright
8cc9a5af1a Rebase 02/23 2022-02-23 14:23:28 -05:00
Josh Slocum
38a75a8b89 Merge branch 'main' into blob_integration 2022-02-17 17:47:38 -06:00
Jon Fu
d399daebed Merge branch 'main' of github.com:apple/foundationdb into jfu-grv-cache 2022-02-15 15:09:40 -05:00
Vaidas Gasiunas
092b5cee4b MVC2.0: Rollback added code 2022-02-14 13:50:42 -08:00
Jon Fu
a63d218e9d simplify test workload and adjust ratekeeper throttling strategy 2022-02-11 16:41:14 -05:00
Jon Fu
458e708272 addressed code review comments: renamed variables, small functional changes, style changes 2022-02-10 16:17:54 -05:00
Josh Slocum
ddfc301d74 Improving memory footprint of change feeds and making it configurable 2022-02-04 16:41:25 -06:00
Jon Fu
6d05ecffc2 remove all multi-version client code related to grv cache 2022-02-02 14:39:17 -05:00
Jon Fu
d8e7fea421 clean up some comments and debug changes 2022-02-02 14:03:32 -05:00
Josh Slocum
ac1fd056dd Added change feed popped read guard for blob workers 2022-01-28 10:45:33 -06:00