78 Commits

Author SHA1 Message Date
Dan Lambright
1b3b4166c6
Merge branch 'main' into vv 2022-04-08 17:18:13 -04:00
Josh Slocum
6276cebad9
Blob integration (#6808)
* Fixing leaked stream with explicit notify failed before destructor

* better logic to prevent races in change feed fetching

* Found new race that makes assert incorrect

* handle server overloaded in initial read from fdb

* Handling more blob error types in granule retry

* Fixing rollback metadata problem, added better debugging

* Fixing version race when fetching change feed metadata

* Better racing split request handling

* fixing assert

* Handle change feed popped check in the blob worker

* fix: do not use a RYW transaction for a versionstamp because of randomize API version (#6768)

* more merge conflict issues

* Change feed destroy fixes

* Fixing change feed destroy and move race

* Check error condition in BG file req

* Using relative endpoints for blob worker interface

* Fixing bug in previous fix

* More destroy and move race fixes

* Don't update empty version on destroy in case it gets rolled back. moved() and removing will take care of ensuring it is not read

* Bug fix (#6796)

* fix: do not use a RYW transaction for a versionstamp because of randomize API version

* fix: if the initialSnapshotVersion was pruned, granule history was incorrect

* added a way to compress null bytes in printable()

* Fixing durability issue with moving and destroying change feeds

* Adding fix for not fully deleting files for a granule that child granules need to re-snapshot

* More destroy and move races

* Fixing change feed destroy and pop races

* Renaming bg prune to purge, and adding a C api and unit test for it

* more cleanup

* review comments

* Observability for granule purging

* better handling for change feed not registered

* Fixed purging bugs (#6815)

* fix: do not use a RYW transaction for a versionstamp because of randomize API version

* fix: if the initialSnapshotVersion was pruned, granule history was incorrect

* added a way to compress null bytes in printable()

* fixed a few purging bugs

Co-authored-by: Evan Tschannen <evan.tschannen@snowflake.com>
2022-04-08 14:15:25 -07:00
Dan Lambright
5bdc525353
Merge branch 'main' into vv 2022-04-08 13:16:04 -04:00
Jingyu Zhou
cfcf0f152c Merge branch 'main-4a085fc84' into vv
Fix Conflicts:
	fdbclient/NativeAPI.actor.cpp
	fdbserver/ClusterRecovery.actor.cpp
	fdbserver/MasterInterface.h
	fdbserver/masterserver.actor.cpp
	flow/error_definitions.h
2022-03-30 22:28:06 -07:00
Jingyu Zhou
b34f4052cd Merge branch 'main-f28dfc12b' into vv
Fix Conflicts:
	fdbclient/MultiVersionTransaction.actor.cpp
	fdbclient/NativeAPI.actor.cpp
	fdbclient/NativeAPI.actor.h
	fdbserver/storageserver.actor.cpp
2022-03-30 21:01:25 -07:00
Jingyu Zhou
00b57d4cce Merge branch 'main-67eba5ec7' into vv
Fix Conflicts:
	fdbclient/CommitProxyInterface.h
	fdbclient/NativeAPI.actor.cpp
	fdbclient/StorageServerInterface.h
	fdbserver/CommitProxyServer.actor.cpp
	fdbserver/storageserver.actor.cpp
2022-03-30 20:05:55 -07:00
Jon Fu
f266695a76 clear shared state map entries on version upgrade and use currentProtocolVersion instead of piping version through function parameters 2022-03-30 15:28:55 -04:00
Jon Fu
9d0a520fe0 add ProtocolVersion to shared state and state cluster map 2022-03-30 13:28:06 -04:00
Jon Fu
bdbe78e48f address code review comments 2022-03-29 17:20:22 -04:00
Jon Fu
7c3151142e Merge branch 'main' of github.com:apple/foundationdb into jfu-grv-cache-multi-threaded 2022-03-23 16:41:33 -04:00
Jon Fu
4466e00716 Fix multithreaded GRV cache code and enable grv caching in multithreaded ctest suite 2022-03-23 16:09:12 -04:00
sfc-gh-tclinkenbeard
a71099471b Update copyright header dates 2022-03-21 13:36:23 -07:00
Jon Fu
9731d96797 Merge branch 'main' of github.com:apple/foundationdb into jfu-grv-cache-multi-threaded 2022-03-16 14:32:33 -04:00
Jon Fu
7ab205391b Revert "remove all multi-version client code related to grv cache"
This reverts commit 6d05ecffc21d8497b4f753d11b17beb6102f32ab.
2022-03-16 14:26:36 -04:00
A.J. Beamon
8bc2b283e1 Create a special keys API to create, delete, and read tenants. Remove the C API to create/delete tenants. 2022-03-15 09:23:30 -07:00
A.J. Beamon
c89b1e3537 Use TenantName type instead of StringRef in various places. Add a function on transactions to get the tenant name being used for the transaction. 2022-03-15 09:21:27 -07:00
A.J. Beamon
c635dcd3ad Add tenant support in the FDB native client 2022-03-15 09:21:27 -07:00
Tao Lin
e2c7c30faf
GetMappedRange support serializable & check RYW & continuation (#6181) 2022-03-10 10:05:44 -08:00
Jingyu Zhou
415970a47b Fix compiler warnings
Initialization order issue.
2022-02-07 11:49:56 -08:00
Dan Lambright
9544379cdf rebase 2022-01-20 11:12:33 -05:00
Josh Slocum
7f4fcc8c2c Added FDBResult and made readBlobGranules use it 2021-12-01 16:22:05 -06:00
Josh Slocum
329091e14f Merge branch 'master' into bg_bindings 2021-11-11 10:13:37 -06:00
Josh Slocum
b8ac4213a1 Switched BG APIs to transaction instead of database 2021-11-11 08:59:06 -06:00
Tao Lin
fdb3b72e35 Introduce GetRangeAndFlatMap to push computations down to FDB
Re-introduce #5609
2021-11-09 13:52:28 -08:00
Tao Lin
586cc3b102
Revert "Introduce GetRangeAndFlatMap to push computations down to FDB" 2021-11-04 08:46:56 -07:00
Tao Lin
6c98e35893 Rename Hop to FlatMap 2021-11-03 13:32:01 -07:00
Tao Lin
0853661d13 Introduce getRangeAndHop to push computations down to FDB 2021-11-03 13:21:16 -07:00
Josh Slocum
d6a31078fe C API for blob granules 2021-11-02 10:01:23 -05:00
Sreenath Bodagala
2aa3b44d4e Merge remote-tracking branch 'apple-upstream/master' into version-vector-prototype
- Conflicts:
	fdbserver/LogSystem.h
	fdbserver/LogSystemConfig.h
	fdbserver/TagPartitionedLogSystem.actor.cpp

- Files modified during merge:

modified:   fdbserver/LogSystem.cpp
modified:   fdbserver/LogSystemConfig.cpp
2021-09-17 19:36:18 +00:00
Sreenath Bodagala
35621a255e - Address review comments 2021-09-16 12:00:26 -04:00
Sreenath Bodagala
ffbdc21ca8 - Address review comments 2021-09-16 12:00:26 -04:00
Sreenath Bodagala
852fc96200 Address simulation test failures caused by:
- Assertion failures in MoveKeys.actor.cpp
- Wrong results returned by getRange()

Changes:

DatabaseContext.h, NativeAPI.actor.[h,cpp]:
- Introduce a new flag, TransactionInfo::readVersionObtainedFromGrvProxy.
- Set this flag to true by default, and clear it when the read version of a
transaction is explicitly set (by using setVersion()).
- Modify getLatestCommitVersions() to not populate "latestCommitVersions" if
this flag is not set. (This will cause storage server to read at the specified
read version.)

- Modify getRange() actor to always use the specified version as the read
version (except when the specified version is latestVersion).

- Modify waitForCommittedVersion(), getRawVersion(), and getConsistentReadVersion()
to update local version vector cache after receiving GetReadVersionReply.

IClientApi.h, IConfigTransaction.h, ISingleThreadTransaction.h,
MultiVersionTransaction[.actor].[h,cpp], ThreadSafeTransaction.[h,cpp],
ApiWorkload.h:
- Add methods to get the spanID of a transaction and also the version vector
cached in a transaction. (Likely to be useful for debugging simulation test
failures.)

VersionVector.h:
- Update "maxVersion" when populating/applying a delta. (Note that empty
mutation messages only update VersionVector::maxVersion.)

BackupWorker.actor.cpp:
- Update local version vector cache after receiving GetReadVersionReply message.

Status.actor.cpp:
- Update local version vector cache and
TransactionInfo::info.readVersionObtainedFromGrvProxy after setting the
read version.
2021-09-16 12:00:26 -04:00
Chaoguang Lin
af5643c998 Fix the issue in CheckStatus where it stuck as we cannot talk to any of the coordinators 2021-09-08 10:05:13 -07:00
FDB Formatster
2c788c233d apply clang-format to *.c, *.cpp, *.h, *.hpp files 2021-08-27 17:07:47 -07:00
Chaoguang Lin
6ef49fe91e Update comments, remove headers 2021-08-11 23:19:39 -07:00
Chaoguang Lin
be796a62a0 Update comments 2021-08-11 18:39:47 -07:00
Chaoguang Lin
bf0d0598dc Refactor all ThrottleAPI functions using template and remove duplicate copies 2021-08-11 23:59:25 +00:00
Chaoguang Lin
618c72f7e2 Draft to use template to rewrite ThrottleApi 2021-08-11 01:07:36 +00:00
Chaoguang Lin
7da5004690 Merge branch 'master' of github.com:apple/foundationdb into refactor-fdbcli 2021-05-06 15:44:26 -07:00
sfc-gh-tclinkenbeard
5c2d7b6080 Create RangeResult type alias 2021-05-03 13:14:16 -07:00
Chaoguang Lin
283b774c2f Merge branch 'master' of github.com:apple/foundationdb into refactor-fdbcli 2021-04-29 12:04:54 -07:00
A.J. Beamon
823873a9aa Address review comments:
Use nullptr instead of NULL
Use const& for a parameter
Add some comments
2021-04-26 14:39:27 -07:00
Chaoguang Lin
f0a236c544 Merge branch 'master' of github.com:apple/foundationdb into refactor-fdbcli 2021-04-21 15:36:11 -07:00
A.J. Beamon
b2d6930103 The multi-version client monitors the cluster's protocol version and only activates the client library that can connect. 2021-04-15 11:45:14 -07:00
A.J. Beamon
3ed0d614d2 Move fdb_get_server_protocol to be a function on the database object. Add an argument for expected_version that can be used to signal that the function shouldn't return unless the protocol version is different. 2021-04-14 12:50:30 -07:00
Chaoguang Lin
107f66e4e1 Merge branch 'master' of github.com:apple/foundationdb into refactor-fdbcli 2021-03-30 16:35:41 -07:00
Chaoguang Lin
35f9fe08a2 Remove unnecessary header in IClientApi.h 2021-03-23 14:44:14 -07:00
Nim Wijetunga
f0d79b3d86 Inital implementation of network busyness 2021-03-15 23:23:56 +00:00
FDB Formatster
df90cc89de apply clang-format to *.c, *.cpp, *.h, *.hpp files 2021-03-10 10:18:07 -08:00
Chaoguang Lin
0947b87b60 Update documentation 2021-02-08 18:47:49 -08:00