117 Commits

Author SHA1 Message Date
Lukas Joswiak
75423a100c Move shared_ptr to save a reference increment and decrement 2022-06-22 14:50:17 -07:00
Lukas Joswiak
4f2b1807e4 Use shared_ptr to track initialization across threads 2022-06-22 14:50:17 -07:00
Lukas Joswiak
1b1a9d4df5 Initialize on main thread 2022-06-22 14:50:17 -07:00
Lukas Joswiak
88557d9169 Simplify function call when transaction is null 2022-06-22 14:50:17 -07:00
Lukas Joswiak
b80ed948f1 Check initialization status before accessing field 2022-06-22 14:50:17 -07:00
Lukas Joswiak
4c2bb0b44e Fix undefined behavior from accessing field of uninitialized object 2022-06-22 14:50:17 -07:00
Josh Slocum
34e6a8f942
Merge pull request #7399 from sfc-gh-jslocum/bg_tenant_improvements
Bg tenant improvements
2022-06-17 11:19:41 -05:00
Josh Slocum
b3597ef3a8 Added plumbing for tenant-aware purge granules 2022-06-16 13:04:34 -05:00
Vishesh Yadav
fd6f6eb06a
Merge pull request #7364 from sfc-gh-ljoswiak/fixes/unnecessary-transaction-initialization
Remove unnecessary ReadYourWritesTransaction initialization
2022-06-14 11:02:31 -07:00
Renxuan Wang
839af5701e
Fix bug in resolveTCPEndpoint() when hostname resolving fails. (#7375)
* Close trace file when error happens in runNetwork().

* Improve the bestCount algorithm in getLeader().

In the current implementation, if the nominees are [0,1], the chosen leader will be 1, which is an exception to other cases and our expectation that if 2 nominees have the same frequency, the one with lower id will be the leader.

* Remove unnecessary new statement.

stream will never be a nullptr.

* Move self->dnsCache out of lambda capture.

Member variables are not capture by default, thus, `host` and `service` are not captured. This somehow successfully compile, but throws std::bad_alloc or basic_string::_S_create exceptions when we call `host+":"+service` in dnsCache.remove().

* Revert unintended change.

* Address comments.
2022-06-13 20:24:30 -07:00
Lukas Joswiak
3b3ef49d40 Remove unnecessary transaction initialization
`ReadYourWritesTransaction` has memory allocated before being passed to
the main thread. This allows both threads to continue to access the
transaction object. Currently, the transaction gets allocated and
initialized on the foreign thread, and then re-initialized on the main
thread. This causes a bunch of extra, unnecessary work for each
`ReadYourWritesTransaction` where the temporary object gets destructed.

The fix is to only allocate memory for the `ReadYourWritesTransaction`
on the foreign thread, and then initialize it once on the main thread.
2022-06-10 16:53:19 -07:00
Vaidas Gasiunas
4d6ec7663f Close file at the end of runNetwork rather than in stopNetwork in order to avoid a race condition,
also in order to continue tracing the pending network thread activity.
Poll event throttler only in the network thread in order to avoid a race condition.
2022-05-31 13:59:09 +02:00
Hao Fu
853e6a346b
Optimization: support removing index conditionally (#7116) 2022-05-13 10:10:43 -07:00
Ray Jenkins
dc9e782ccc
OpenTelemetry Tracing Perf Fixes (#6990) 2022-05-02 14:56:51 -05:00
Ray Jenkins
1c5bf135d5
Revert "Migrate to OpenTelemetry tracing. (#6855)" (#6941)
This reverts commit 5df3bac110d9b5b88931b008b852433688bb7eb0.
2022-04-25 09:29:56 -05:00
Ray Jenkins
5df3bac110
Migrate to OpenTelemetry tracing. (#6855) 2022-04-20 09:26:37 -05:00
Dan Lambright
1b3b4166c6
Merge branch 'main' into vv 2022-04-08 17:18:13 -04:00
Josh Slocum
6276cebad9
Blob integration (#6808)
* Fixing leaked stream with explicit notify failed before destructor

* better logic to prevent races in change feed fetching

* Found new race that makes assert incorrect

* handle server overloaded in initial read from fdb

* Handling more blob error types in granule retry

* Fixing rollback metadata problem, added better debugging

* Fixing version race when fetching change feed metadata

* Better racing split request handling

* fixing assert

* Handle change feed popped check in the blob worker

* fix: do not use a RYW transaction for a versionstamp because of randomize API version (#6768)

* more merge conflict issues

* Change feed destroy fixes

* Fixing change feed destroy and move race

* Check error condition in BG file req

* Using relative endpoints for blob worker interface

* Fixing bug in previous fix

* More destroy and move race fixes

* Don't update empty version on destroy in case it gets rolled back. moved() and removing will take care of ensuring it is not read

* Bug fix (#6796)

* fix: do not use a RYW transaction for a versionstamp because of randomize API version

* fix: if the initialSnapshotVersion was pruned, granule history was incorrect

* added a way to compress null bytes in printable()

* Fixing durability issue with moving and destroying change feeds

* Adding fix for not fully deleting files for a granule that child granules need to re-snapshot

* More destroy and move races

* Fixing change feed destroy and pop races

* Renaming bg prune to purge, and adding a C api and unit test for it

* more cleanup

* review comments

* Observability for granule purging

* better handling for change feed not registered

* Fixed purging bugs (#6815)

* fix: do not use a RYW transaction for a versionstamp because of randomize API version

* fix: if the initialSnapshotVersion was pruned, granule history was incorrect

* added a way to compress null bytes in printable()

* fixed a few purging bugs

Co-authored-by: Evan Tschannen <evan.tschannen@snowflake.com>
2022-04-08 14:15:25 -07:00
Dan Lambright
5bdc525353
Merge branch 'main' into vv 2022-04-08 13:16:04 -04:00
Jingyu Zhou
4fd414a8ed Merge branch 'main' into vv
Fix Conflicts:
	fdbclient/NativeAPI.actor.cpp
2022-03-31 09:38:36 -07:00
Jingyu Zhou
cfcf0f152c Merge branch 'main-4a085fc84' into vv
Fix Conflicts:
	fdbclient/NativeAPI.actor.cpp
	fdbserver/ClusterRecovery.actor.cpp
	fdbserver/MasterInterface.h
	fdbserver/masterserver.actor.cpp
	flow/error_definitions.h
2022-03-30 22:28:06 -07:00
Jingyu Zhou
b34f4052cd Merge branch 'main-f28dfc12b' into vv
Fix Conflicts:
	fdbclient/MultiVersionTransaction.actor.cpp
	fdbclient/NativeAPI.actor.cpp
	fdbclient/NativeAPI.actor.h
	fdbserver/storageserver.actor.cpp
2022-03-30 21:01:25 -07:00
Jingyu Zhou
00b57d4cce Merge branch 'main-67eba5ec7' into vv
Fix Conflicts:
	fdbclient/CommitProxyInterface.h
	fdbclient/NativeAPI.actor.cpp
	fdbclient/StorageServerInterface.h
	fdbserver/CommitProxyServer.actor.cpp
	fdbserver/storageserver.actor.cpp
2022-03-30 20:05:55 -07:00
Jon Fu
f266695a76 clear shared state map entries on version upgrade and use currentProtocolVersion instead of piping version through function parameters 2022-03-30 15:28:55 -04:00
Jon Fu
9d0a520fe0 add ProtocolVersion to shared state and state cluster map 2022-03-30 13:28:06 -04:00
Jon Fu
53d252fda0 Merge branch 'main' of github.com:apple/foundationdb into jfu-grv-cache-multi-threaded 2022-03-29 17:23:08 -04:00
Jon Fu
bdbe78e48f address code review comments 2022-03-29 17:20:22 -04:00
Josh Slocum
721c64b698 Adding Blob Granule Client C Unit Tests 2022-03-28 15:42:33 -05:00
Jon Fu
7c3151142e Merge branch 'main' of github.com:apple/foundationdb into jfu-grv-cache-multi-threaded 2022-03-23 16:41:33 -04:00
Jon Fu
4466e00716 Fix multithreaded GRV cache code and enable grv caching in multithreaded ctest suite 2022-03-23 16:09:12 -04:00
sfc-gh-tclinkenbeard
a71099471b Update copyright header dates 2022-03-21 13:36:23 -07:00
Jon Fu
9731d96797 Merge branch 'main' of github.com:apple/foundationdb into jfu-grv-cache-multi-threaded 2022-03-16 14:32:33 -04:00
Jon Fu
7ab205391b Revert "remove all multi-version client code related to grv cache"
This reverts commit 6d05ecffc21d8497b4f753d11b17beb6102f32ab.
2022-03-16 14:26:36 -04:00
A.J. Beamon
8bc2b283e1 Create a special keys API to create, delete, and read tenants. Remove the C API to create/delete tenants. 2022-03-15 09:23:30 -07:00
A.J. Beamon
c89b1e3537 Use TenantName type instead of StringRef in various places. Add a function on transactions to get the tenant name being used for the transaction. 2022-03-15 09:21:27 -07:00
A.J. Beamon
c635dcd3ad Add tenant support in the FDB native client 2022-03-15 09:21:27 -07:00
Tao Lin
e2c7c30faf
GetMappedRange support serializable & check RYW & continuation (#6181) 2022-03-10 10:05:44 -08:00
Dan Lambright
9544379cdf rebase 2022-01-20 11:12:33 -05:00
Josh Slocum
3afe9fb6e0 MVC bug fixes 2021-12-10 12:47:53 -06:00
Josh Slocum
7f4fcc8c2c Added FDBResult and made readBlobGranules use it 2021-12-01 16:22:05 -06:00
Josh Slocum
329091e14f Merge branch 'master' into bg_bindings 2021-11-11 10:13:37 -06:00
Josh Slocum
b8ac4213a1 Switched BG APIs to transaction instead of database 2021-11-11 08:59:06 -06:00
Vaidas Gasiunas
51b8ccf7d3 Merge remote-tracking branch 'apple/master' into notify-client-lib-changes 2021-11-10 18:40:34 +01:00
Vaidas Gasiunas
d6a5cc8460 MVC2.0: Rollback static initialization of the local client API instance 2021-11-10 15:55:33 +01:00
Tao Lin
fdb3b72e35 Introduce GetRangeAndFlatMap to push computations down to FDB
Re-introduce #5609
2021-11-09 13:52:28 -08:00
Tao Lin
586cc3b102
Revert "Introduce GetRangeAndFlatMap to push computations down to FDB" 2021-11-04 08:46:56 -07:00
Tao Lin
6c98e35893 Rename Hop to FlatMap 2021-11-03 13:32:01 -07:00
Tao Lin
0853661d13 Introduce getRangeAndHop to push computations down to FDB 2021-11-03 13:21:16 -07:00
Josh Slocum
e3839e198a Cleanup 2021-11-03 13:41:41 -05:00
Josh Slocum
a5b2f464e3 Refactored granule materialization code to be called by c client as well as internal granule management 2021-11-03 13:19:07 -05:00