foundationdb/flow/swift_task_priority.cpp
Konrad `ktoso` Malawski c26aa0b2a3
Introduce initial Swift support in fdbserver (#10156)
* [fdbserver] workaround the FRT type layout issue to get Swfit getVersion working

* MasterData.actor.h: fix comment typo

* masterserver.swift: some tweaks

* masterserver.swift: remove getVersion function, use the method

* masterserver.swift: print replied version to output for tracing

* [swift] add radar links for C++ interop issues found in getVersion bringup

* Update fdbserver.actor.cpp

* Migrate MasterData closer to full reference type

This removes the workaround for the FRT type layout issue, and gets us closer to making MasterData a full reference type

* [interop] require a new toolchain (>= Oct 19th) to build

* [Swift] fix computation of toAdd for getVersion Swift implementation

* add Swift to FDBClient and add async `atLeast` to NotifiedVersion

* fix

* use new atLeast API in master server

* =build fixup link dependencies in swift fdbclient

* clocks

* +clock implement Clock using Flow's notion of time

* [interop] workaround the immortal retain/release issue

* [swift] add script to get latest centos toolchain

* always install swift hooks; not only in "test" mode

* simulator - first thing running WIP

* cleanups

* more cleanup

* working snapshot

* remove sim debug printlns

* added convenience for whenAtLeast

* try Alex's workaround

* annotate nonnull

* cleanup clock a little bit

* fix missing impls after rebase

* Undo the swift_lookup_Map_UID_CommitProxyVersionReplies workaround

No longer needed - the issue was retain/release

* [flow][swift] add Swift version of BUGGIFY

* [swiftication] add CounterValue type to provide value semantics for Counter types on the Swift side

* remove extraneous requestingProxyUID local

* masterserver: initial Swift state prototype

* [interop] make the Swiftied getVersion work

* masterserver - remove the C++ implementation (it can't be supported as state is now missing)

* Remove unnecessary SWIFT_CXX_REF_IMMORTAL annotations from Flow types

* Remove C++ implementation of CommitProxyVersionReplies - it's in Swift now

* [swift interop] remove more SWIFT_CXX_REF_IMMORTAL

* [swift interop] add SWIFT_CXX_IMMORTAL_SINGLETON_TYPE annotation for semanticly meaningful immortal uses

* rename SWIFT_CXX_REF_IMMORTAL -> UNSAFE_SWIFT_CXX_IMMORTAL_REF

* Move master server waitForPrev to swift

* =build fix linking swift in all modules

* =build single link option

* =cmake avoid manual math, just get "last" element from list

* implement Streams support (#18)

* [interop] update to new toolchain #6

* [interop] remove C++ vtable linking workarounds

* [interop] make MasterData proper reference counted SWIFT_CXX_REF_MASTERDATA

* [interop] use Swift array to pass UIDs to registerLastCommitProxyVersionReplies

* [interop] expose MasterServer actor to C++ without wrapper struct

* [interop] we no longer need expose on methods 🥳

* [interop] initial prototype of storing CheckedContinuation on the C++ side

* Example of invoking a synchronous swift function from a C++ unit test. (#21)

* move all "tests" we have in Swift, and priority support into real modules (#24)

* Make set continuation functions inline

* Split flow_swift into flow_swift and flow_swift_future to break circular dependency

* rename SwiftContinuationCallbackStruct to FlowCallbackForSwiftContinuation

* Future interop: use a method in a class template for continuation set call

* Revert "Merge pull request #22 from FoundationDB/cpp-continuation" (#30)

* Basic Swift Guide (#29)

Co-authored-by: Alex Lorenz <arphaman@gmail.com>

* Revert "Revert "Merge pull request #22 from FoundationDB/cpp-continuation" (#30)"

This reverts commit c025fe6258c4c4904d5e70cd549d408bb61f2802.

* Restore the C++ continuation, but it seems waitValue is broken for CInt somehow now

* disable broken tests - waitValue not accessible

* Streams can be async iterated over (#27)

Co-authored-by: Alex Lorenz <arphaman@gmail.com>

* remove work in progress things (#35)

* remove some not used (yet) code

* remove expose func for CInt, it's a primitive so we always have witness info (#37)

* +masterdata implement provideVersions in Swift (#36)

* serveLiveCommittedVersion in Swift (#38)

* Port updateLiveCommittedVersion to swift (#33)

Co-authored-by: Konrad `ktoso` Malawski <konrad_malawski@apple.com>

* Implement updateRecoveryData in Swift (#39)

Co-authored-by: Alex Lorenz <arphaman@gmail.com>

* Simplify flow_swift to avoid multiple targets and generate separate CheckedContinuation header

* Uncomment test which was blocked on extensions not being picked up (#31)

* [interop] Use a separate target for Swift-to-C++ header generation

* reduce boilerplate in future and stream support (#41)

* [interop] require interop v8 - that will fix linker issue (https://github.com/apple/swift/issues/62448)

* [interop] fix swift_stream_support.h Swift include

* [interop] bump up requirement to version 9

* [interop] Generalize the Flow.Optional -> Swift.Optional conversion using generics

* [WIP] masterServer func in Swift (#45)

* [interop] Try conforms_to with a SWIFT_CONFORMS_TO macro for Optional conformance (#49)

* [interop] include FlowOptionalProtocol source file when generating Flow_CheckedContinuation.h

This header generation step depends on the import of the C++ Flow module, which requires the presence of FlowOptionalProtocol

* conform Future to FlowFutureOps

* some notes

* move to value() so we can use discardable result for Flow.Void

* make calling into Swift async funcs nicer by returning Flow Futures

* [interop] hide initial use of FlowCheckedContinuation in flow.h to break dependency cycle

* [fdbserver] fix an EncryptionOpsUtils.h modularization issue (showed up with modularized libc++)

* Pass GCC toolchain using CMAKE_Swift_COMPILE_EXTERNAL_TOOLCHAIN to Swift's clang importer

* [interop] drop the no longer needed libstdc++ include directories

* [cmake] add a configuration check to ensure Swift can import C++ standard library

* [swift] include msgpack from msgpack_DIR

* [interop] make sure the FDB module maps have 'export' directive

* add import 'flow_swift' to swift_fdbserver_cxx_swift_value_conformance.swift

This is needed for CONFORMS_TO to work in imported modules

* make sure the Swift -> C++ manually bridged function signature matches generated signature

* [interop][workaround] force back use of @expose attribute before _Concurrency issue is fixed

* [interop] make getResolutionBalancer return a pointer to allow Swift to use it

We should revert back to a reference once compiler allows references again

* [interop] add a workaround for 'pop' being marked as unsafe in Swift

* masterserver.swift: MasterData returns the Swift actor pointer in an unsafe manner

* Add a 'getCopy' method to AsyncVar to make it more Swift friendly

* [interop] bump up the toolchain requirement

* Revert "[interop][workaround] force back use of @expose attribute before _Concurrency issue is fixed"

This reverts commit b01b271a76d1677bbb7c5c9c64cdad4b8b2b9612.

* [interop] add FIXME comments highlighting new issue workarounds

* [interop] adopt the new C++ interoperability compiler flag

* [interop] generate swift compile commands

* Do not deduplicate Swift compilation commands

* [interop] generate swift compile commands

* Do not deduplicate Swift compilation commands

* flow actorcompiler.h: add a SWIFT_ACTOR empty macro definition

This is needed to make the actor files parsable by clangd

* [cmake] add missing dependencies

* experimental cross compile

* [cmake] fix triple in cross-compiled cmake flags

* [interop] update to interop toolchain version 16

* [x-compile] add flags for cross-compiling boost

* cleanup x-compile cmake changes

* [cmake] fix typo in CMAKE_Swift_COMPILER_EXTERNAL_TOOLCHAIN config variable

* [interop] pass MasterDataActor from Swift to C++ and back to Swift

* [fdbserver] Swift->C++ header generation for FDBServer should use same module cache path

* Update swift_get_latest_toolchain.sh to fetch 5.9 toochains

* set HAVE_FLAG_SEARCH_PATHS_FIRST for cross compilation

* Resolve conflicts in net2/sim2/actors, can't build yet

* undo SWIFT_ACTOR changes, not necessary for merge

* guard c++ compiler flags with is_cxx_compile

* Update flow/actorcompiler/ActorParser.cs

Co-authored-by: Evan Wilde <etceterawilde@gmail.com>

* update the boost dependency

* Include boost directory from the container for Swift

* conform flow's Optional to FlowOptionalProtocol again

* Guard entire RocksDBLogForwarder.h with SSD_ROCKSDB_EXPERIMENTAL to avoid failing on missing rocksdb APIs

* remove extraneous merge marker

* [swift] update swift_test_streams.swifto to use vars in more places

* Add header guard to flow/include/flow/ThreadSafeQueue.h to fix moduralization issue

* Update net and sim impls

* [cmake] use prebuilt libc++ boost only when we're actually using libc++

* [fdbserver] Swift->C++ header generation for FDBServer should use same module cache path

* fixups after merge

* remove CustomStringConvertible conformance that would not be used

* remove self-caused deprecation warnings in future_support

* handle newly added task priority

* reformatting

* future: make value() not mutating

* remove FIXME, not needed anymore

* future: clarify why as functions

* Support TraceEvent in Swift

* Enable TraceEvent using a class wrapper in Swift

* prearing WITH_SWIFT flag

* wip disabled failing Go stuff

* cleanup WITH_SWIFT_FLAG and reenable Go

* wip disabled failing Go stuff

* move setting flag before printing it

* Add SWIFT_IDE_SETUP and cleanup guides and build a bit

* Revert "Wipe packet buffers that held serialized WipedString (#10018)"

This reverts commit e2df6e33029897360f8e11b3aea8fef97393a98c.

* [Swift] Compile workaround in KeyBackedRangeMap; default init is incorrect

* [interop] do not add FlowFutureOps conformance when building flow clang module for Flow checked continuation header pre-generation

* make sure to show  -DUSE_LIBCXX=OFF in readme

* readme updates

* do not print to stderr

* Update Swift and C++ code to build with latest Swift 5.9 toolchain now that we no longer support universal references and bridge the methods that take in a constant reference template parameter correctly

* Fix SERVER_KNOBS and enable use them for masterserver

* Bump to C++20, Swift is now able to handle it as well

* Put waitForPrev behind FLOW_WITH_SWIFT knob

* Forward declare updateLiveCommittedVersion

* Remove unused code

* fix wrong condition set for updateLiveCommittedVersion

* Revert "Revert "Wipe packet buffers that held serialized WipedString (#10018)""

This reverts commit 5ad8dce0525fb1844664ed2ccd7ba595db6913dd.

* Enable go-bindings in cmake

* Revert "Revert "Wipe packet buffers that held serialized WipedString (#10018)""

This reverts commit 5ad8dce0525fb1844664ed2ccd7ba595db6913dd.

* USE_SWIFT flag so we "build without swift" until ready to by default

* uncomment a few tests which were disabled during USE_SWIFT enablement

* the option is WITH_SWIFT, not USE

* formatting

* Fix masterserver compile error

* Fix some build errors.

How did it not merge cleanly? :/

* remove initializer list from constructor

* Expect Swift toolchain only if WITH_SWIFT is enabled

* Don't require Flow_CheckedContinuation when Swift is disabled

* Don't compile FlowCheckedContinuation when WITH_SWIFT=OFF

* No-op Swift macros

* More compile guards

* fix typo

* Run clang-format

* Guard swift/bridging include in fdbrpc

* Remove printf to pass the test

* Remove some more printf to avoid potential issues

TODO: Need to be TraceEvents instead

* Remove __has_feature(nullability) as its only used in Swift

* Don't use __FILENAME__

* Don't call generate_module_map outside WITH_SWIFT

* Add some more cmake stuff under WITH_SWIFT guard

* Some more guards

* Bring back TLSTest.cpp

* clang-format

* fix comment formatting

* Remove unused command line arg

* fix cmake formatting in some files

* Address some review comments

* fix clang-format error

---------

Co-authored-by: Alex Lorenz <arphaman@gmail.com>
Co-authored-by: Russell Sears <russell_sears@apple.com>
Co-authored-by: Evan Wilde <etceterawilde@gmail.com>
Co-authored-by: Alex Lorenz <aleksei_lorenz@apple.com>
Co-authored-by: Vishesh Yadav <vishesh_yadav@apple.com>
Co-authored-by: Vishesh Yadav <vishesh3y@gmail.com>
2023-06-02 16:09:28 -05:00

271 lines
5.8 KiB
C++

/*
* swift_concurrency_hooks.cpp
*
* This source file is part of the FoundationDB open source project
*
* Copyright 2013-2022 Apple Inc. and the FoundationDB project authors
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
#include "flow/swift.h"
#include "flow/swift_concurrency_hooks.h"
#include "flow/swift/ABI/Task.h"
#include "flow/TLSConfig.actor.h"
// FIXME: surely there must be some more automatic way to maintain the mappings with Swift/C++ interop.
TaskPriority swift_priority_to_net2(swift::JobPriority p) {
// printf("[c++][%s:%d](%s) converting a priority (priority: %zu)\n", __FILE_NAME__, __LINE__, __FUNCTION__, p);
TaskPriority fp = TaskPriority::Zero;
switch (static_cast<std::underlying_type<swift::JobPriority>::type>(p)) {
case 255:
fp = TaskPriority::Max;
break;
case 200:
fp = TaskPriority::RunLoop;
break;
case 173:
fp = TaskPriority::ASIOReactor;
break;
case 73:
fp = TaskPriority::RunCycleFunction;
break;
case 72:
fp = TaskPriority::FlushTrace;
break;
case 71:
fp = TaskPriority::WriteSocket;
break;
case 70:
fp = TaskPriority::PollEIO;
break;
case 69:
fp = TaskPriority::DiskIOComplete;
break;
case 68:
fp = TaskPriority::LoadBalancedEndpoint;
break;
case 67:
fp = TaskPriority::ReadSocket;
break;
case 66:
fp = TaskPriority::AcceptSocket;
break;
case 65:
fp = TaskPriority::Handshake;
break;
case 64:
fp = TaskPriority::CoordinationReply;
break;
case 63:
fp = TaskPriority::Coordination;
break;
case 62:
fp = TaskPriority::FailureMonitor;
break;
case 61:
fp = TaskPriority::ResolutionMetrics;
break;
case 60:
fp = TaskPriority::Worker;
break;
case 59:
fp = TaskPriority::ClusterControllerWorker;
break;
case 58:
fp = TaskPriority::ClusterControllerRecruit;
break;
case 57:
fp = TaskPriority::ClusterControllerRegister;
break;
case 56:
fp = TaskPriority::ClusterController;
break;
case 55:
fp = TaskPriority::MasterTLogRejoin;
break;
case 54:
fp = TaskPriority::ProxyStorageRejoin;
break;
case 53:
fp = TaskPriority::TLogQueuingMetrics;
break;
case 52:
fp = TaskPriority::TLogPop;
break;
case 51:
fp = TaskPriority::TLogPeekReply;
break;
case 50:
fp = TaskPriority::TLogPeek;
break;
case 49:
fp = TaskPriority::TLogCommitReply;
break;
case 48:
fp = TaskPriority::TLogCommit;
break;
case 47:
fp = TaskPriority::ReportLiveCommittedVersion;
break;
case 46:
fp = TaskPriority::ProxyGetRawCommittedVersion;
break;
case 45:
fp = TaskPriority::ProxyMasterVersionReply;
break;
case 44:
fp = TaskPriority::ProxyCommitYield2;
break;
case 43:
fp = TaskPriority::ProxyTLogCommitReply;
break;
case 42:
fp = TaskPriority::ProxyCommitYield1;
break;
case 41:
fp = TaskPriority::ProxyResolverReply;
break;
case 40:
fp = TaskPriority::ProxyCommit;
break;
case 39:
fp = TaskPriority::ProxyCommitBatcher;
break;
case 38:
fp = TaskPriority::TLogConfirmRunningReply;
break;
case 37:
fp = TaskPriority::TLogConfirmRunning;
break;
case 36:
fp = TaskPriority::ProxyGRVTimer;
break;
case 35:
fp = TaskPriority::GetConsistentReadVersion;
break;
case 34:
fp = TaskPriority::GetLiveCommittedVersionReply;
break;
case 33:
fp = TaskPriority::GetLiveCommittedVersion;
break;
case 32:
fp = TaskPriority::GetTLogPrevCommitVersion;
break;
case 31:
fp = TaskPriority::UpdateRecoveryTransactionVersion;
break;
case 30:
fp = TaskPriority::DefaultPromiseEndpoint;
break;
case 29:
fp = TaskPriority::DefaultOnMainThread;
break;
case 28:
fp = TaskPriority::DefaultDelay;
break;
case 27:
fp = TaskPriority::DefaultYield;
break;
case 26:
fp = TaskPriority::DiskRead;
break;
case 25:
fp = TaskPriority::DefaultEndpoint;
break;
case 24:
fp = TaskPriority::UnknownEndpoint;
break;
case 23:
fp = TaskPriority::MoveKeys;
break;
case 22:
fp = TaskPriority::DataDistributionLaunch;
break;
case 21:
fp = TaskPriority::Ratekeeper;
break;
case 20:
fp = TaskPriority::DataDistribution;
break;
case 19:
fp = TaskPriority::DataDistributionLow;
break;
case 18:
fp = TaskPriority::DataDistributionVeryLow;
break;
case 17:
fp = TaskPriority::BlobManager;
break;
case 16:
fp = TaskPriority::DiskWrite;
break;
case 15:
fp = TaskPriority::UpdateStorage;
break;
case 14:
fp = TaskPriority::CompactCache;
break;
case 13:
fp = TaskPriority::TLogSpilledPeekReply;
break;
case 12:
fp = TaskPriority::BlobWorkerReadChangeFeed;
break;
case 11:
fp = TaskPriority::BlobWorkerUpdateFDB;
break;
case 10:
fp = TaskPriority::BlobWorkerUpdateStorage;
break;
case 9:
fp = TaskPriority::FetchKeys;
break;
case 8:
fp = TaskPriority::RestoreApplierWriteDB;
break;
case 7:
fp = TaskPriority::RestoreApplierReceiveMutations;
break;
case 6:
fp = TaskPriority::RestoreLoaderFinishVersionBatch;
break;
case 5:
fp = TaskPriority::RestoreLoaderSendMutations;
break;
case 4:
fp = TaskPriority::RestoreLoaderLoadFiles;
break;
case 3:
fp = TaskPriority::LowPriorityRead;
break;
case 2:
fp = TaskPriority::Low;
break;
case 1:
fp = TaskPriority::Min;
break;
case 0:
fp = TaskPriority::Zero;
break;
default: {
printf("[c++]Unknown priority: %zu\n", p);
abort();
}
}
// return static_cast<std::underlying_type<TaskPriority>::type>(fp);
return fp;
}