* packaging/docker/Dockerfile
Add fdb-aws-s3-credentials-fetcher-sidecar container.
Runs perpetual script that writes blob-credentials.json to /var/fdb.
* packaging/docker/build-images.sh
Build and publish new sidecar container
* packaging/docker/fdb-aws-s3-credentials-fetcher/README.md
* packaging/docker/fdb-aws-s3-credentials-fetcher/fdb-aws-s3-credentials-fetcher.go
* packaging/docker/fdb-aws-s3-credentials-fetcher/go.mod
* packaging/docker/fdb-aws-s3-credentials-fetcher/go.sum
Script that fetches credentials via IRSA (IAM Roles for Service Accounts).
* packaging/docker/fdb-aws-s3-credentials-fetcher/fdb-aws-s3-credentials-fetcher.go
Match the key generated by fdbserver internally.
* fdbclient/S3BlobStore.actor.cpp
Add some logging around fail-to-find-credentials -- why.
* * fdbclient/tests/aws_fixture.sh
Use the fdb-aws-s3-credentials-fetcher script fetching credentials if available in ctests.
* fdbclient/tests/s3client_test.sh
TMPDIR might not be defined when we print usage.
Co-authored-by: Johannes Scheuermann <johscheuer@users.noreply.github.com>
`CreateColumnFamilyWithImport()` expects that the value inside
handle is `nullptr`. This patch fixed a codepath where we pass
a stale handle left by destroyed column family.
* Fix isOnMainThread in Simulation and Testing
isOnMainThread() is used to check if the currently running task
is on the FDB's event loop. However, in simulation this behaviour
is broken and always returns false.
In other modes such as UnitTest mode since `runTests()` is called before
`g_network->run()`, but without a wait() statement the event loop never
gets chance to set itself as main thread and the tests never sees
current thread as main thread. Therefore we add a yield inside
`runTests()` so yield control back to caller block and continue
with g_network->run() which eventually schedule it back after
initialization.
* Update fdbserver/tester.actor.cpp
Co-authored-by: Syed Paymaan Raza <1238752+spraza@users.noreply.github.com>
---------
Co-authored-by: Syed Paymaan Raza <1238752+spraza@users.noreply.github.com>
Otherwise, the pop version can become larger than the actual saved version when
switching to the regular pulling mode. Because the pop version is larger,
mutations larger than saved version could be popped and no long available.
Add gRPC file transfer service
* grpc: Add file size check
* grpc: change test addresses
* Fix CI/CD failure
* Disable gRPC for build
* Fixes for new gRPC in new build image
* Move FileTransfer definitions to CPP file
* Add compile time switch NO_MULTIREGION_TEST. When set, simulation tests
will not create configurations with more than one region. Tests requiring
multiple regions are ignored.
* While the RUN_IGNORED_TESTS setting allows running tests that have been
marked as ignored, this should not apply to multiregion tests. Multiregion
tests must be completely disabled if the NO_MULTIREGION setting is enabled.
---------
Co-authored-by: Dan Lambright <hlambright@apple.com>
* Add multiparting to s3client.
Fix boost::urls::parse_uri 's dislike of credentialed blobstore urls.
* fdbclient/BulkLoading.cpp
Add blobstore regex to extract credentials before feeding the boost
parse_uri.
* fdbclient/include/fdbclient/S3BlobStore.h
* fdbclient/S3BlobStore.actor.cpp
Add cleanup of failed multipart -- abortMultiPartUpload l(s3 will do
this in the background eventually but lets clean up after ourselves).
Also add getObjectRangeMD5 so can do multipart checksumming.
* fdbclient/S3Client.actor.cpp
Change upload file and download file to do multipart always.
Retry too.
* fdbclient/S3Client_cli.actor.cpp
Add command line to trace rather than output.
* Address Zhe review
* More logging around part upload and download
* Undo assert that proved incorrect; restore the old length math
doing copy in readObject.
Cleanup around TraceEvents in HTTTP.actor.
* Undo commented out cleanup -- for debugging
* formatting
---------
Co-authored-by: stack <stack@duboce.com>
