32 Commits

Author SHA1 Message Date
Renxuan Wang
2d78468cc0 Fix the workflow of not using proxy. 2022-08-22 11:16:48 -07:00
Renxuan Wang
657e6f8b60 Complete the proxy support for backup to S3. 2022-08-22 11:16:48 -07:00
Renxuan Wang
dc9599f2e9
Get network interfaces ready for https proxy. (#7556)
* Move HTTP from fdbclient/ to fdbrpc/.

* Move md5 and libb64 to contrib/.

* Get network interfaces ready for https proxy.

* Rebase
2022-07-25 17:08:32 -07:00
Markus Pilman
8123070c2a move rapidxml to its own target 2022-06-29 16:10:14 -06:00
Markus Pilman
8af056e7b0 fdbclient now compiling 2022-06-23 18:05:36 -06:00
Markus Pilman
ffaf15c12a moved wellknownendpoints and fixed some includes 2022-06-23 17:03:53 -06:00
Xiaoge Su
21ee76a44d fixup! Reformat source #2 2022-06-14 13:22:18 -07:00
Xiaoge Su
c2676df2f8 fixup! Reformat source 2022-06-14 13:22:18 -07:00
Xiaoge Su
9fb6e5bb05 fixup! Fix the clang error when using std::move
This patch is to fix the compile error

/root/src/fdbclient/S3BlobStore.actor.cpp:410:9: error: moving a local
object in a return statement prevents copy elision
[-Werror,-Wpessimizing-move]
 return std::move(resource);
        ^
/root/src/fdbclient/S3BlobStore.actor.cpp:410:9: note: remove std::move
call here
 return std::move(resource);
        ^~~~~~~~~~        ~
1 error generated.
2022-06-14 13:22:18 -07:00
Zhanwei Wang
e632aef1c7
Make backup work with s3 compatible service (#6355)(#6382) (#7324)
1. Support virtual hosting endpoint.

2. On-premise s3 compatible storage service may use IP instead of s3 form domain name,
especially for development/test environment.

Instead of parsing service and region from domain name,

1). Hard code "s3" as service name in v4 signature
2). Add new parameter to allow pass region name from url

3. Fix creating bucket issue on aws, adding request body.
2022-06-13 13:33:05 -07:00
Zhanwei Wang
da346d718f Fix compatibility issue of s3 backup
1. For v4 signature, http request path should be encoded, currently only parameters are encoded.
2. For v2 signature, http request date header should be in form of either the http date header or x-amz-date, currently in form of ISO 8601
3. Empty response header value cause http_bad_response error.
4. Skip verifying MD5 for range get request and if http response code 206 is returned.
2022-05-25 09:48:54 -05:00
Renxuan Wang
9190fffbb6 Commit suggestion.
Co-authored-by: Jingyu Zhou <jingyuzhou@gmail.com>
2022-05-09 13:56:33 -07:00
Renxuan Wang
13b2b455b1 HTTP proxy for backup.
Only support non-TLS blobs now.
2022-05-09 13:56:33 -07:00
Sam Gwydir
5403a29ecb
add WolfSSL support (#6682)
remove extraneous include
2022-04-28 16:53:38 -07:00
Renxuan Wang
465ff712b6
Move Hostname to its own files. (#6759)
* Change DNS cache to use std::map.

Revert commit 90c259d84e95dd35e01149c0a86bd18e82e33930, because if we use unordered_map, toString() can be inconsistent.

* Move ClientKnob::COORDINATOR_HOSTNAME_RESOLVE_DELAY to FlowKnob::HOSTNAME_RESOLVE_DELAY.

* Move Hostname to its own files.

Also, add resolve-related variables and functions in Hostname.
2022-04-04 19:04:51 -07:00
Renxuan Wang
e1775627ab Add a check on proxy format. 2022-03-29 10:17:14 -07:00
Renxuan Wang
0a332ee1c1 Add proxy option to backup and restore params. 2022-03-28 17:10:49 -07:00
Josh Slocum
5684fcace4 Merge branch 'main' into s3_sdk_credentials 2022-03-24 15:44:41 -05:00
Jingyu Zhou
902c10c299 Fix missing blob access key in credentials
This causes failures of backup_auth_missing: ErrorDescription="Cannot find
authentication details (such as a password or secret key) for the specified
Backup Container URL"

This is found by manually running "fdbdecode".
2022-03-24 09:26:41 -07:00
Josh Slocum
ace2750ae3 Build AWS SDK as part of FDB, and use it to acquire blob credentials. This supports IAM role-based authentication, in addition to all existing credential types. 2022-03-24 10:09:06 -05:00
sfc-gh-tclinkenbeard
a71099471b Update copyright header dates 2022-03-21 13:36:23 -07:00
A.J. Beamon
250a88e682 Enforce that trace event suppression calls happen first when using trace event call chaining. Fix various instances where we weren't following this requirement. 2022-02-24 12:25:52 -08:00
Bala Namasivayam
0f36157c2c Clang format files 2022-02-07 17:53:05 -08:00
Bala Namasivayam
a420098fc5 Support AWS v4 header for s3 backup and restore 2022-02-07 17:53:05 -08:00
sfc-gh-tclinkenbeard
ac274cf479 clang-format S3BlobStore* files 2021-05-10 10:25:32 -07:00
Kevin King
b10ea7677e [fdbbackup] check for credentials inside of setAuthHeaders 2021-05-10 10:21:37 -07:00
Kevin King
e3114a4656 [fdbbackup] make credentials optional
when blob storage is configured on a private network (such as an AWS
VPC), there isn't a need to specify http auth
2021-05-10 08:42:27 -07:00
FDB Formatster
df90cc89de apply clang-format to *.c, *.cpp, *.h, *.hpp files 2021-03-10 10:18:07 -08:00
sfc-gh-tclinkenbeard
5b2e88b187 Use structured bindings in for loops 2020-12-27 01:46:20 -04:00
sfc-gh-tclinkenbeard
4669f837fa Add uses of makeReference 2020-11-07 22:10:18 -08:00
sfc-gh-tclinkenbeard
ca26cb57a2 Reapply BlobStore bug fix to S3BlobStore.actor.cpp
Because BlobStore.actor.cpp was renamed, this commit was lost in the
merge, so reapplying here
2020-10-30 10:06:35 -07:00
sfc-gh-tclinkenbeard
55e86c1336 Disambiguate between S3BlobStore and other blob stores 2020-10-29 20:42:23 -07:00