Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
3,125 Commits
189 Branches
453 Tags
379 MiB
Commit Graph

131 Commits (32e5f8c7154e31bdb30faa301973601ca8a242b2)

Author SHA1 Message Date
Marco Neumann 1cbe4031d3 feat: allow passing a session token to AWS stores 2021-05-27 11:53:12 +02:00
Marco Neumann d5df7c381a test: accept optional `AWS_ENDPOINT` for OS tests 2021-05-26 18:14:58 +02:00
Marco Neumann 464667d8b8 docs: clarify the FilePath directory handling 2021-05-20 14:20:56 +02:00
Marco Neumann ccd094dfcf refactor: streamline directory handling in path parsing 2021-05-20 10:22:20 +02:00
Marco Neumann ff004c1a51 feat: add `ObjectStore.path_from_dirs_and_filename` 2021-05-17 11:33:22 +02:00
Marco Neumann 4d105713a5 feat: add convenience `parsed_path!` marco 2021-05-07 09:51:44 +02:00
Marco Neumann fadf0176d4 feat: impl Hash for DirsAndFileName and PathParts 2021-05-07 09:51:44 +02:00
Marco Neumann d726cf4829 refactor: make `object_store::path::parts` public 2021-05-07 09:51:44 +02:00
DCjanus 9eb658eac5 feat: s3 compatibility storages support 2021-05-06 23:27:37 +08:00
Marco Neumann 1f5a9b5ebf feat: make DirsAndFileName members public 
Helps with #1253.
 2021-05-03 09:51:00 +02:00
Marco Neumann ca5351088a feat: add Path->DirsAndFileName conversion 
This allows for easier parsing of received paths.

Helps with #1253.
 2021-05-03 09:51:00 +02:00
Nga Tran 140d96dbea feat: tests ffor loading data to object store and make sure twe still query read buffer 2021-04-28 15:59:17 -04:00
Marco Neumann eddc9319ff docs: deny broken intradoc links 2021-04-27 13:22:28 +02:00
Marco Neumann 53e83f8385 feat: add throttled store wrapper 2021-04-27 09:31:48 +02:00
Carol (Nichols || Goulding) d8a92a8570 test: Don't run any integration tests unless TEST_INTEGRATION is set 2021-04-22 14:21:50 -04:00
Carol (Nichols || Goulding) c7e6a0a797 fix: Use wb rather than wal in object storage test path names for consistency 2021-04-21 17:43:03 +00:00
Carol (Nichols || Goulding) f136931225 fix: Inconsistent ordering lints 2021-04-19 08:48:11 -04:00
Marco Neumann fd0da7e74a chore: upgrade arrow and Rust 
See https://github.com/apache/arrow/pull/10082 for upstream PR.
 2021-04-19 14:00:04 +02:00
Nga Tran 657bfa1b20 refactor: address Andrew's comments 2021-04-16 17:44:46 -04:00
Nga Tran 4c23ca8888 feat: full implementation of parquet's read_filter for review 2021-04-16 16:03:24 -04:00
Raphael Taylor-Davies 1997324344
feat: mutable buffer snapshotting (#1179) 
* feat: mutable buffer snapshotting

* chore: review feedback
 2021-04-13 12:14:54 +00:00
Jake Goulding 16cc37e7f3
chore: update cloud-storage to 0.9 (#1165) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-04-09 14:37:51 +00:00
Carol (Nichols || Goulding) 82588d5c72 fix: Don't return Result from test functions 2021-04-07 12:40:00 -04:00
Marko Mikulicic 8bd7a39607 fix: Use InstanceMetadataProvider directly 
Rusoto's ChainProvider swallows the error message produced by the underlying InstanceMetadataProvider
(see d59d716f09/rusoto/credential/src/lib.rs (L397)) making it hard for us to know why it's not working in our staging cluster.
 2021-03-15 13:55:43 +00:00
Marko Mikulicic 3c8a58266a
feat: Make S3 auth parameters optional 
so that if not present, the aws client library can use builtin auth providers,
such as the InstanceMetadataProvider, which is commonly used to get the credentials
granted to the AWS VM via cloud native mechanism.
 2021-03-15 10:11:01 +01:00
Carol (Nichols || Goulding) d67a03f616 fix: Improved tests found another unimplemented hiding 2021-03-04 16:18:31 -05:00
Carol (Nichols || Goulding) ca2f74063e fix: Test using the object store wrapper interface 2021-03-04 16:18:11 -05:00
Carol (Nichols || Goulding) 16df4c542b fix: Rename AZURE_STORAGE_MASTER_KEY to AZURE_STORAGE_ACCESS_KEY 2021-03-04 10:15:46 -05:00
Carol (Nichols || Goulding) e9fedfae17 fix: Use AWS_DEFAULT_REGION instead of AWS_REGION 2021-03-04 10:15:30 -05:00
Carol (Nichols || Goulding) 37746173d9 feat: Change azure object store to only get config from args, not env 2021-03-04 10:14:42 -05:00
Carol (Nichols || Goulding) 02d981451d feat: Implement Google Cloud Storage-related CLI arguments 2021-03-04 10:14:42 -05:00
Carol (Nichols || Goulding) ef13c1023e feat: Change google cloud object store to get config from args, not env 2021-03-04 10:14:42 -05:00
Carol (Nichols || Goulding) 06236e796b feat: Change aws object store to get config from args, not env 2021-03-04 10:14:42 -05:00
Carol (Nichols || Goulding) c7ef18337c feat: Consolidate all bucket config into one option/env var 
Fixes #869.
 2021-02-25 15:53:20 -05:00
Marko Mikulicic 12b768b8f1 fix: Escape empty string PathPart 
Empty directory names are silently ignored and can lead to very surprising effects
such as directory layouts missing a level. This makes it hard to reason about directory structures.

A sane object store path API should either disallow empty names or deal with them gracefully.

Since we already have to escape file/directory names using the minimum common denominator valid character
set for known cloud providers, it feels quite natural to treat this empty dir/file name problem as encoding problem.
 2021-02-24 21:13:56 +00:00
Marko Mikulicic 9e521a2ea1 feat: Plug GCS list_with_delimiter impl 
And fix env.example. Now GCS can be used to persist snapshots.
 2021-02-23 21:06:53 +00:00
Marko Mikulicic 9860def4b2 feat: Add S3 plumbing to iodx config 2021-02-23 14:55:55 +00:00
Carol (Nichols || Goulding) cff12da3a1 fix: Upgrade to released version of cloud_storage 
Fixes #801.
 2021-02-22 13:01:06 -05:00
Carol (Nichols || Goulding) a42103f436 Merge remote-tracking branch 'origin/main' into cn/google-list-with-delimiter 2021-02-22 12:53:46 -05:00
Marko Mikulicic 81739bf486 docs: Fix typo 2021-02-22 10:35:23 +00:00
Carol (Nichols || Goulding) cc6738c6f3 fix: Check for AZURE_STORAGE_MASTER_KEY in the test macro too 2021-02-18 16:53:06 -05:00
Carol (Nichols || Goulding) fcd4f91909 feat: Implement list_with_delimiter for Azure storage 2021-02-18 16:37:23 -05:00
Carol (Nichols || Goulding) 57942b51b7 feat: Update to latest Azure sdk to get delimiter support 
Needed these PRs:
  - https://github.com/Azure/azure-sdk-for-rust/pull/176
  - https://github.com/Azure/azure-sdk-for-rust/pull/179

Also needed to enable the queue feature to get the azure_storage crate
compiling; at the moment, the code is still being reorganized and the
features aren't independent yet:
https://github.com/Azure/azure-sdk-for-rust/issues/177
 2021-02-18 14:59:06 -05:00
Jake Goulding 484adcc257 chore: fix typo in an error message 2021-02-18 14:57:38 -05:00
Marko Mikulicic 536c1724bd feat: Allow to put streams of unknown length to objectstore 
Addresses the API aspect of #818

Adds a utility module that helps computing the length of a stream while buffering it
for later replay (in-memory or spilling it in a temporary file).
 2021-02-18 16:49:18 +00:00
Carol (Nichols || Goulding) f934a21efe test: Update tests to match new cloud storage error behavior 2021-02-17 14:23:34 -05:00
Carol (Nichols || Goulding) ef54131afb feat: Gets google cloud list_with_delimiter tests passing 2021-02-17 14:23:33 -05:00
Edd Robinson 2b642a8da6 refactor: add arc clone lint 2021-02-15 12:38:19 +00:00
Jake Goulding dad426d02e
fix: Report a failure to parse an AWS datetime (#794) 
* fix: Report a failure to parse an AWS datetime

* refactor: use SNAFU context selectors instead of enum variants
 2021-02-12 15:10:49 +00:00
Raphael Taylor-Davies c7e8a68fbe
fix: enable tokio::fs for object_store crate (#788) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-02-11 12:12:57 +00:00