076f67285d
Merge branch 'main' into cn+jg/file-delimiter
2021-02-05 09:45:34 -05:00
47bc28460e
refactor: rename partition, table, and column in parition_meta for clarity ( #757 )
...
* refactor: rename partition, table, and column in parition_meta for clarity
2021-02-05 08:00:22 -05:00
de7bc7d645
feat: add column name to the partition metadata summaries ( #755 )
2021-02-05 07:20:16 -05:00
3ec483b769
refactor: Reduce async in mutable buffer, use std::sync ( #749 )
...
* refactor: Reduce async in mutable buffer, use std::sync
* fix: logical confict with new code
2021-02-05 06:47:40 -05:00
a06fdf3a5f
refactor: rename http_routes module to http ( #751 )
2021-02-05 05:52:06 -05:00
fbf776c6b3
chore: Clean up Cargo.tomls ( #754 )
...
* fix: test_helpers crate should only be a dev-dep
* fix: object_store no longer has a build script, so no longer needs a build dep
* chore: Alphabetize all Cargo.tomls
2021-02-04 18:56:02 -05:00
14cb4b4fdc
fix: Fix copy&pasted check_response descriptions ( #740 )
...
Copy pasta is best pasta
2021-02-04 18:28:08 -05:00
5c3661dd91
chore: refactor how columns are kept in the table in the mutable buffer
...
This one is a bit of a yak shave in advance of adding column names to the summary statistics. I needed the column and its name (or identifier) to be together, rather than the id to index map that existed before. I think the table_id and column_id stuff should be refactored out over time since they add a ton of complexity to the code and don't add much value. Having those as Strings would be much easier and probably be a drop in the bucket for memory usage. Basically, I don't think they need to be interned. But that would be an even more massive refactor touching so many things in the MutableBuffer, I leave it as a later exercise.
Hopefully this makes the code simpler and cleaner in the interim and it gives me the column_id with the column so that I can easily look up the name when generating the summary statistics for a chunk.
2021-02-04 16:31:55 -05:00
8b18003e19
test: Don't check file metadata because SystemTime is not monotonic
...
See https://doc.rust-lang.org/std/time/struct.SystemTime.html
2021-02-04 15:46:11 -05:00
fa8594327d
test: Add a better failure message to aid debugging
2021-02-04 15:12:33 -05:00
1f8043a3f8
feat: add approximate memory size tracking to mutable buffer
...
This updates the mutable buffer, partitions, chunks, dictionary, tables, and individual columns to be able to return their approximate memory size used. This doesn't aim to be exact. There are spots where I'm not counting table or column pointers or the partition key. My expectation is that the data size will dominate and a few pointers here and there won't matter.
2021-02-04 13:50:43 -05:00
abbd29aeeb
fix: Use Self in From impls
2021-02-04 13:40:33 -05:00
3f1434e0e4
refactor: Remove redundant parts of error variant names
...
This info is now conveyed by the module each error comes from.
2021-02-04 13:04:52 -05:00
af7a5fa952
fix: Use walkdir::Errors within disk::Errors
...
I didn't want the object store lib Error to have to know about walkdir,
but I feel better about it now that this error type is scoped to the
disk module. The walkdir errors might have a bit more information.
2021-02-04 13:04:52 -05:00
80581c9084
fix: Remove vestigial error types
2021-02-04 13:04:52 -05:00
f8fb24b88c
refactor: Extract In-memory memory::Error
2021-02-04 13:04:52 -05:00
8e6a06ebb2
refactor: Extract Azure azure::Error
2021-02-04 13:04:52 -05:00
841f4ee314
refactor: Extract AWS S3 aws::Error
2021-02-04 13:04:52 -05:00
1492e52e57
refactor: Extract Google Cloud Storage gcp::Error
2021-02-04 13:04:52 -05:00
f795c56c8d
refactor: Start splitting up the object store error type; extract disk::Error
...
It's starting to get out of control. Time to fix that.
2021-02-04 13:04:52 -05:00
f9454fb57f
feat: Implement list_with_delimiter for File object store
...
Fixes #688 .
2021-02-04 13:04:52 -05:00
5b18f7dbea
feat: Hook DirsAndFileName push_part_as_dir to FilePath
2021-02-04 13:04:52 -05:00
a7cd8a2796
feat: Add a way to unset a file name in an object store path
2021-02-04 13:04:52 -05:00
c722188c5a
feat: Connect parts_after_prefix from DirsAndFileName to FilePath
...
This will be useful in the File object store's list_with_delimiter.
2021-02-04 13:04:52 -05:00
c66efa80d1
feat: Implement PartialOrd and Ord for FilePath
...
This allows storing FilePaths in a BTreeSet and ordering FilePaths.
2021-02-04 13:04:52 -05:00
fd39315388
refactor: Improve error name
...
I thought this only had to do with the InMemory Put request, but it's a
bit more general than that. Hopefully clarifying the purpose of this
error.
2021-02-04 13:04:52 -05:00
678044e08a
fix: test isn't a special extension recognized by object storage
...
This test was invalid because there are cases in which we use the
assumption that all file names in object storage should end with
`.json`, `.parquet`, or `.segment`.
2021-02-04 13:04:52 -05:00
13b097728f
fix: Improve code, comments and error messages for consistency ( #737 )
...
* fix: Improve code, comments and error messages for consistency
* fix: update error message
2021-02-04 12:05:24 -05:00
1ff9a3ab87
Merge pull request #741 from influxdata/listdb
...
feat: Implement list databases API
2021-02-04 17:06:47 +01:00
202a4187a1
feat: Implement list databases API
...
Ref #722
2021-02-04 16:50:40 +01:00
42de58497e
Merge pull request #742 from influxdata/routerify-error-handling
...
feat: consistent global error handling and logging
2021-02-04 15:04:25 +00:00
29314a6118
feat: consistent global error handling and logging
2021-02-04 13:15:17 +00:00
d5ebf9c3da
chore: Update deps again ( #738 )
2021-02-04 06:02:05 -05:00
6906e6bdd3
refactor: standardize erorr handling in rpc service ( #728 )
2021-02-03 14:50:00 -05:00
d66eae1a44
feat: Implement TableProvider for Trait for `Db` ( #730 )
...
* feat: Implement TableProvider for Db
Gets us selection pushdown in plans, sets us up for predicate pushdown
Includes: SendableRecordBatchStreams for mutable buffer and read buffer results
fixup snapshots
* docs: comments
2021-02-03 14:18:47 -05:00
4b524dbb35
Merge pull request #712 from influxdata/jg/flight-server
2021-02-03 10:25:51 -05:00
1f1ebefd7a
fix: Switch to database name instead of org/bucket for the Flight API
2021-02-03 10:02:06 -05:00
fc091041c5
fix: Propagate query planning error instead of unwrapping
2021-02-03 09:56:56 -05:00
b0a3ac1a65
docs: Leave a TODO note
2021-02-03 09:56:56 -05:00
19673657c2
refactor: Extract the Flight server into its own module
2021-02-03 09:56:56 -05:00
51d18e6ac7
refactor: Move GrpcService definition up a module
2021-02-03 09:56:56 -05:00
9ff2ee99d8
refactor: Move gRPC make_server up a module
2021-02-03 09:56:56 -05:00
8975b1cbf5
fix: into_iter in the assert_table_eq macro
...
This makes it so the macro can take `&[String]` or `&[&str]`.
2021-02-03 09:56:56 -05:00
39338b195b
feat: Basic scaffolding of a flight server do_get
2021-02-03 09:56:56 -05:00
95d5f759df
chore: Organize use imports
2021-02-03 09:56:56 -05:00
a5e09366b0
feat: Export arrow-flight from arrow-deps
2021-02-03 09:56:56 -05:00
4954cef08d
Merge pull request #675 from influxdata/cn+jg/osp-types
2021-02-03 09:56:30 -05:00
0f8ef9c7d5
Merge branch 'main' into cn+jg/osp-types
2021-02-03 09:09:04 -05:00
abc26a33c1
chore: Update dependencies (again) ( #718 )
...
* chore: Update dependencies (again)
* refactor: update for changes in DataFusion API
* fix: fmt
* fix: clippy
2021-02-02 18:33:01 -05:00
47101896c8
Merge pull request #719 from mkmik/editorconfig
...
style: Add editor config
2021-02-02 17:34:59 +00:00
Carol (Nichols || Goulding)
Paul Dix
Andrew Lamb
Raphael Taylor-Davies
Marko Mikulicic
Paul Dix
Carol (Nichols || Goulding)
Jake Goulding
Raphael Taylor-Davies
Dom