Still insert them into the database and associate them with namespaces,
but don't ever query them back out.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Adds a single-tenant mode (CST) to the IOx routers.
Single-tenancy mode differs in two main ways:
* V1 write endpoint is partially supported
* V2 write endpoint ignores "org" parameter
The "normal" mode is "multi tenant" which is the default operational
mode, and all existing behaviour remains unchanged. Single tenant mode
can be enabled by specifying INFLUXDB_IOX_SINGLE_TENANCY=true.
Request parsing is delegated to two implementations of the
WriteParamExtractor trait, one each for CST and MT - the logic of each
"mode" is defined within these files and all other functionality is
common between the two.
This commit also renames some of the error types for clarity
(NoSpecified -> NoOrgBucketSpecified, other NotSpecified ->
NoQueryParams, etc).
Note: single tenant code requires testing
There was a mix of different ways of returning errors - this commit
unifies them, adds some documentation to the returned errors, and
removes the capitalisation.
Errors should be lower-case so they compose nicely like this:
"something failed: super important error: inner error"
rather than:
"something failed: Super important error: Inner error"
Adds proptests to assert correctness of set operations against a
SequenceNumberSet, by ensuring they match those of a known-good
implementation from the stdlib (HashSet).
Ensure (de-)serialisation correctness by asserting a round trip results
in equal sets.
Prior to this commit, the SequenceNumberSet accepted values up to
u32::MAX (approx 4.2 billion) which works out to be ~50 days of
continious ID allocation at 1k writes/s.
This commit changes the SequenceNumberSet to accept the full range of a
u64, with the caveat that outside of the u32 range, values may be
ordered incorrectly. See:
https://github.com/influxdata/influxdb_iox/issues/7260
Fortunately values from SequenceNumberSet are used as an identity, and
never for ordering. We should remove the PartialOrd bounds on
SequenceNumber once Kafka has been cleaned up to encode this in the type
system.
* feat: split start-level files that overlap wiht many files
* test: split files and theit split times
* test: split test for L1 and L2 files
* feat: full implementation that support large-size overlapped files
* chore: modify comments to reflect the changes
* fix: typo
* chore: update test output
* docs: clearer comments
* chore: remove empty test files. Will add in in a separate PR
* chore: Apply suggestions from code review
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
* chore: address review comments
* chore: Apply suggestions from code review
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
* refactor: add a knob to turn large-size overlaps on and off
* fix: typo
* chore: update test output after merging main
* fix: split_times should not include the max_time of the file
* fix: fix an overlap bug while limitting number of files to compact
* test: unit tests for different overlap cases of limit files to compact
* chore: increase time range of the tests to let the split files work correctly
* fix: skip compacting files of tiny ranges
* test: add tests for time range 1
* chore: address review comments
* chore: remove enable_large_size_overlap_files knob
* fix: fix a bug that sort L1 files in thier min_time instead of l0_max_created_at
* refactor: use the same order_files function afer merging main into branch
* chore: typos and clearer comments
* chore: remove obsolete comments
* chore: add asserts per review suggestion
---------
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
The maximum number of tables is part of the Namespace, which is already
loaded in its entirety. This commit copies the value into the
NamespaceSchema, making it available for the router to utilise.
* feat: introduce a new way of max_sequence_number for ingester, compactor and querier
* chore: cleanup
* feat: new column max_l0_created_at to order files for deduplication
* chore: cleanup
* chore: debug info for chnaging cpu.parquet
* fix: update test parquet file
Co-authored-by: Marco Neumann <marco@crepererum.net>
* feat: function to get parttion candidates from partition table
* chore: cleanup
* fix: make new_file_at the same value as created_at
* chore: cleanup
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Carol (Nichols || Goulding)
Carol (Nichols || Goulding)
wiedld
wiedld
Dom Dwyer
Andrew Lamb
Marco Neumann
Nga Tran
dependabot[bot]
Stuart Carnie
Marko Mikulicic