* test: remove group skips
* test: inline previously extended testcases that broke with added sort
Some tests had a sort added so they'd pass in cloud, vanilla, and OSS.
This also broke some extended testcases here in OSS since pushdown rules
no longer match the plan.
This diff "inlines" the testcase body for each of these so we get our
coverage back, but leaves a FIXME to note that it would be better if we
could somehow thread the needle and retain the extension.
Filters that contain `or` may change between cursor resets so we must remember to update the condition in the read cursor.
```flux
|> filter(fn: (r) => ((r["_field"] == "field1" and r["_value"]==true) or (r["_field"] == "field2" and r["_value"] == false)))
```
Closes https://github.com/influxdata/flux/issues/4804
* build(flux): update flux to v0.171.0
* chore: remove testing.loadStorage from tests
Also update skip lists in Flux test harness
* chore: remove now redundant Go end-to-end unit tests
This testing is all now provided by the `fluxtest` harness.
This changes the statistics merging to use the `Add` method on the
entire statistics struct instead of only the metadata. The purpose of
this is that we want to utilize the existing function for merging the
statistics so that additional properties can be added without modifying
the query controller.
At the same time, there are certain properties that are computed in the
controller and we want to ensure they aren't double counted if flux
starts computing these itself. So we blacklist certain attributes that
we compute so that if flux is modified to return these values, we just
ignore them until we change our own code to use those values.
In immediate terms, we're changing to use the `Add` method so that we
can add profiles to the statistics and have those profiles propagate to
the query controller. This property doesn't exist yet so we can't add it
and we don't want to add it after flux is modified because it could
break the operator profile.
But, we also don't want to only use `Add` because we want to move the
properties such as `TotalAllocated` and `TotalDuration` into flux itself
and to remove the controller. But, the controller needs to be compatible
with whatever changes we make to flux so that there's no circumstance
where functionality stops working.
* fix(fluxtest): update Flux tests for new option support
The Flux test harness now allows inheriting options, this updates the
test cases with the new syntax and simplifies any tests that had to
duplicate the options.
* build(flux): update flux to v0.165.0
* chore: upgrade flux to v0.167.0
Co-authored-by: Nathaniel Cook <nvcook42@gmail.com>
Co-authored-by: Paul Hummer <paul@eventuallyanyway.com>
Flux HTTP and template fetching requests do not perform IP address
checks for local addresses. This behavior on the one hand allows SSRF
(Server Side Request Forgery) attacks via authenticated requests but on
the other hand is useful for scenarios that have legitimate requirements
to fetch from private addresses (eg, hosting templates internally or
performing flux queries to local resources during development).
To not break existing installations, the default behavior will remain
the same but a new --hardening-enabled option is added to influxd to
turn on IP address verification and limit both flux and template
fetching HTTP requests to non-private addresses. We plan to enable new
security features that aren't suitable for the default install with this
option. Put another way, this new option is intended to be used to make
it easy to turn on all security options when running in production
environments. The 'Manage security and authorization' section of the
docs will also be updated for this option.
Specifically for flux, when --hardening-enabled is specified, we now
pass in PrivateIPValidator{} to the flux dependency configuration. The
flux url validator will then tap into the http.Client 'Control'
mechanism to validate the IP address since it is called after DNS lookup
but before the connection starts.
For pkger (template fetching), when --hardening-enabled is specified,
the template parser's HTTP client will be configured to also use
PrivateIPValidator{}. Note that /api/v2/stacks POST ('init', aka create)
and PATCH ('update') only store the new url to be applied later with
/api/v2/templates/apply. While it is possible to have InitStack() and
UpdateStack() mimic net.DialContext() to setup a go routine to perform a
DNS lookup and then loop through the returned addresses to verify none
are for a private IP before storing the url, this would add considerable
complexity to the stacks implementation. Since the stack's urls are
fetched when it is applied and the IP address is verified as part of
apply (see above), for now we'll keep this simple and not validate the
IPs of the stack's urls during init or update.
Lastly, update pkger/http_server_template_test.go's Templates() test for
disabled jsonnet to also check the contents of the 422 error (since the
flux validator also returns a 422 with different message). Also, fix the
URL in one of these tests to use a valid path.
Fixes a few issues:
* flux needs to write to the replication service, instead of the engine directly.
* the replication service incorrectly had value receiver methods, I think this
was just an accident. Pointer receivers make things easier to reason about. Also
with value receivers flux was not picking up the replication config properly.
* The flux to() function previously did not receive the org properly for internal
writes. Previously this was not necessary as the write path only needs the bucket
ID at this level (after authentication). But now we need the org id to look up
replications properly.
Closes#23183
* build(flux): update flux to v0.156.0
* chore(flux/schema): update schema tests to assert planner rules
The schema test where updated in Flux, this updates them here so that we
can assert that the planner rules are applied. See note about copied
data.
Co-authored-by: Nathaniel Cook <nvcook42@gmail.com>
* feat: Add WITH KEY to show tag keys
* fix: add tests for multi measurement tag value queries
* chore: fix linter problems
* chore: revert influxql changes to keep WITH KEY disabled
* chore: add TODO for moving flux tests to flux repo
Co-authored-by: Sam Arnold <sarnold@influxdata.com>
This updates the flux integration to use the `flux/array` package rather
than directly using the `arrow/array` package.
Flux recently switched to wrapping the array types from arrow and
creating its own array package to be used for table columns instead of
directly referencing the arrow package. This allows us to keep a
consistent interface, but potentially change internal implementations
without changing downstream consumers. Most recently, the
`*array.String` type has some of its own optimizations for certain array
patterns.
This change updates the flux integration to use the new API.
* fix: field metaqueries take fast path if predicate is only on `_measurement`
* chore: update CHANGELOG
* test: add test for fields with measurement predicate
The kind for experimental-to is in conflict with the one used by native
flux. In the past, native flux didn't use the kind and expected
implementations to register their own procedure spec with that value.
In the latest Flux, this isn't the case and it is using the kind for a
native implementation. As we have not implemented the requirements for
the native implementation, we are still replacing the procedure spec but
using a different name that's unique to our implementation.
Jeffrey Smith II
Nathaniel Cook
Owen Nelson
Christopher M. Wolff
Jonathan A. Sternberg
Markus Westerlind
Brandon Pfeifer
Alfonso Acosta
Dane Strandboge
Jamie Strandboge
Sam Arnold
Paul Hummer
William Baker
Andrew Lamb
Dane Strandboge
Daniel Moran