* chore: bump testcontainers to v0.15.0
* chore: run go mod tidy
* chore: update test to latest version of testcontainers
* chore: update package
* fix: use collectors.NewGoCollector instead
SA1019 detected by staticcheck
* feat: start work on remotes/replications phone home data
* feat: add remotes/replications phone home data (no tests
* refactor: use erroring binary conversions
* style: gofmt
* refactor: improve some error handling
* style: cleanup
* feat: add tests
* refactor: just list remotes/replications rather than decrement
* chore: linting fix
Co-authored-by: DStrand1 <dstrandboge@influxdata.com>
Flux HTTP and template fetching requests do not perform IP address
checks for local addresses. This behavior on the one hand allows SSRF
(Server Side Request Forgery) attacks via authenticated requests but on
the other hand is useful for scenarios that have legitimate requirements
to fetch from private addresses (eg, hosting templates internally or
performing flux queries to local resources during development).
To not break existing installations, the default behavior will remain
the same but a new --hardening-enabled option is added to influxd to
turn on IP address verification and limit both flux and template
fetching HTTP requests to non-private addresses. We plan to enable new
security features that aren't suitable for the default install with this
option. Put another way, this new option is intended to be used to make
it easy to turn on all security options when running in production
environments. The 'Manage security and authorization' section of the
docs will also be updated for this option.
Specifically for flux, when --hardening-enabled is specified, we now
pass in PrivateIPValidator{} to the flux dependency configuration. The
flux url validator will then tap into the http.Client 'Control'
mechanism to validate the IP address since it is called after DNS lookup
but before the connection starts.
For pkger (template fetching), when --hardening-enabled is specified,
the template parser's HTTP client will be configured to also use
PrivateIPValidator{}. Note that /api/v2/stacks POST ('init', aka create)
and PATCH ('update') only store the new url to be applied later with
/api/v2/templates/apply. While it is possible to have InitStack() and
UpdateStack() mimic net.DialContext() to setup a go routine to perform a
DNS lookup and then loop through the returned addresses to verify none
are for a private IP before storing the url, this would add considerable
complexity to the stacks implementation. Since the stack's urls are
fetched when it is applied and the IP address is verified as part of
apply (see above), for now we'll keep this simple and not validate the
IPs of the stack's urls during init or update.
Lastly, update pkger/http_server_template_test.go's Templates() test for
disabled jsonnet to also check the contents of the 422 error (since the
flux validator also returns a 422 with different message). Also, fix the
URL in one of these tests to use a valid path.
Fixes a few issues:
* flux needs to write to the replication service, instead of the engine directly.
* the replication service incorrectly had value receiver methods, I think this
was just an accident. Pointer receivers make things easier to reason about. Also
with value receivers flux was not picking up the replication config properly.
* The flux to() function previously did not receive the org properly for internal
writes. Previously this was not necessary as the write path only needs the bucket
ID at this level (after authentication). But now we need the org id to look up
replications properly.
Closes#23183
* fix: remove nats for scraper processing
Scrapers now use go channels instead of NATS and interprocess communication.
This should fix#23085 .
Additionally, found and fixed#23106 .
* chore: fix formatting
* chore: fix static check and go.mod
* test: fix some flaky tests
* fix: mark NATS arguments as deprecated
* feat: api/v2/config endpoint for runtime config
* feat: use a type switch
* fix: add tests
* chore: add config key to returned json
* chore: update swagger ref
* feat: error out when config file contains invalid options
* feat: debug logging when loading a config file
* fix: only detect flags from 1.x
* test: update tests to use toml configs
* feat: added replications queue management to launcher tasks
* refactor: separated sql logic into replications service rather than durable queue manager
* refactor: extended replications feature flag to launcher code and minor change to startup function param
* chore: added unit test coverage for replications server startup queue management
* refactor: made error messages reusable and factored out unecessary string from queue management tests
* refactor: changed queue management error names to pass linter check
* feat: sql down migrations
* refactor: different name for up migrations
* chore: update migrations ref in svc tests
* build: add lint step to verify sql migration names match
* feat: added durable queue management to replications service
* refactor: improved mapping of replication streams to durable queues
* refactor: modified replication stream durable queues to use user-specified engine path
* chore: generated test mocks for replications DurableQueueManager
* chore: add test coverage for replications durable queue manager
* refactor: made changes based on code review, added mutex to durableQueueManager, improved error logging
* chore: ran make fmt
* refactor: further improvements to error logging
* Modify launcher shutdown to reduce field count
* Move tracing setup into its own method
* Move metastore setup into its own method
* Delete unused getter methods
* Fix imports
* feat: new metadata backup endpoint
* feat: added restore/sql API endpoint
* fix: content-type is multipart/mixed, part names are kv and sql
* fix: changed multipart manifest to buckets and made it .json
* feat: added lock for backing up sqlite and bolt dbs
* fix: use read lock instead of write lock on kv during backup
* fix: use filepath.Join for temp dirs
* feat(notebooks): demo store functionality
* feat: demo store with context and middleware
* refactor: without context
* fix: update endpoint and status code for delete
* chore: changed name
* chore: added some tests
* fix: address review comments
Manuel de la Peña
Jeffrey Smith II
Owen Nelson
Sam Arnold
Jamie Strandboge
Jonathan A. Sternberg
William Baker
Daniel Moran
mcfarlm3