This changes the statistics merging to use the `Add` method on the
entire statistics struct instead of only the metadata. The purpose of
this is that we want to utilize the existing function for merging the
statistics so that additional properties can be added without modifying
the query controller.
At the same time, there are certain properties that are computed in the
controller and we want to ensure they aren't double counted if flux
starts computing these itself. So we blacklist certain attributes that
we compute so that if flux is modified to return these values, we just
ignore them until we change our own code to use those values.
In immediate terms, we're changing to use the `Add` method so that we
can add profiles to the statistics and have those profiles propagate to
the query controller. This property doesn't exist yet so we can't add it
and we don't want to add it after flux is modified because it could
break the operator profile.
But, we also don't want to only use `Add` because we want to move the
properties such as `TotalAllocated` and `TotalDuration` into flux itself
and to remove the controller. But, the controller needs to be compatible
with whatever changes we make to flux so that there's no circumstance
where functionality stops working.
* fix(query): accept queue-size > 0 when concurrency = 0
* fix(influxd): revert defaults for query settings to avoid validation err
* test: lower the default query concurrency used by test launchers
A static initialization is not desirable in the main binaries, as it forces all
paths of code to init, but it is still useful in tests. It allows static
intialization to be performed once for all tests and eliminates the need to
always add the FluxInit call. Added a fluxinit/static package that calls
fluxinit.FluxInit() to replace the builtin package. This hides the nature of
the initialization and makes it clear that it is mandatory initialization code
getting called.
I think it is a nice utility function that I would like to use
elsewhere.
So I decided to move it out to the tracing package.
Signed-off-by: Gianluca Arbezzano <gianarb92@gmail.com>
This removes the spec and updates the lang package usage to make use of
passing in the runtime as a parameter.
It removes all direct dependendencies on the flux runtime from the http
package.
* chore: Remove several instances of WithLogger
* chore: unexport Logger fields
* chore: unexport some more Logger fields
* chore: go fmt
chore: fix test
chore: s/logger/log
chore: fix test
chore: revert http.Handler.Handler constructor initialization
* refactor: integrate review feedback, fix all test nop loggers
* refactor: capitalize all log messages
* refactor: rename two logger to log
The controller now supports setting an initial memory limit and setting
a maximum amount of memory that the controller may use separately from
the memory quota per query and the concurrency quota.
This allows the controller to increase the concurrency quota to a larger
number while setting the maximum amount of memory to a lower amount than
would be required for all queries to use 100% of their allowable memory.
Functionally, this means that a query will have a soft limit for an
initial memory byte quota that a query is guaranteed to have, a shared
pool that it is allowed access to in the case it uses more, and a hard
limit that no query may exceed to prevent runaway queries from taking
over the entire pool.
This change is completely backwards compatible with older configurations
as the new options will default to values that mimic the old behavior
where a query is allocated the full amount of its memory quota and the
maximum amount of memory is based on the concurrency quota and this
maximum memory quota.
In addition to the above, this also fixes a bug in the controller that
allowed it to run more than its concurrency as executing queries. This
happened when the results had finished being sent by the executor, but
the query had not yet been read and/or serialized. The executor would be
freed up and would take the next query even though the previous query
hadn't yet been finalized with `Done()`.
The http error schema has been changed to simplify the outward facing
API. The `op` and `error` attributes have been dropped because they
confused people. The `error` attribute will likely be readded in some
form in the future, but only as additional context and will not be
required or even suggested for the UI to use.
Errors are now output differently both when they are serialized to JSON
and when they are output as strings. The `op` is no longer used if it is
present. It will only appear as an optional attribute if at all. The
`message` attribute for an error is always output and it will be the
prefix for any nested error. When this is serialized to JSON, the
message is automatically flattened so a nested error such as:
influxdb.Error{
Msg: errors.New("something bad happened"),
Err: io.EOF,
}
This would be written to the message as:
something bad happened: EOF
This matches a developers expectations much more easily as most
programmers assume that wrapping an error will act as a prefix for the
inner error.
This is flattened when written out to HTTP in order to make this logic
immaterial to a frontend developer.
The code is still present and plays an important role in categorizing
the error type. On the other hand, the code will not be output as part
of the message as it commonly plays a redundant and confusing role when
humans read it. The human readable message usually gives more context
and a message like with the code acting as a prefix is generally not
desired. But, the code plays a very important role in helping to
identify categories of errors and so it is very important as part of the
return response.
We are planning to change the allocator interface within flux to use the
arrow allocator. To make the release easier, this updates the test in
advance to use the arrow allocator instead of the to be changed memory
allocator interface from flux.
The secret service is tested by creating a secret and then attempting to
use it in a flux query. There is one test where accessing the secret
should work and one where it should return that the action is forbidden.
Jonathan A. Sternberg
Alfonso Acosta
William Baker
Daniel Moran
Sam Arnold
Dan Moran
Adrian Thurston
Paul Hummer
Christopher M. Wolff
Brett Buddin
Stuart Carnie
Yiqun (Ethan) Zhang
Paul Hummer
Gianluca Arbezzano
Lorenzo Affetti
Gianluca Arbezzano
Alirie Gray
Gavin Cabbage
Jacob Marble
Nathaniel Cook
docmerlin (j. Emrys Landivar)