This will break `perf_image` until the new CI image is built due to the
newly required `--all-tags` parameter to `docker push` that isn't
available for the docker version we run on buster.
Use `codegen-units = 1`, thin-LTO and debug section compression to make our binary smaller (which is good for deploy and
test times) and faster.
# Summary
The binary size of `influxdb_iox` after building with:
```console
$ cargo build --release --no-default-features --features="aws,gcp,azure,jemalloc_replacing_malloc"
```
The profile was:
```toml
[profile.release]
debug = true
```
The commit was:
```text
89ece8b493
```
The size results are:
| Method | Size |
| ------------------------------------------ | ----- |
| baseline | 833MB |
| baseline + dbg compression | 222MB |
| baseline + strip | 49MB |
| codegen-units | 520MB |
| codegen-units + strip | 40MB |
| codegen-units + dbg compression | 143MB |
| thin LTO | 715MB |
| thin LTO + strip | 49MB |
| thin LTO + dbg compression | 199MB |
| codegen-units + thin LTO | 449MB |
| codegen-units + thin LTO + strip | 40MB |
| codegen-units + thin LTO + dbg compression | 130MB |
For the methods that were successfully measured I couldn't really see any compile time differences on my laptop.
# Methods
## Strip
Remove debug symbols. We don't really want this, so this is just to get an idea of the size
```console
$ strip baseline
```
## Debug Sections compression
Debug sections make a large amount of our binary size (a stripped executable is 49MB instead of 833MB). Since we like to
have debug symbols we cannot just strip them. However these symbols are only used for:
- backtrace generation (something went wrong, not BAU)
- profiling
- debugging
So in normal operation and most test scenarios, we're just wasting memory. So we could compress them:
```console
$ objcopy --compress-debug-sections baseline baseline-dbg_compressed
```
There is also elfutils:
```console
$ eu-elfcompress test
```
Elfutils nearly ends up with the same size (220MB instead of 222MB that objcopy achieves), but takes more time and is
probably not worth it.
Note that compressed debug sections exist since many years. The Rust ecosystem supports reading them since over a year,
see:
- <https://github.com/gimli-rs/gimli/issues/195>
- <https://github.com/rust-lang/backtrace-rs/issues/342>
## Codegen Units
The rust compiler parallelizes codegen work. This split into units however means that optimizations are somewhat
limited. This can be change by:
```toml
[profile.release]
...
codegen-units = 1
```
As a nice side effect this should also make our code faster.
## Thin LTO
Get LLVM to run "thin" Link Time Optimization:
```toml
[profile.release]
...
lto = "thin"
```
As a nice side effect this should also make our code faster.
## Fat LTO
Get LLVM to run "fat" Link Time Optimization:
```toml
[profile.release]
...
lto = "fat"
```
There are no results for this because this took a massive amount of memory and CPU time and did not finish on my system.
* feat: migrate server to DbWrite (#2724)
* chore: print perf log output
* fix: don't suppress CI status code
* chore: review feedback
* fix: don't error on empty line protocol write payloads
* fix: test
* fix: test
* feat: Do not rebuild query_tests if .sql or .expected change
* feat: Add CI check
* refactor: move some sql tests to .sql files
* tests: port tests / expected results to data files
* fix: restore old name check-flatbuffers
For now we ignore warnings (e.g. "crate is yanked" or "crate is
unmaintained") because:
- internal crates w/ names of crates.io crates (e.g. `query`) are
treated like crates.io-crates even though they shouldn't, see
https://github.com/rustsec/rustsec/issues/232
- many crates are currently unmaintained and require a bit of upstream
work (e.g. `chrono` is currently not very active but uses an old
version of `time` which uses the unmaintained `stdweb`)
Closes#2575.
* fix: perf broken after proto changes
* feat: add perf to CI
* chore: try different python version selector script
* chore: several fixes to CI
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
* feat: add jaeger and otlp flags
* chore: add jaeger and otlp features to CI test and deploy image
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
* feat: make aws, gcp, azure dependencies optional
* fix: only run object store tests if the features are enabled
* fix: clean up testing
* fix: rename step
* fix: add to list of jobs
* fix: remove test with object store
* fix: review comments
The `Cargo.toml` file already says:
```toml
[profile.release]
debug = true
```
but we were overriding it in the circleci release script to include only line number info,
probably to shave off some time from the total build time.
The ability to debug a production binary outweighs the few seconds or even a minute overhead it takes to
add the debug infos. If building with full debug infos take much much longer, next time we disable them
it would be nice to include such a dramatic fact in the comments next to where we disable debug infos :-)
If the layer is cached, docker build won't print the output of the RUN command.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Adds a GitHub CI action:
* Uses the rust-toolchain file to install the requested version.
* Performs a build with default cargo target
* Runs workspace tests
* Checks cargo fmt is happy
* Runs clippy for lints, denying warnings
All commands taken from the existing CircleCI pipeline.
Currently no caching / reuse of build container.
* feat: Add parquet writer, hook up conversion in dstool
* fix: use bigger executor for test
* fix: less cloning
* fix: make unsupported messages less pejorative
* fix: fmt
* fix: Rename writer and do not require std::File, add example
* fix: clippy and fmt
* fix: remove unnecessary module in end to end tests
* fix: remove strange use of tempfile
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: cleanup use
* fix: Use more specific error messages
* fix: comment tweak
* fix: touchup temp path creation
* fix: clippy!
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
By default, the RocksDB C library is compiled using the number of
cores on the machine. In CircleCI, this is 36 cores. Unfortunately,
that appears to completely blow out the memory usage, causing the C
compiler invocations to be killed.
This commit reduces the *entire* parallelism of the build to avoid
that.
This configures a circleci pipeline that runs fmt, lint, test, and build operations.
I changed the fmt command to --check since --overwrite is no longer supported.
The pipeline will always run on nightly builds of rust.
Marco Neumann
Raphael Taylor-Davies
Carol (Nichols || Goulding)
kodiakhq[bot]
Andrew Lamb
Nga Tran
Jacob Marble
Marko Mikulicic
Marko Mikulicic
Andrew Lamb
Edd Robinson
Carol (Nichols || Goulding)
Luke Bond
Raphael Taylor-Davies
Dom
AJ Bond
Jake Goulding
Jake Goulding