So that we don't have to skip running tests when cargo hakari makes a
change, but we won't have infinite cargo hakari commits if there's some
sort of problem.
* chore: remove references to perf_image in CI
* chore: adding gitops adapter image build in CI
* chore: gitops adapter bin now same as dir & package so docker build works
* fix: circle config package change after renaming gitops adapter package
* chore: remove references to perf_image in CI
* chore: typos in circle config comments
* chore: restored ability to build branch using parameter; documented it
* chore: fixed indentation faux-pas in circle config
* docs: clarified build-triggering instructions in circle config comments
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
All features are now covered by rskafka. This also removes the need to
specify a server ID for write buffer consumers. This was only used for
rdkafka since there we needed to specify a consumer group, even though
we did not use any transactions.
This removes 3 "nonexisting region" tests that where testing very
specific error behavior that no local emulator (minio and localstack)
replicate and that don't add much value. It's better to test our AWS
code at all than being to picky.
This will break `perf_image` until the new CI image is built due to the
newly required `--all-tags` parameter to `docker push` that isn't
available for the docker version we run on buster.
Use `codegen-units = 1`, thin-LTO and debug section compression to make our binary smaller (which is good for deploy and
test times) and faster.
# Summary
The binary size of `influxdb_iox` after building with:
```console
$ cargo build --release --no-default-features --features="aws,gcp,azure,jemalloc_replacing_malloc"
```
The profile was:
```toml
[profile.release]
debug = true
```
The commit was:
```text
89ece8b493
```
The size results are:
| Method | Size |
| ------------------------------------------ | ----- |
| baseline | 833MB |
| baseline + dbg compression | 222MB |
| baseline + strip | 49MB |
| codegen-units | 520MB |
| codegen-units + strip | 40MB |
| codegen-units + dbg compression | 143MB |
| thin LTO | 715MB |
| thin LTO + strip | 49MB |
| thin LTO + dbg compression | 199MB |
| codegen-units + thin LTO | 449MB |
| codegen-units + thin LTO + strip | 40MB |
| codegen-units + thin LTO + dbg compression | 130MB |
For the methods that were successfully measured I couldn't really see any compile time differences on my laptop.
# Methods
## Strip
Remove debug symbols. We don't really want this, so this is just to get an idea of the size
```console
$ strip baseline
```
## Debug Sections compression
Debug sections make a large amount of our binary size (a stripped executable is 49MB instead of 833MB). Since we like to
have debug symbols we cannot just strip them. However these symbols are only used for:
- backtrace generation (something went wrong, not BAU)
- profiling
- debugging
So in normal operation and most test scenarios, we're just wasting memory. So we could compress them:
```console
$ objcopy --compress-debug-sections baseline baseline-dbg_compressed
```
There is also elfutils:
```console
$ eu-elfcompress test
```
Elfutils nearly ends up with the same size (220MB instead of 222MB that objcopy achieves), but takes more time and is
probably not worth it.
Note that compressed debug sections exist since many years. The Rust ecosystem supports reading them since over a year,
see:
- <https://github.com/gimli-rs/gimli/issues/195>
- <https://github.com/rust-lang/backtrace-rs/issues/342>
## Codegen Units
The rust compiler parallelizes codegen work. This split into units however means that optimizations are somewhat
limited. This can be change by:
```toml
[profile.release]
...
codegen-units = 1
```
As a nice side effect this should also make our code faster.
## Thin LTO
Get LLVM to run "thin" Link Time Optimization:
```toml
[profile.release]
...
lto = "thin"
```
As a nice side effect this should also make our code faster.
## Fat LTO
Get LLVM to run "fat" Link Time Optimization:
```toml
[profile.release]
...
lto = "fat"
```
There are no results for this because this took a massive amount of memory and CPU time and did not finish on my system.
Carol (Nichols || Goulding)
Raphael Taylor-Davies
Marko Mikulicic
Dom Dwyer
jaymebrd
Luke Bond
Jacob Marble
Marco Neumann
Paul Dix
kodiakhq[bot]