Use `codegen-units = 1`, thin-LTO and debug section compression to make our binary smaller (which is good for deploy and
test times) and faster.
# Summary
The binary size of `influxdb_iox` after building with:
```console
$ cargo build --release --no-default-features --features="aws,gcp,azure,jemalloc_replacing_malloc"
```
The profile was:
```toml
[profile.release]
debug = true
```
The commit was:
```text
89ece8b493
```
The size results are:
| Method | Size |
| ------------------------------------------ | ----- |
| baseline | 833MB |
| baseline + dbg compression | 222MB |
| baseline + strip | 49MB |
| codegen-units | 520MB |
| codegen-units + strip | 40MB |
| codegen-units + dbg compression | 143MB |
| thin LTO | 715MB |
| thin LTO + strip | 49MB |
| thin LTO + dbg compression | 199MB |
| codegen-units + thin LTO | 449MB |
| codegen-units + thin LTO + strip | 40MB |
| codegen-units + thin LTO + dbg compression | 130MB |
For the methods that were successfully measured I couldn't really see any compile time differences on my laptop.
# Methods
## Strip
Remove debug symbols. We don't really want this, so this is just to get an idea of the size
```console
$ strip baseline
```
## Debug Sections compression
Debug sections make a large amount of our binary size (a stripped executable is 49MB instead of 833MB). Since we like to
have debug symbols we cannot just strip them. However these symbols are only used for:
- backtrace generation (something went wrong, not BAU)
- profiling
- debugging
So in normal operation and most test scenarios, we're just wasting memory. So we could compress them:
```console
$ objcopy --compress-debug-sections baseline baseline-dbg_compressed
```
There is also elfutils:
```console
$ eu-elfcompress test
```
Elfutils nearly ends up with the same size (220MB instead of 222MB that objcopy achieves), but takes more time and is
probably not worth it.
Note that compressed debug sections exist since many years. The Rust ecosystem supports reading them since over a year,
see:
- <https://github.com/gimli-rs/gimli/issues/195>
- <https://github.com/rust-lang/backtrace-rs/issues/342>
## Codegen Units
The rust compiler parallelizes codegen work. This split into units however means that optimizations are somewhat
limited. This can be change by:
```toml
[profile.release]
...
codegen-units = 1
```
As a nice side effect this should also make our code faster.
## Thin LTO
Get LLVM to run "thin" Link Time Optimization:
```toml
[profile.release]
...
lto = "thin"
```
As a nice side effect this should also make our code faster.
## Fat LTO
Get LLVM to run "fat" Link Time Optimization:
```toml
[profile.release]
...
lto = "fat"
```
There are no results for this because this took a massive amount of memory and CPU time and did not finish on my system.
* feat: migrate server to DbWrite (#2724)
* chore: print perf log output
* fix: don't suppress CI status code
* chore: review feedback
* fix: don't error on empty line protocol write payloads
* fix: test
* fix: test
* feat: sql down migrations
* refactor: different name for up migrations
* chore: update migrations ref in svc tests
* build: add lint step to verify sql migration names match
* build: install a specific version of git-cliff for changelog job
* build: use pre-built image
* chore: correct image name
* chore: update deps
* chore: remove test job
* feat: Do not rebuild query_tests if .sql or .expected change
* feat: Add CI check
* refactor: move some sql tests to .sql files
* tests: port tests / expected results to data files
* fix: restore old name check-flatbuffers
For now we ignore warnings (e.g. "crate is yanked" or "crate is
unmaintained") because:
- internal crates w/ names of crates.io crates (e.g. `query`) are
treated like crates.io-crates even though they shouldn't, see
https://github.com/rustsec/rustsec/issues/232
- many crates are currently unmaintained and require a bit of upstream
work (e.g. `chrono` is currently not very active but uses an old
version of `time` which uses the unmaintained `stdweb`)
Closes#2575.
* fix: perf broken after proto changes
* feat: add perf to CI
* chore: try different python version selector script
* chore: several fixes to CI
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
* fix: systemd service -- handle 40x and block indefinitely
* chore: update changelog
* chore: add pkg_run_test to CI
* chore: fix curl syntax for auth setup in pkg_run_test
* feat: add jaeger and otlp flags
* chore: add jaeger and otlp features to CI test and deploy image
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
* Regenerate protos using gogo 1.3.2
* Add protos to generate, add checkgenerate to CI
* Address proto warning
* Add generator tooling to Makefile
* Delete recursive Makefiles, simplify generation run by goreleaser
* Use env bash for fetch-ui-assets
* Add static-data to clean target
* feat: make aws, gcp, azure dependencies optional
* fix: only run object store tests if the features are enabled
* fix: clean up testing
* fix: rename step
* fix: add to list of jobs
* fix: remove test with object store
* fix: review comments
* chore: enable ingest performance tests (#21624)
* chore: add run script for performance tests
* chore: update circle config to run performance tests
* chore: adjustments to perf test config
* remove 1.8 branch here, as that branch is already configured for perf
tests
* remove extraneous comments in config and perftest script
* move perftest setup logic out of circle config and into a script
* use a specific ubuntu machine image for circle driver
* omit gzip/gunzip from perf test dataset manipulations
* chore: use r5.2xlarge instance type and put perf test dataset in ramdisk
* chore: fix query logic in nightly aws teardown (#21678)
* Stop using GOCACHE, it only ever grows in size and the upload/download ends up dominating job runtime
* Use RAMdisks in build/test steps where we expect it'll help runtime
The `Cargo.toml` file already says:
```toml
[profile.release]
debug = true
```
but we were overriding it in the circleci release script to include only line number info,
probably to shave off some time from the total build time.
The ability to debug a production binary outweighs the few seconds or even a minute overhead it takes to
add the debug infos. If building with full debug infos take much much longer, next time we disable them
it would be nice to include such a dramatic fact in the comments next to where we disable debug infos :-)
* feat: update e2e to use monitor-ci
* build: local copy of test script
* build: use stable docker image for e2e
* fix: use a working image for e2e ci job
If the layer is cached, docker build won't print the output of the RUN command.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
* feat: synchronize ui folder with ui repository
* docs: updated README in /ui to reflect new UI asset build process
* chore: update CHANGELOG
* chore: re-worded CHANGELOG message and added trailing newline to fetch_ui_assets.sh script
* fix: leveraged queryBuilder test code from ui repo to fix flaky queryBuilder test on chrome
* docs: added comments to fetch_ui_assets script explaining where the assets come from
* chore: empty commit to trigger CI
* Reduce cache usage across job types (I saw weird errors about files missing from paths in the GOCACHE)
* Split our Dockerfile into multiple stages, with all `RUN` commands in the 1st stage
* Uninstall any system-provided emulators before installing the recommended `binfmt` QEMU
* Move our Docker setup to the front of the cross-build steps
* Build the 1st stage of our Docker immediately after setting up the cross-builder, to fail fast on any issues
* Delete unused configs
* Delete unused test steps
* Reorder our main config file to start with top-level pipelines, and to group conceptually-related jobs
* Introduce new jobs to pull dependencies, to reduce duplicated work across test/lint/build steps
* Rename our caches to bust our current huge sizes
* Use unique names for caches per-step, to avoid conflicts
* Fix result collection for JS tests
* Create Dockerfile for running build & release in CI
* Update CircleCI config to use new Docker image
* Update goreleaser to add a linux ARM64 release
Adds a GitHub CI action:
* Uses the rust-toolchain file to install the requested version.
* Performs a build with default cargo target
* Runs workspace tests
* Checks cargo fmt is happy
* Runs clippy for lints, denying warnings
All commands taken from the existing CircleCI pipeline.
Currently no caching / reuse of build container.
* chore: update circleci config to use golang:1.15 images
* fix: Update Go version and use temporary override for Arrow
The Arrow override is to fix race detection problem. Once the
Arrow PR merges, this override can be removed.
Co-authored-by: Stuart Carnie <stuart.carnie@gmail.com>
- Update CIrcleCI configuration to start release process on an RC build
- Update .goreleaser.yml:
- Start building armel and armhf binaries and rpm and debian packages.
- Generate sha256 checksum file.
- launcher.go: do not use `max` module to escape integeroverflow problem for armel and armhf builds
- Start using `v0.142.0` of goreleaser
- Added pre and post install/uninstall scripts for rpm amd deb packages
Installing `osxcross` to `/opt` requires superuser permissions. This is
built as root in flux, but requires `sudo` for circleCI. Additionally,
there was a problem with `o64-clang` not being on the `PATH` that we
would have discovered the next time we tried to release.
* feat: Add parquet writer, hook up conversion in dstool
* fix: use bigger executor for test
* fix: less cloning
* fix: make unsupported messages less pejorative
* fix: fmt
* fix: Rename writer and do not require std::File, add example
* fix: clippy and fmt
* fix: remove unnecessary module in end to end tests
* fix: remove strange use of tempfile
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: cleanup use
* fix: Use more specific error messages
* fix: comment tweak
* fix: touchup temp path creation
* fix: clippy!
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
It was a request from boonito QA to start running nightly builds 2 hours earlier, instead of 7am UTC to start running nightly builds at 5am UTC (10pm PDT) to give them time to debug test failures.
* chore(litmus): run integration tests on master:
- aafter successful E2E run after master build, run integration tests
modified: .circleci/config.yml
modified: etc/litmus_fail_notify.sh
modified: etc/litmus_success_notify.sh
* chore(litmus): litmus_integration run should depend on litmus_daily test run
modified: .circleci/config.yml
This adds support for using pkg-config to build libflux inside of the
flux dependency. The build can occur by either installing `pkg-config`
into your path or the `env` script can be used to invoke it from the go
modules.
By default, the RocksDB C library is compiled using the number of
cores on the machine. In CircleCI, this is 36 cores. Unfortunately,
that appears to completely blow out the memory usage, causing the C
compiler invocations to be killed.
This commit reduces the *entire* parallelism of the build to avoid
that.
This configures a circleci pipeline that runs fmt, lint, test, and build operations.
I changed the fmt command to --check since --overwrite is no longer supported.
The pipeline will always run on nightly builds of rust.
* ci: run go race tests in parallel over 8 containers
Signed-off-by: Chris Goller <goller@gmail.com>
* ci: add timing hint for splitting go tests
Signed-off-by: Chris Goller <goller@gmail.com>
* ci: remove spurious grep
Signed-off-by: Chris Goller <goller@gmail.com>
* ci: mod=readonly should ignore unknown internal/promqltests path
Signed-off-by: Chris Goller <goller@gmail.com>
* ci: echo the packages tested
Signed-off-by: Chris Goller <goller@gmail.com>
* ci: remove readonly experiment
Signed-off-by: Chris Goller <goller@gmail.com>
* ci: filter promql internal tests
The promql internal tests has a special go.mod that shows
up in go list. That directory cannot be tested.
Signed-off-by: Chris Goller <goller@gmail.com>
* build: split go linting from tests
Signed-off-by: Chris Goller <goller@gmail.com>
* build: split js linting from tests
Signed-off-by: Chris Goller <goller@gmail.com>
* ci: run go tests without verbose option
Signed-off-by: Chris Goller <goller@gmail.com>
* ci: parallelize eslint with 8 containers
Signed-off-by: Chris Goller <goller@gmail.com>
* ci: cache node modules for eslint
Signed-off-by: Chris Goller <goller@gmail.com>
* ci: lint circle yaml
Signed-off-by: Chris Goller <goller@gmail.com>
* ci: cd into ui directory for node caching
Signed-off-by: Chris Goller <goller@gmail.com>
* ci: note eslint parallelization
Signed-off-by: Chris Goller <goller@gmail.com>
* ci: update yarn cache directory to global
Signed-off-by: Chris Goller <goller@gmail.com>
* ci: cache jstest yarn downloads
Signed-off-by: Chris Goller <goller@gmail.com>
* ci: note parallelization of jest tests
Signed-off-by: Chris Goller <goller@gmail.com>
Signed-off-by: Lorenzo Affetti <lorenzo.affetti@gmail.com>
Signed-off-by: Julius Volz <julius.volz@gmail.com>
move to internal
update flux to v0.50
Revert "move to internal"
This reverts commit bcd4caffbd44135f1dbeac4163cb2a22a751f45a.
promtests/internal --> internal/promtests
* chore: add new types for NotiicationRules
* build(client): remove client and build each build
* chore: update gitignore to exclude generated client
* fix: allow client dir to be empty
* docs: add README to client dir
* fix(build): add autogen client to eslintignore
* build: gen client for ci js tests
Daniel Moran
Sam Arnold
William Baker
kodiakhq[bot]
Marco Neumann
Raphael Taylor-Davies
Andrew Lamb
Cody Shepherd
pierwill
Nga Tran
Jacob Marble
Dane Strandboge
Marko Mikulicic
Marko Mikulicic
Andrew Lamb
Luke Bond
Luke Bond
Edd Robinson
Carol (Nichols || Goulding)
William Baker
Bucky Schwarz
Jonathan A. Sternberg
Raphael Taylor-Davies
Wojciech Kocjan
Dom
Gershon Shif
Russ Savage
Stuart Carnie
Stephanie Engel
Zoe Steinkamp
Adrian Thurston
Gavin Cabbage
karel-rehor
Alex Boatwright
Paul Hummer
AJ Bond
jlapacik
Jake Goulding
Ariel Salem
Chris Goller
Christopher M. Wolff
Jake Goulding
j. Emrys Landivar (docmerlin)
Yiqun Zhang
Julius Volz
Andrew Watkins