Commit Graph

1316 Commits (4f8adb6f6d47fbc41e9d2113c7dd561acd52c5cb)

Author SHA1 Message Date
Andriy Dzikh 3afe680c92 Use THE_COMMIT env var instead of ghprbActualCommit. 2021-07-07 13:15:33 -07:00
Andriy Dzikh 50e07ae7ec Move environment start saving up in minikube_set_pending. 2021-07-07 11:19:35 -07:00
Andriy Dzikh ae294d2067 Make sync_tests.sh remove any _integration suffix. 2021-07-07 09:21:00 -07:00
Medya Ghazizadeh 429b00d4f0
Merge pull request #11857 from ilya-zuyev/ilyaz/upload_arm64_to_gcs
fix kvm2-arm64 scripts
2021-07-02 10:10:27 -04:00
Medya Ghazizadeh a1300b1d32
Merge pull request #11874 from andriyDev/MoreCharts
More charts
2021-07-02 09:45:19 -04:00
Medya Ghazizadeh 5ecd53b1ea
Merge pull request #11794 from ilya-zuyev/ilyaz/test_report_links
Show GCS report links in integration test logs
2021-07-02 09:37:14 -04:00
Sharif Elgamal 7c45056827
Merge pull request #11851 from andriyDev/fix-flake-sync
Replace integration test "hooks" with standalone scripts
2021-07-01 16:17:54 -07:00
Andriy Dzikh 5d0eba44ec Add test count/fail count chart to environment page. 2021-07-01 16:02:22 -07:00
Andriy Dzikh a442d79877 Remove STARTED_ENVIRONMENTS from common.sh. 2021-07-01 15:53:52 -07:00
Andriy Dzikh d1b6549fbb Add duration graph to environment flake rates. 2021-07-01 14:29:19 -07:00
Ilya Zuyev 6e46d895dc Merge branch 'master' into ilyaz/test_report_links 2021-07-01 13:51:14 -07:00
Ilya Zuyev 5ccd38c0f7 fix output formatting 2021-07-01 13:47:40 -07:00
Medya Gh d2876b1414 auto bump kubernetes version using github actions 2021-07-01 16:32:39 -04:00
Andriy Dzikh 3904c6deab Ignore upload fail. 2021-07-01 13:32:20 -07:00
Andriy Dzikh 5dfb6726bd Make upload_tests use a remote GCS file. 2021-07-01 13:32:20 -07:00
Ilya Zuyev 3b5675d9e0 fix 2021-07-01 13:31:44 -07:00
Ilya Zuyev 2beaea9fc1 fix 2021-07-01 12:27:06 -07:00
Ilya Zuyev ea6d16db41 for PR builds: build kvm2-arm64 deb 2021-07-01 12:14:03 -07:00
Andriy Dzikh b5caf568ac Add growth column to flake list. 2021-07-01 12:01:09 -07:00
Andriy Dzikh d8ffe3f9fa Add rank column to flake rate list. 2021-07-01 11:47:40 -07:00
Andriy Dzikh a203d5f644 Replace integration test "hooks" with register_tests to fill started_environments and sync_tests to fill finished environments. 2021-07-01 11:09:34 -07:00
Steven Powell a9157c2433 improve Windows Docker clean slate 2021-06-30 16:42:09 -07:00
Medya Ghazizadeh c2f18d9d4b
Merge pull request #11841 from sharifelgamal/ugly-err
remove ugly error message from kicbase and ISO autobuild
2021-06-30 17:26:37 -04:00
Sharif Elgamal d8f8e3faf7
Merge pull request #11803 from sharifelgamal/mac-test
try less parallelization for macOS tests
2021-06-30 14:17:48 -07:00
Sharif Elgamal 06c1da3076 make URLs easier to copy 2021-06-30 14:01:14 -07:00
Medya Ghazizadeh a50406e269
Merge pull request #11767 from ilya-zuyev/ilyaz/arm64_kvm_release
Update release scripts to build arm64 version of kvm2 driver
2021-06-30 16:49:09 -04:00
Sharif Elgamal 1e7f8cbaa7 remove ugly error message from kicbase and ISO autobuild 2021-06-30 13:40:18 -07:00
Medya Ghazizadeh 323733c532
Merge pull request #11816 from andriyDev/FlakeRateSync
Report flakes only after synchronizing all integration tests
2021-06-30 16:38:08 -04:00
Medya Ghazizadeh b8596f1a13
Merge pull request #11811 from medyagh/go_mod_godame
bump go mod libs manually
2021-06-30 15:50:45 -04:00
Sharif Elgamal 6d75f2e7c8 Merge branch 'master' of github.com:kubernetes/minikube into mac-test 2021-06-29 16:37:54 -07:00
Sharif Elgamal 7030c8c0e4
Merge pull request #11814 from sharifelgamal/fixit
fix generate-docs github action
2021-06-29 15:10:08 -07:00
Andriy Dzikh 7ea94ed00a Add license and comment describing what the script does. 2021-06-29 14:51:31 -07:00
Andriy Dzikh e3883d0b56 Fix writing wrong files, and added better comments. 2021-06-29 14:47:13 -07:00
Sharif Elgamal 7fabb90eaf remove old script 2021-06-29 13:56:33 -07:00
Andriy Dzikh f2c64cdb2b Add support for syncing Docker_Windows. 2021-06-29 13:32:54 -07:00
Sharif Elgamal 5d11a87a7b Merge branch 'master' of github.com:kubernetes/minikube into mac-test 2021-06-29 13:27:14 -07:00
Medya Gh a714161f31 update go-github to v36 2021-06-29 15:46:23 -04:00
Sharif Elgamal 73e42a9a8c change parallelization in code 2021-06-29 11:46:38 -07:00
Andriy Dzikh 858aab3f55 Create sync_tests.sh to be triggered by each integration test on completion. 2021-06-29 11:44:36 -07:00
Medya Ghazizadeh cb750c21ba
Merge pull request #11797 from spowelljr/updateTimeToK8sSettingss
Make minor changes to time-to-k8s benchmarking
2021-06-29 12:26:28 -04:00
Sharif Elgamal 065929ffd0 try less parallelization for macOS tests 2021-06-29 09:22:42 -07:00
Steven Powell 8b45846088 make minor changes to time-to-k8s benchmarking 2021-06-28 15:25:58 -07:00
Ilya Zuyev bed20c522b In integration test logs: show GCS https links to reports 2021-06-28 14:34:05 -07:00
Andriy Dzikh 4f114dafb9 Add comment to report_flakes.sh to see environment flake charts. 2021-06-28 09:19:43 -07:00
Medya Ghazizadeh 4d078ae82f
Merge pull request #11766 from andriyDev/EnvPage
Create page with all tests for an environment in flake rate charts
2021-06-25 19:27:00 -04:00
Sharif Elgamal 13d03a588b idk trying stuff 2021-06-25 15:53:35 -07:00
Sharif Elgamal bd2a9daa00 let's fix generate_docs for good 2021-06-25 14:12:38 -07:00
Andriy Dzikh e9932aa7db Improve error handling to actually render in the UI. 2021-06-25 11:13:46 -07:00
Andriy Dzikh f57d7ae723 Create loading bar while downloading flake chart data. 2021-06-25 11:05:49 -07:00
Andriy Dzikh 637e8326fd Implement proper body-by-lines iterator. 2021-06-25 10:47:06 -07:00
Ilya Zuyev cb31ce2e7d remove kvm2-arm64 rpm 2021-06-25 10:08:40 -07:00
Andriy Dzikh eeaf03e4a3 Add name of test to each point in environment flake page. 2021-06-25 09:43:47 -07:00
Andriy Dzikh b1ec38f808 Add links to charts for each individual test on environment page. 2021-06-25 09:39:13 -07:00
Andriy Dzikh c067ea6cda Create table to show all flake rates for a specific environment. 2021-06-25 09:39:13 -07:00
Andriy Dzikh e22ddb9238 Compute flakeiness based on last 15 days and only chart top 10 flakiest tests. 2021-06-25 09:39:13 -07:00
Andriy Dzikh 9d41adbf06 Create page for viewing all tests for an environment. 2021-06-25 09:39:11 -07:00
Andriy Dzikh fc6c4eb04e Move existing formatting/charting code into its own function. 2021-06-25 09:29:54 -07:00
Andriy Dzikh f404d5976f Refactor flake chart data processing to simplify into two steps: aggregation and formatting. 2021-06-25 09:26:57 -07:00
Ilya Zuyev 0033bf7ed3 make targets for deb and rpm 2021-06-24 20:02:21 -07:00
Andriy Dzikh a0c5634dfd Make nicer fonts for test tooltips in flake charts. 2021-06-24 11:57:07 -07:00
Andriy Dzikh 1c7d6b719c Add links to hashes in flake charts. 2021-06-24 11:51:00 -07:00
Andriy Dzikh 0ff73cf645 Group commit hashes into one entry rather than one entry per run. 2021-06-24 11:42:50 -07:00
Steven Powell e3326f88df finishing last of comments 2021-06-23 15:03:03 -07:00
Steven Powell 0a5aadf502 addressed more comments 2021-06-23 10:42:39 -07:00
Steven Powell 39208c56c4 addressed some of the comments 2021-06-23 09:38:59 -07:00
Andriy Dzikh fbf61e3e33 Fix running a non-existent command in upload_integration_report. 2021-06-22 14:55:49 -07:00
Sharif Elgamal 055e20c28d fix generate-docs bugs part 2 2021-06-22 12:58:38 -07:00
Sharif Elgamal 269514601b better title for PR 2021-06-22 12:27:14 -07:00
Sharif Elgamal 74635e495a more fixes :) 2021-06-22 10:47:44 -07:00
Sharif Elgamal 23a31d7365 fix generate_docs bugs 2021-06-21 15:35:59 -07:00
Sharif Elgamal 811ff18161
Merge pull request #11677 from sharifelgamal/auto-gen-docs
Automatically run `make generate-docs` after every PR merge
2021-06-18 14:30:08 -07:00
minikube-bot c8fb43220d remove exit on error 2021-06-18 13:56:02 -07:00
Sharif Elgamal 248942f830 move around check 2021-06-18 13:46:30 -07:00
Sharif Elgamal 1200975bc7 debug 2021-06-18 13:28:01 -07:00
Medya Ghazizadeh 5501afa9ef
Merge pull request #11683 from andriyDev/FixNoFlakesComment
Fix flake rates being commented about when no failed tests are present.
2021-06-17 17:58:44 -04:00
Sharif Elgamal 5816b2a0c7 golang 1.6.4 2021-06-17 13:51:37 -07:00
Sharif Elgamal f8b2ebcc63 do not run workflow if there is no secret 2021-06-17 13:25:13 -07:00
Sharif Elgamal d37a3e0ba2 make script executable 2021-06-17 12:41:37 -07:00
Sharif Elgamal c1a7937184 change workflow condition 2021-06-17 12:08:50 -07:00
Medya Ghazizadeh 7d9bcf8626
Merge pull request #11684 from andriyDev/ChangeFlakeColors
Update flake chart colors
2021-06-17 13:00:46 -04:00
Medya Ghazizadeh 3c01c8a152
Merge pull request #11675 from andriyDev/FixReportFlakes
Stop using sudo for check_install_gh
2021-06-17 13:00:04 -04:00
Andriy Dzikh 300230bd5c Update flake chart colors. 2021-06-17 09:41:40 -07:00
Andriy Dzikh 5b8b5ce102 Fix flake rates being commented about when no failed tests are present. 2021-06-17 09:28:27 -07:00
Andriy Dzikh 9f601ea393 Fix commenting to a PR instead of an issue. 2021-06-16 18:03:52 -07:00
Sharif Elgamal c91e981e5f Merge branch 'master' of github.com:kubernetes/minikube into auto-gen-docs 2021-06-16 16:30:01 -07:00
Andriy Dzikh c037e5f62f Stop using sudo for check_install_gh. 2021-06-16 15:29:50 -07:00
Steven Powell 8f019e0418
Merge pull request #11659 from spowelljr/updateTimeToK8s
Create automated time-to-k8s benchmarks on release
2021-06-16 20:04:11 +00:00
Sharif Elgamal 7b5381b697 add generate-docs script 2021-06-16 09:23:54 -07:00
Steven Powell 25c5bec652 change back iterations to 5 2021-06-15 10:30:08 -07:00
Steven Powell 599e7c49e9 create automated time-to-k8s benchmarks on release 2021-06-15 10:17:46 -07:00
Andriy Dzikh 231fbee7b5 Fix collect_data_manual not being formatted correctly. 2021-06-15 09:55:32 -07:00
Medya Ghazizadeh 6a9076391d
Merge pull request #11602 from andriyDev/Flakes
Create a system for generating public flake rate reporting.
2021-06-15 12:51:29 -04:00
Sharif Elgamal 76d74191d8
Merge pull request #11623 from sharifelgamal/1.21.0
Bump Makefile for 1.21.0 release
2021-06-11 08:46:19 -07:00
Andriy Dzikh 1c1fdbff42 Make compute_flake_rate also compute average test duration. 2021-06-10 14:00:47 -07:00
Andriy Dzikh 728229c719 Add more environments to the allowed environments for commenting. 2021-06-10 13:36:23 -07:00
Andriy Dzikh dc6cb0b671 Rename collect_data.sh to collect_data_manual.sh and make header comment more clear. 2021-06-10 13:30:35 -07:00
Andriy Dzikh 884216db9e Make report_flakes.sh abort if there are no failed tests. 2021-06-10 13:27:00 -07:00
Andriy Dzikh 7e785c1c1e Make collect_data.sh optimzie and upload to GCS automatically. 2021-06-10 13:19:05 -07:00
Andriy Dzikh a98db3511e Add description and example usage to all shell scripts. 2021-06-10 13:16:16 -07:00
Andriy Dzikh 2b3f7cedd7 Remove bootstrap from flake_chart.html. 2021-06-10 13:02:56 -07:00
Sharif Elgamal 79e1b4e99d Bump Makefile for 1.21.0 release 2021-06-10 11:34:06 -07:00
Andriy Dzikh ecaee4d932 Add better comments for optimize_data and process_data. 2021-06-10 11:16:44 -07:00
Andriy Dzikh 79f8de1bcb Add comment for testEntry. 2021-06-10 11:11:34 -07:00
Andriy Dzikh e9e7b85e02 Make types and functions private. 2021-06-10 11:08:44 -07:00
Steven Powell cc89907cca add -Force to folder rm 2021-06-10 10:40:49 -07:00
Andriy Dzikh e089973f65 Create SplitEntryMap type to simplify some definitions. 2021-06-09 15:21:43 -07:00
Steven Powell 3b27578610 delete `minikube-integration` folder in case last test failed to delete it 2021-06-09 15:11:03 -07:00
Andriy Dzikh fb8e4d982b Clean up compute_flake_rate.go. 2021-06-08 13:06:28 -07:00
Andriy Dzikh fcbae7eaa1 Make sure gh is present when running report_flakes.sh. 2021-06-08 11:54:08 -07:00
Andriy Dzikh 3aa922813c Fix wrong number of parameters for report_flakes.sh. 2021-06-08 11:50:14 -07:00
Andriy Dzikh 716f690189 Change copyright to 2021. 2021-06-07 16:39:06 -07:00
Andriy Dzikh 139d7e3771 Fix lints in compute_flake_rate.go and compute_flake_rate_test.go. 2021-06-07 16:36:01 -07:00
Andriy Dzikh 8f953781a2 Create report_flakes script to comment on PRs about flake rates of failed tests. 2021-06-07 14:44:49 -07:00
Andriy Dzikh 7c46154600 Rename jenkins_upload_tests.sh to upload_tests.sh.
Since these scripts are already in the jenkins folder, having the jenkins prefix is redundant.
2021-06-07 14:12:31 -07:00
Andriy Dzikh 501b238841 Use "set -eu -o pipefail" for all scripts.
Previously failing commands in scripts wouldn't make them actually fail. Now it does!
2021-06-07 14:09:32 -07:00
Andriy Dzikh 9e7f1ebbf0 Make computing flake rates print out percentages (with fixed 2 decimal precision) rather than floats. 2021-06-07 13:49:40 -07:00
Andriy Dzikh a80f3bc5ae Add license to upload_tests script. 2021-06-07 13:49:05 -07:00
Steven Powell b39f171268 ensure Windows integration tests are ephemeral 2021-06-04 14:08:37 -07:00
Andriy Dzikh cec82877d8 Format flake rates into CSV containing environment, test, and flake rate. 2021-06-03 14:25:04 -07:00
Andriy Dzikh d245cfcdf7 Move all flake rate files to Jenkins to allow auto-upload. 2021-06-03 14:12:03 -07:00
Andriy Dzikh 40fdbe61ae Allow Jenkins to append to the flake rate data. 2021-06-03 14:01:06 -07:00
Andriy Dzikh d7d3593a89 Rewrite process_data.sh to no longer depend on git log.
This now means we depend on the date being contained within the details.
2021-06-03 13:38:44 -07:00
Medya Gh 74fe0e9f62 bump gopogh 2021-06-02 16:30:50 -07:00
Andriy Dzikh 65be305aab Add test for ComputeFlakeRates. 2021-06-02 16:17:39 -07:00
Andriy Dzikh df6f7a8485 Add test for FilterRecentEntries. 2021-06-02 15:48:12 -07:00
Andriy Dzikh 4e9718a28b Add test for SplitData. 2021-06-02 14:15:25 -07:00
Andriy Dzikh 401bcbfe0a Move comparison code to its own function. 2021-06-02 14:12:43 -07:00
Andriy Dzikh 9d4153f0ab Create test for ReadData. 2021-06-02 14:00:01 -07:00
Andriy Dzikh 60929009d6 Create FilterRecentEntries to only include the last N run dates. 2021-06-02 11:01:33 -07:00
Andriy Dzikh e6a553f679 Compute flake rate of all entries split by environment and test. 2021-06-02 11:00:40 -07:00
Andriy Dzikh de6cff23db Split entries based on environment and test name. 2021-06-02 11:00:40 -07:00
Andriy Dzikh ef33b8661c Create new compute_flake_rate.go with basic CSV parsing. 2021-06-02 11:00:38 -07:00
Andriy Dzikh fbf4e03eb9 Add license to sh scripts. 2021-06-01 14:06:54 -07:00
Andriy Dzikh 78e9838283 Refactor collect_data into collect_data + process_data.
This allows processing data coming from other sources.
2021-06-01 14:02:06 -07:00
Andriy Dzikh 419f2506e6 Add "data optimization" which treats empty fields in data.csv as equivalent to the previous entry.
This optimization takes data size down from 41 MB to 16MB which is ~40% which is huge!
2021-06-01 12:15:16 -07:00
Andriy Dzikh 424c954770 Add duration to existing graph with appropriate labels. 2021-06-01 11:33:22 -07:00
Andriy Dzikh b55c6726ef Use URL search query to select test and environment. 2021-06-01 10:18:26 -07:00
Andriy Dzikh ae62edbd18 Update JS to accept duration data. 2021-06-01 10:17:29 -07:00
Andriy Dzikh b5151a6d89 Add duration to data collection. 2021-06-01 09:46:14 -07:00
Andriy Dzikh f9f3ef9a09 Pass driverName into Preload and PreloadExists to report no preload for BareMetal. 2021-05-27 15:02:00 -07:00
Steven Powell ee5faa6fc4
Merge branch 'master' into automateTimeToK8s 2021-05-26 16:45:13 -07:00
Steven Powell 913d096d03 add note to release notes to remind to run `make time-to-k8s-benchmark` 2021-05-26 16:44:21 -07:00
Steven Powell 766cfd1e76 add copyright boilerplate 2021-05-26 16:26:45 -07:00
Steven Powell c30dbb0ead clean up var init 2021-05-26 15:59:50 -07:00
Steven Powell 2530048689 removed unused consts 2021-05-26 15:50:36 -07:00
Steven Powell a81761ea4a add make command to auto benchmark time-to-k8s and update website 2021-05-26 15:45:11 -07:00
Andriy Dzikh 2e5ea59774 Refactor data cleaning. 2021-05-26 09:36:44 -07:00
Andriy Dzikh 328d54ef63 Create first flake rate chart. 2021-05-25 16:20:56 -07:00
Andriy Dzikh b45b4c9a0b Include Google Charts. Refactor to wait for Google Charts and test data loading at the same time. 2021-05-25 13:42:53 -07:00
Andriy Dzikh b33e272435 Create basic parsing of CSV test data. 2021-05-25 13:42:12 -07:00