782cd749bb
enhance: make new go package to manage proto ( #39129 )
...
issue: #39095
pr: #39114
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-10 10:13:01 +08:00
ebfd917bb6
fix: make asan avaiable when building milvus image ( #37804 )
...
issue: #35854
pr: #37041
- USE_ASAN will not enable the Debug mode.
- replace USE_ASAN by `ldd` to make generate right so in milvus image.
Signed-off-by: chyezh <chyezh@outlook.com>
Co-authored-by: yellow-shine <sammy.huang@zilliz.com>
2024-11-19 17:28:32 +08:00
af5e32d00b
enhance: refine the pipeline ( #37456 )
...
https://github.com/milvus-io/milvus/pull/37412
---------
Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-11-06 10:24:30 +08:00
1916aec67a
enhance: [skip e2e] add make run-test-cpp gtest ( #35829 ) ( #35837 )
...
pr: https://github.com/milvus-io/milvus/pull/35829
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-08-29 21:15:02 +08:00
00202b0d7e
enhance: add skip 3rdparty env var ( #35748 )
...
issue: #35611
pr: #35736
- call `conan install` is too slow for developer, add option to skip it.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-29 14:23:01 +08:00
a4533f1b8a
enhance: optimize milvus core building ( #35660 )
...
issue: #35549,#35611,#35633
pr: #35610
- remove milvus_segcore milvus_indexbuilder..., add libmilvus_core
- core building only link once
- move opendal compilation into cmake
- fix odr
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-27 18:55:00 +08:00
7062e6fcbf
enhance: enable asan for milvus ( #35687 )
...
issue: #35626
pr: #35627
Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-26 09:54:58 +08:00
e2542a1bf5
enhance: Update protobuf-go to protobuf-go v2 ( #34394 ) ( #35555 )
...
issue: #34252
pr: #34394 #35072 #35084
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-21 18:50:58 +08:00
ce53e79f12
fix: enable milvus.yaml check ( #34567 ) ( #35446 )
...
See #32168
pr: #34567 #35152
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-08-13 19:00:23 +08:00
ddb52e71cb
enhance: pinpoint revision of 3thirdparty package ( #35316 )
...
https://github.com/milvus-io/milvus/pull/35325
---------
Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-08-08 09:46:22 +08:00
3a997e8056
enhance: docker-compose first then try to use docker compose ( #35227 )
...
https://github.com/milvus-io/milvus/issues/35209
https://github.com/milvus-io/milvus/pull/35208
---------
Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-08-02 19:37:09 +08:00
f90322a8ca
enhance: upgrade conan version ( #35216 )
...
https://github.com/milvus-io/milvus/pull/35215
Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-08-02 19:26:15 +08:00
a1a0a56f86
enhance: async search and retrieve in cgo ( #34200 )
...
issue: #33132
pr: #33133
other pr: #33228 , #34084 , #33946
- implement future-based cgo utility
- async search and retrieve in cgo
- modify gc configuration document
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-04 13:02:09 +08:00
6b348e4e91
enhance: [2.4] Add go-deadlock as unittest only dependency ( #33063 ) ( #34322 )
...
Cherry-pick from master
pr: #33063
See also #33062
This PR:
- Add lock.RWMutex & lock.Mutex alias to switch implementation based on
build flags
- When build flags has test in it, use go-deadlock to detect possible
deadlocks
- Replace all sync.RWMutex & sync.Mutex in datacoord pkg
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-02 18:48:10 +08:00
99586066f5
feat: [cherry-pick] Major compaction ( #34326 )
...
This PR cherry-picks the following commits:
fix: speed up segment lookup via channel name in datacoord (#33530 )
needed by the next commit
feat: Major compaction (#33620 )
issue: #30633
pr: #33620
---------
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: yiwangdr <80064917+yiwangdr@users.noreply.github.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
2024-07-02 18:29:01 +08:00
c924b0b502
enhance: [cherry-pick] Refine index code and support analyze data ( #34311 )
...
This PR primary picks up the support analyzing functionality, including
the following commits:
- main functionality: https://github.com/milvus-io/milvus/pull/33651
- refine indexnode code: https://github.com/milvus-io/milvus/pull/33458
- related fixes:
- https://github.com/milvus-io/milvus/pull/33832
- https://github.com/milvus-io/milvus/pull/33161
issue: #30633
master prs: #33651 , #33458 , #33832 , #33161
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
Co-authored-by: chasingegg <chao.gao@zilliz.com>
Co-authored-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-07-02 09:50:39 +08:00
7b4c1a0a96
enhance: [skip e2e] use disk ann when os is ubuntu, rocky or amazon ( #34244 )
...
pr: https://github.com/milvus-io/milvus/pull/34236
Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
2024-06-27 20:50:13 +08:00
22e6807e9a
feat: support inverted index for array ( #33452 ) ( #34053 )
...
pr: https://github.com/milvus-io/milvus/pull/33184
pr: https://github.com/milvus-io/milvus/pull/33452
pr: https://github.com/milvus-io/milvus/pull/33633
issue: https://github.com/milvus-io/milvus/issues/27704
Co-authored-by: xiaocai2333 <cai.zhang@zilliz.com>
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: cai.zhang <cai.zhang@zilliz.com>
2024-06-24 10:50:03 +08:00
b3d425f50a
enhance: Upgrade go version from 1.20 to 1.21 ( #33940 )
...
issue #32982
related pr in master:
pr: #33047 #33150 #33176 #33351 #33202 #33192
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-06-20 14:52:00 +08:00
2638735d05
fix: [Cherry-pick][2.4] Download and install cmake for the current platform, not x86_64 only ( #33439 )
...
issue https://github.com/milvus-io/milvus/issues/32476
pr: https://github.com/milvus-io/milvus/pull/32548
tested on x86_64 and aarch64. I'm not sure what needs to be done on some
exotic architectures.
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
2024-05-29 16:16:00 +08:00
244d2c04f6
feat: Add `milvusclient` package and migrate GoSDK ( #32907 )
...
Related to #31293
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-10 18:01:30 +08:00
54a51b1236
enhance: Support dynamic config for opentelemetry trace ( #32169 )
...
relate: https://github.com/milvus-io/milvus/issues/31940
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-05-09 17:43:30 +08:00
b6c06ff2c3
fix: Removed tests on non existent directories and solved name resolution errors ( #32266 )
...
issue: #32254
Signed-off-by: Ald392 <imagesai32@gmail.com>
2024-04-30 11:33:26 +08:00
f5e1fc757c
fix: [skip e2e] dev container yaml indentation error ( #32601 )
...
issue: #32594
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-04-26 10:15:27 +08:00
72b054e0c3
doc: change milvus docker image version to v2.4.0 ( #32489 )
...
change milvus docker image version to v2.4.0
Signed-off-by: ping.liu <ping.liu@zilliz.com>
2024-04-23 09:47:23 +08:00
744a54a534
enhance: enforce milvus.yaml assertion in UT ( #32357 )
...
See #32168
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-04-19 16:47:20 +08:00
1cd15d9322
test: support segment release in integration test ( #31190 )
...
issue: #29507
Notice that api_testonly.go files should be guarded by compiler tag
`test`, so that production build rules don't compile them and these APIs
don't get misused.
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2024-04-09 11:39:17 +08:00
f399416b92
enhance: Use gotestsum to run go unit test ( #31622 )
...
See also #31490
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-27 15:29:10 +08:00
4da8b6607d
enhance: Add scripts to use `gotestsum` to execute integration test ( #31490 )
...
See also #31489
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-22 10:29:06 +08:00
d5ba221b25
doc: change milvus docker image version to v2.4.0-rc.1 ( #31461 )
...
Signed-off-by: ping.liu <ping.liu@zilliz.com>
2024-03-20 22:15:05 +08:00
26539a1b6d
doc: change milvus docker image version to 2.3.12 ( #31336 )
...
Signed-off-by: ping.liu <ping.liu@zilliz.com>
2024-03-18 14:19:05 +08:00
375190e76e
fix: cpp format check not work ( #30767 )
...
fix : #30765
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-22 19:40:53 +08:00
976b6fc0e4
enhance: change opendal as compile configurable ( #30384 )
...
#30373
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-02-20 19:16:52 +08:00
f4559cbe54
doc: change image version to 2.3.9 ( #30695 )
...
Signed-off-by: ping.liu <ping.liu@zilliz.com>
2024-02-20 18:52:52 +08:00
32cff25f97
enhance: decrease coordinator init time ( #29822 )
...
This PR mainly improve two items:
1. Target observer should refresh loading status during init time. An
uninitialized loading status blocks search/query. Currently, the target
observer refreshes every 10 seconds, i.e. we'd need to wait for 10s for
no reason. That's also the reason why we constantly see false log
"collection unloaded" upon mixcoord restarts.
2. Delete session when service is stopped. So that the new service
doesn't need to wait for the previous session to expire (~10s).
Item 1 is the major improvement of this PR, which should speed up init
time by 10s.
Item 2 is not a big concern in most cases as coordinators usually shut
down after stop(). In those cases, coordinator restart triggers serverID
change which further triggers an existing logic that deletes expired
session. This PR only fixes rare cases where serverID doesn't change.
integration test:
`go test -tags dynamic -v -coverprofile=profile.out -covermode=atomic
tests/integration/coordrecovery/coord_recovery_test.go -timeout=20m`
Performance after the change:
Average init time of coordinators: 10s
Hardware: M2 Pro
Test setup: 1000 collections with 1000 rows (dim=128) per collection.
issue: #29409
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2024-02-05 14:00:12 +08:00
5cdf0f2490
doc: fix standalone stop cannot start issue ( #30481 )
...
Signed-off-by: ping.liu <ping.liu@zilliz.com>
2024-02-02 21:05:05 +08:00
e4a033858e
doc: Add Milvus standalone all in one script ( #30459 )
...
Signed-off-by: ping.liu <ping.liu@zilliz.com>
2024-02-02 18:05:04 +08:00
ec688f5bf6
fix: failed to download OpenDAL ( #30380 )
...
OpenDAL's url has been changed
fix #30379
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-02-02 14:35:04 +08:00
265453f400
enhance: [skip e2e]revert back to original way to archive, upload and download ( #30248 )
...
Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2024-02-01 10:15:04 +08:00
9f8eb0e527
enhance: make integration test case timeout configurable ( #30073 )
...
currently integration test may timeout if any case run time is above 3
minutes. This duration was hard coded.
This PR change this duration into a customized parameter and could be
passed via test running commands.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-18 12:22:54 +08:00
031243fee7
feat: support mmap for marisa trie ( #29613 )
...
this supports mmap for marisa trie index
related https://github.com/milvus-io/milvus/issues/21866
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-11 10:22:50 +08:00
4b3de64733
enhance: add rust to install_dep.sh ( #29586 )
...
fix : #29585
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-01-07 20:04:12 +08:00
3561586edf
feat: Add import reader for binlog ( #28910 )
...
This PR defines the new import reader interfaces and implement a binlog
reader for import.
issue: https://github.com/milvus-io/milvus/issues/28521
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-05 11:48:47 +08:00
5f00bad4b8
fix: link with install path's libblob-chunk-manager ( #29496 )
...
issue: #29494
1. link with install path's libblob-chunk-manager
2. performance of `ShouldBindWith` is better than `ShouldBindBodyWith`
3. the middleware shouldn't read the unrefreshed parameter repeatly
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-31 20:02:48 +08:00
e41fd6fbde
enhance: Move proxy client manager to util package ( #28955 )
...
issue: #28898
This PR move the `ProxyClientManager` to util package, in case of
reusing it's implementation in querycoord
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-20 19:22:42 +08:00
56d7225673
fix: Add LD_LIBRARY_PATH when local start milvus ( #29287 )
...
#28165
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-12-19 10:08:41 +08:00
a602171d06
enhance: Refactor runtime and expr framework ( #28166 )
...
#28165
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-12-18 12:04:42 +08:00
ed79505d31
enhance: use specific rust version to compile opendal ( #29106 )
...
/kind improvement
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-12-13 16:34:38 +08:00
d2207cac27
fix: Fix rustup update ( #29093 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-11 11:36:37 +08:00
36f69ea031
feat: integrate storagev2 in building index of segcore ( #28768 )
...
issue: https://github.com/milvus-io/milvus/issues/28655
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-12-05 16:48:54 +08:00
Zhen Ye
yellow-shine
zhenshan.cao
wei liu
Ted Xu
congqixia
wayblink
cai.zhang
Jiquan Long
shaoting-huang
Alexander Guzhva
aoiasd
Aldrin
pingliu
yiwangdr
zhagnlu
yah01
yihao.dai
PowderLi
XuanYang-cn
Bingyi Sun