milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
18,360 Commits
146 Branches
120 Tags
575 MiB
Commit Graph

7713 Commits (5b45a138b12dcff145a8ff671598282b4c69015a)

Author SHA1 Message Date
smellthemoon a9982aa7c6
Remove topic empty restriction when preCreated topic (#27351) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-10-10 18:53:32 +08:00
congqixia b91a5ef42c
Refine log and err handling in querycoord broker (#27546) 
- Add log.Ctx(ctx) for all log occurences
- Use `merr.CheckRPCErr` for all grpc response error handling

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-10 11:49:32 +08:00
MrPresent-Han cb71a3e235
rm dependency to rc when getting recovery info(#25363) (#27405) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-10-09 18:51:32 +08:00
congqixia eca79d149c
Add ctx control for observer manual check methods (#27531) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-09 11:07:33 +08:00
yah01 3759857bc5
Refine Proxy errors (#27499) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-09 10:09:33 +08:00
XuanYang-cn 56c94cdfa7
Add channel manager in DataNode (#27308) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-10-08 21:37:33 +08:00
yah01 41495ed266
Improve the error message for getting all indexes of collection (#27389) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-08 21:23:32 +08:00
Jiquan Long e4f73cc805
Add host & enable_disk to session (#27507) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-10-08 20:05:31 +08:00
PowderLi 8d3069b1db
update openssl to 3.1.2 (#27399) 
deal with root path's normalization

Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-10-08 19:17:31 +08:00
yah01 a715165306
Set timeout for leader observer syncing (#27504) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-08 16:55:31 +08:00
Xiaofan 41124f281a
Remove parser dependency (#27514) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-10-08 15:05:31 +08:00
smellthemoon 241f2d77fb
Fix miss to handle error from GetCollectionID (#26712) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-10-08 11:53:32 +08:00
SimFG a3188231b3
Remove the log for GetFlushableSegments in the datacoord (#27503) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-10-08 10:05:31 +08:00
Sheldon 5ba158a3f5
fix knowhere version-changing (#27508) 
Update KNOWHERE_VERSION for the first occurrence

Signed-off-by: Sheldon <chuanfeng.liu@zilliz.com>
 2023-10-08 08:35:32 +08:00
congqixia cfe475a2ea
Add `SegmentLevel` field for L0 delta (#27502) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-07 20:55:31 +08:00
congqixia 5d558623fe
Add revive sub-lints and fix existing problems (#27495) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-07 20:53:38 +08:00
congqixia 80eb5434d3
Fix var-naming caused by old PRs (#27501) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-07 14:43:32 +08:00
yah01 8394b3a1ec
Block creating new error from status reason (#27426) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-07 11:29:32 +08:00
congqixia cd5f03f80c
Add var-name sub linter in revive (#27424) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-07 10:09:31 +08:00
Jiquan Long 5c1abfa2cc
Print the server id when active-standby switch (#27119) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-10-07 10:01:31 +08:00
Jiquan Long 0f14d18201
Optimize the codec code of session (#27360) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-10-01 10:33:30 +08:00
yihao.dai 7d0dd0047d
Use path.Join when init ChunkCache path (#27433) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-30 17:05:27 +08:00
yah01 63ac43a3b8
Refine errors for import (#27379) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-30 10:31:28 +08:00
zhenshan.cao dbdb9e15d8
Update Knowhere version (#27445) 
Signed-off-by: Li Liu <li.liu@zilliz.com>
Co-authored-by: Li Liu <li.liu@zilliz.com>
 2023-09-29 14:23:28 +08:00
cai.zhang 2b64cfc772
Check if insert data matches schema (#27401) 
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-09-28 19:01:27 +08:00
Jiquan Long 370fdaf50d
Record engine version for segment index (#27384) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-28 18:03:28 +08:00
congqixia 6ff4ffa6c2
Fix scheduler may panic during close (#27431) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-28 16:53:26 +08:00
yihao.dai 106c17f304
Make read ahead policy in ChunkCache configurable (#27291) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-28 15:47:27 +08:00
cai.zhang 91244f0498
Return element type for describing collection (#27363) 
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-09-28 14:55:26 +08:00
congqixia 258e1ccd66
Refine querynode scheduler lifetime (#26915) 
This PR refines scheduler lifetime control:
- Move private tri-state into lifetime package
- Make scheduler block incoming "Add" task
- Make scheduler Stop wait until all previously accepted task done

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-28 10:21:26 +08:00
congqixia 8c59dba329
Refine queryHook mockery (#27394) 
This PR move `QueryHook` interface to `optimizers` pkg
Update all mockery generated files to latest
Add makefile entry for `QueryHook`

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-28 10:01:26 +08:00
SimFG c9653b1683
Add some log and improve TestSessionProcessActiveStandBy test case (#27403) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-28 09:35:27 +08:00
yah01 a8ce1b6686
Refine QueryCoord stopping (#27371) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-27 16:27:27 +08:00
Enwei Jiao b80a3e19d3
Add code for PanicInfo (#27364) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-27 12:01:28 +08:00
wei liu 4071132f6a
reload loading collection when qc recover (#27300) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-09-27 11:43:28 +08:00
XuanYang-cn 5c5f9aa05e
Enhance newDataSyncService (#27277) 
- Add flowgraph.Assemble assembles nodes in flowgraph.go
- remove fgCtx in newDataSyncService
- Add newServiceWithEtcdTickler func, reduce param numbers to 3
- Remove unnecessary params
  - config.maxQueueLength, config.maxParallelish

See also: #27207

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-27 11:07:25 +08:00
SimFG 9fb4c27a90
Fix rootcoord failure to start when reaching the limit of role num (#27361) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-27 11:03:26 +08:00
cai.zhang dedb90f85f
Fix error message for creating scalar index (#27382) 
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-09-27 10:33:27 +08:00
congqixia a3dd2756cf
Add predicates for TxnKV operations (#27365) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-27 10:21:26 +08:00
yah01 2d6a968233
Refine QueryNode errors (#27380) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-26 22:09:28 +08:00
congqixia fd979a725a
Fix query_coord.proto comment format caused lint issue (#27375) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-26 20:35:27 +08:00
sre-ci-robot e02228b5ad
Update knowhere commit (#27357) 
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2023-09-26 18:57:37 +08:00
yah01 6539a5ae2c
Refine DataCoord status (#27262) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-26 17:15:27 +08:00
cai.zhang 30a6c6072e
Fix bug for loading array data panic (#27343) 
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-09-26 12:23:26 +08:00
yihao.dai 37db176f6a
Fix empty function name in monitoring (#27347) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-26 10:43:25 +08:00
wayblink 7dd0be1b2c
Enable bulkinsert binlog data with partitionkey (#27241) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-09-26 10:25:25 +08:00
wei liu 27046e242b
add timeout for graceful stop (#27326) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-09-26 10:15:25 +08:00
jaime 7f7c71ea7d
Decoupling client and server API in types interface (#27186) 
Co-authored-by:: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-09-26 09:57:25 +08:00
yah01 24354b166c
Fix unit test failed when run single test (#27348) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-26 09:23:25 +08:00
foxspy 5db4a0489e
dynamic index version control (#27335) 
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-25 21:39:27 +08:00
XuanYang-cn 676024ff38
Add rpc to notify channel operation (#27172) 
- Add NotifyChannelOperation
- Add CheckChannelOperationProgress

See also: #25309

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-25 14:05:27 +08:00
aoiasd 0a0e4f0547
Remove old mockChannelsMgrs, use mockery MockChannelsMgr instead (#27318) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-09-25 10:31:26 +08:00
wei liu 9433a24f5d
fix component not exit when liveness check failed (#27236) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-09-22 19:13:25 +08:00
MrPresent-Han 4b12cb8847
fix unstable ut due to unstable sort of unique set (#27302) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-09-22 19:07:26 +08:00
xige-16 fa06265756
Fix cgo pointer param error (#27313) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-09-22 18:49:26 +08:00
congqixia 1d76565894
Add metrics for garbage collection (#27303) 
Also fix second metrics usage in compaction

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-22 18:47:25 +08:00
yihao.dai 8f4aaa2da8
Remove ValidateIndexedFieldsData (#27254) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-22 18:31:25 +08:00
foxspy fa033e586a
disable growing index for flat (#27309) 
Signed-off-by: xianliang <xianliang.li@zilliz.com>
 2023-09-22 14:19:24 +08:00
congqixia 670cb386e7
Add back `gocritic` linter and fix related issues (#27289) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-22 10:05:26 +08:00
foxspy 370b6fde58
milvus support multi index engine (#27178) 
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-22 09:59:26 +08:00
wei liu fc9a9a7926
fix panic in getFlushState (#27237) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-09-21 14:53:25 +08:00
yihao.dai 217ed38bb4
Improve FlushChannels log (#27265) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-21 10:03:31 +08:00
SimFG 26f06dd732
Format the code (#27275) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-21 09:45:27 +08:00
yah01 93e2eb78c9
Delete only if primary keys exist (#25292) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-09-20 19:03:25 +08:00
congqixia 16b35e07b3
Fix `TestSessionSuite/TestKeepAliveRetryActiveCancel` unit test logic (#27231) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-20 18:59:23 +08:00
yah01 773080df24
Fix miss to handle the error of allocating ts (#27264) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-20 18:43:23 +08:00
XuanYang-cn 916528f51a
Add initCtx in getDataSyncService (#27199) 
Passing initCtx to all IO funcs in newDataSyncService,
so when ctx.Canceled, newDataSyncService would return.

See also: #25309

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-20 16:03:23 +08:00
yah01 9baff1b81c
Rename error to match the nameing rules (#27214) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-20 10:59:24 +08:00
yah01 b4f86ea55e
Construct all success status with merr (#27226) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-20 10:57:23 +08:00
yah01 338848fbc3
Remove unused code and dup imports (#27228) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-20 10:55:23 +08:00
Xiaofan 6635398a6d
Fix Bin log concurrency by adding a pool (#27189) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-09-19 18:05:22 +08:00
congqixia 19e4deb792
Make garbage collector fetch meta after list from storage (#27203) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-19 17:05:34 +08:00
yah01 0a750408d0
Skip delta logs have been applied (#26971) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-19 16:21:23 +08:00
XuanYang-cn 09505ea78e
Move etcd watch related code into eventmanager (#27192) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-19 15:27:38 +08:00
MrPresent-Han 33e3e78937
add metrics for query reduce (#27201) 
/kind improvement
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-09-19 14:53:22 +08:00
cai.zhang a362bb1457
Support array datatype (#26369) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-09-19 14:23:23 +08:00
XuanYang-cn 10116b85ac
Release before remove when releasing flowgraphs (#27191) 
GetAndRemove removes the fg from manager immediately,
while the flowgraph is still releasing. This PR will remove
the fg from flowgraphManager AFTER flowgraphs released.

- Add Remove for ConcurrentMap
- Move collections() into flowgraph manager

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-19 11:53:22 +08:00
yah01 b9ab18d692
Replace error code with merr for RateLimiter (#27176) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-19 10:37:23 +08:00
sre-ci-robot fc694bd56d
Update knowhere commit (#27190) 
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2023-09-19 10:35:22 +08:00
congqixia cc9974979f
Add staticcheck linter and fix existing problems (#27174) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-19 10:05:22 +08:00
PowderLi 4feb3fa7c6
support azure (#26398) 
Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-09-19 10:01:23 +08:00
yihao.dai fd73213539
Increase FlushChannel timeout to 3min (#27180) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-18 19:49:22 +08:00
yihao.dai 4b2802033d
Fix datanode panic due to concurrent compaction and delete processing (#27167) 
Co-authored-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-09-18 19:35:29 +08:00
yah01 0459a662e4
use MADV_WILLNEED for scalar column data (#27170) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-18 18:05:22 +08:00
Enwei Jiao 8e17bc3013
Use merr to handle segcore errors (#27137) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-18 14:07:22 +08:00
yah01 a6b98740b7
Replace all status with only error string (#27125) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-18 14:05:28 +08:00
MrPresent-Han 7939f0e7d5
enable ctx traceId for assignsegment on dc(#26972) (#27108) (#27030) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-09-18 11:39:20 +08:00
Xiaofan e51cb739eb
Remove some logs in querynode delegator (#27156) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-09-18 10:57:20 +08:00
MrPresent-Han 4f6122dd4a
enable ctx traceId for assignsegment on dc(#26972) (#27108) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-09-18 09:53:22 +08:00
sre-ci-robot a11136b158
Update knowhere commit (#27159) 
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2023-09-18 09:41:22 +08:00
xige-16 488b423e1b
Fix high cpu usage caused by proto.size (#27054) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-09-17 20:55:21 +08:00
yihao.dai c162c6a4c8
Increase FlushChannel retry times (#27140) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-16 00:41:20 +08:00
congqixia 6ee5fedfef
Fix lock not applied in quota center (#27133) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-15 19:11:25 +08:00
yihao.dai 7a59de1fb1
Fix flush slowly (#27123) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-15 17:45:21 +08:00
yihao.dai 060d3563ba
Fix compile error at core/storage (#27121) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-15 14:41:21 +08:00
sre-ci-robot c85c255eb1
Update knowhere commit (#27109) 
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2023-09-15 13:59:19 +08:00
yihao.dai bb6711f28c
Add ChunkCache: support get vector from storage (#26142) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-15 10:21:20 +08:00
yah01 168e82ee10
Fix panic while handling with the nil status (#27040) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-15 10:09:21 +08:00
cai.zhang 22d7fa4e1c
Use GetXXX instead of directly using dots to get value (#27045) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-09-15 10:07:19 +08:00
foxspy 18cbe26876
fix growing hasRawData (#27105) 
Signed-off-by: xianliang <xianliang.li@zilliz.com>
 2023-09-14 21:11:18 +08:00
yihao.dai fe5781e2b8
Maintain compatibility of GetFlushState with old client (#27077) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-14 19:57:20 +08:00
Jiquan Long 6f2c1214ed
Fix cloudProvider not passed (#27065) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-14 19:21:20 +08:00
Jiquan Long 24a04effe9
Delete failed index tasks ASAP (#27093) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-14 19:19:20 +08:00
yah01 7bb20fa394
Fix the double nil return values from RPC call (#27101) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-14 19:01:19 +08:00
congqixia c32a9a1d4a
Fix grpcclient use needReset as needRetry for cancel&timeout code (#27091) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-14 17:47:26 +08:00
yihao.dai 8c65791a3d
Fix proxy panic (#27066) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-14 14:27:26 +08:00
Enwei Jiao 0afdfdb9af
Remove other Exceptions, keeps SegcoreError only (#27017) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-14 14:05:20 +08:00
congqixia edde3cf1c7
Add tracer for querycoord tasks (#27058) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-14 09:59:19 +08:00
Jiquan Long af5619e73e
Fix misusing chunk manager (#27051) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-13 18:11:20 +08:00
XuanYang-cn ea2b909c6d
Fix historm metrics buckets range for index (#26976) 
See also: #26743

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-13 17:21:18 +08:00
yah01 3d05ddf505
Reduce cpp test time (#27043) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-13 15:41:18 +08:00
congqixia f0d0651989
Do not reset connection immediately if grpc code is `Canceled` or `DeadlineExceeded` (#27014) 
We found lots of connection reset & canceled due to recent retry change
Current implementation resets connection no matter what the error code is
To sync behavior to previous retry, skip reset connection only if cancel error happens too much.

Also adds a config item for minResetInterval for grpc reset connection

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-13 15:01:18 +08:00
PowderLi c033580af4
show index info while GetSegmentInfo (#26981) 
according to QueryNode::GetSegmentInfo

Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-09-13 11:37:18 +08:00
congqixia 394c898b4c
Discard SyncDistribution set action from legacy querycoord (#27027) 
Since Milvus in lower version (< 2.3.0), there is no load info in set action
which may corrupt data integrity and cause panicking

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-13 09:35:18 +08:00
yihao.dai dd2cb1d44a
Flush by flush channels (#26859) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-12 21:07:19 +08:00
cqy123456 0ff4ddc76c
remove VectorMemNMIndex (#27000) 
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2023-09-12 17:13:18 +08:00
yah01 00c65fa0d7
Refine QueryNode errors (#27013) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-12 16:07:18 +08:00
yiwangdr f85af0732c
remove MultiRemoveWithPrefix (#26924) 
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2023-09-12 15:07:17 +08:00
yah01 45d9fb5929
Fix wrong error codes & names (#26904) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-12 14:41:18 +08:00
Jiquan Long 802a6a86c0
Fix printing too many warnings (#27015) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-12 13:39:17 +08:00
aoiasd e107d0794c
support complex delete expression (#25752) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-09-12 10:19:17 +08:00
sre-ci-robot dde3cd2f93
Update knowhere commit (#26998) 
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2023-09-12 09:49:18 +08:00
congqixia ac45af585b
Make segment loaded successful put in manager even ctx done (#26992) 
Leave segment loaded in manager even wait other segment failed
See also #26908
Fix error case in distributed scenario

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-11 21:33:18 +08:00
yah01 3203ce1654
Reduce copy while retrieving primary keys (#26616) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-09-11 21:31:18 +08:00
Enwei Jiao c3f15c6b95
Refactor duplicate error class into one place (#26985) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-11 20:43:17 +08:00
congqixia adfb5298c6
Refine `TestSessionProcessActiveStandBy` unit test logic (#26980) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-11 18:13:17 +08:00
congqixia c45c32fad4
Set task reason for collection released (#26962) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-10 15:15:17 +08:00
MrPresent-Han 2101f2d289
fix unstable checker id due to go map iteration(#26943) (#26944) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-09-10 10:11:16 +08:00
congqixia 2a5d574a0d
Fix querynodev2 concurrent load logic (#26959) 
Fix logic error from #26926
function `waitSegmentLoadDone` shall return error when context is done

Make delegator control concurrency for each same segment
Related to #26908

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-10 07:41:18 +08:00
Futao Wei ad1cc00a18
Fix typo in comment (#26951) 
Signed-off-by: unfode <forrest.futao.wei@gmail.com>
 2023-09-09 20:13:16 +08:00
congqixia c6116d1819
Remove segment to LocalSegment type assertion (#26931) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-09 10:35:16 +08:00
congqixia 758aad705d
Fix checker using default interval after manual check (#26953) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-09 08:29:16 +08:00
Jiquan Long 9dae896395
Fix indexnode log (#26927) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-08 16:57:16 +08:00
congqixia c8f9f22c4a
Fix segment loader return false success (#26926) 
`waitSegmentLoadDone` did not check waitCh result is success or failure
after load return without error, delegator will assume all segments are loaded

This PR changes waitCh to loadResult with `sync.Cond` with `atomic.Int32` to represent status

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-08 16:41:16 +08:00
Xu Tong 9166011c4a
Add float16 vector (#25852) 
Signed-off-by: Writer-X <1256866856@qq.com>
 2023-09-08 10:03:16 +08:00
XuanYang-cn 7f1ae35e72
Add timeout in dispatcher, AsConsumer and Seek (#26686) 
See also: #25309

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-08 09:51:17 +08:00
SimFG 0901b76732
Avoid the panic when the status of rpc response is nil (#26910) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-07 19:23:15 +08:00
congqixia af5c01082b
Refine delegator lifetime control (#26881) 
- Add SafeChan interface in lifetime package
- Embed SafeChan into  interface
- Replace private lifetime struct in delegator package with
- Refine delegator on-going task lifetime control and wait all accepted task done
- Fix potential goroutine leakage from  if delegator closed concurrently

/kind improvement

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-07 10:11:15 +08:00
sre-ci-robot 56a6559fe7
Update knowhere commit (#26888) 
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2023-09-07 09:19:16 +08:00
yiwangdr 337edc321b
tikv integration (#26246) 
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2023-09-07 07:25:14 +08:00
cai.zhang 8b5b137207
Should set index task retry not failed when task canceled (#26878) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-09-06 19:01:14 +08:00
yah01 4a2377e669
Fix retrieving may return more result than limit (#26670) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-06 17:47:15 +08:00
SimFG 28681276e2
Improve the retry of the rpc client (#26795) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-06 17:43:14 +08:00
Jiquan Long 61c7b0990d
Workaround fix ChannelManager holding mutex too long (#26870) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-06 14:29:40 +08:00
zhagnlu 9b3e744b24
Fix embedded milvus log bug (#26864) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-09-06 14:27:14 +08:00
MrPresent-Han 528948559f
fix false load failure for long unserviable period(#26813) (#26818) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-09-06 12:57:15 +08:00
wei liu 0e2085b77f
fix dc standby to active (#26810) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-09-06 10:41:49 +08:00
wayblink d7b6f3a9e8
Enhance FlushAll/GetFlushAllState API (#26802) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-09-06 10:35:48 +08:00
sre-ci-robot 78a2638fd4
Update knowhere commit (#26861) 
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2023-09-06 09:21:49 +08:00
bjzhjing 548c82eca5
Refactor storage.MergeInsertData() to optimize the merging process (#26839) 
Benchmark Milvus with https://github.com/qdrant/vector-db-benchmark and
specify the datasets as 'deep-image-96-angular'. Meanwhile, do perf
profiling during 'upload + index' stage of vector-db-benchmark and see
the following hot spots.

39.59%--github.com/milvus-io/milvus/internal/storage.MergeInsertData
        |
        |--21.43%--github.com/milvus-io/milvus/internal/storage.MergeFieldData
        |          |
        |          |--17.22%--runtime.memmove
        |                     |
        |                     |--1.53%--asm_exc_page_fault
        |                     ......
        |
        |--18.16%--runtime.memmove
                   |
                   |--1.66%--asm_exc_page_fault
                   ......

The hot code path is in storage.MergeInsertData() which updates
buffer.buffer by creating a new 'InsertData' instance and merging both
the old buffer.buffer and addedBuffer into it. When it calls golang
runtime.memmove to move buffer.buffer which is with big size (>1M), the
hot spots appear.

To avoid the above overhead, update storage.MergeInsertData() by
appending addedBuffer to buffer.buffer, instead of moving buffer.buffer
and addedBuffer to a new 'InsertData'. This change removes the hot spots
'runtime.memmove' from perf profiling output. Additionally, the 'upload
+ index' time, which is one performance metric of vector-db-benchmark,
is reduced around 60% with this change.

Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
 2023-09-05 21:41:48 +08:00
Enwei Jiao 73f4347ac6
Fix code checker error (#26844) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-05 15:51:48 +08:00
congqixia fe7f7ea237
Add cpp-build and setenv for querynode mockery regen command (#26843) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-05 14:13:49 +08:00
Enwei Jiao fb0705df1b
Decouple basetable and componentparam (#26725) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-05 10:31:48 +08:00
congqixia 4b58c71908
Add ctx parameter for organizeTask and GetWorker method (#26835) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-05 10:05:48 +08:00
sre-ci-robot c132c53b1a
Update knowhere commit (#26840) 
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2023-09-05 09:05:49 +08:00
congqixia 1a8cf5c415
Organize all mockery generation commands in Makefile (#26826) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-04 21:19:48 +08:00
wei liu 1097776477
stop heartbeat if reach heartbeat limit (#26728) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-09-04 17:51:48 +08:00
cai.zhang 622077f9ad
Refine error message for json length exceed max length (#26676) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-09-04 17:47:48 +08:00
XuanYang-cn b2e7cbdf4b
Remove TimeTravel in compactor (#26785) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-04 17:41:48 +08:00
cai.zhang d7cd1f2a6d
Fix bug for get index state with compaction segment (#26822) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-09-04 17:29:48 +08:00
yihao.dai 4340cbfba2
Merge syncCPLagTooBehind policy into syncPeriodically policy (#26713) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-04 14:37:47 +08:00
chyezh 0530fd80c9
[Fixup] remove nats from default (#26791) 
Signed-off-by: chyezh <ye.zhen@zilliz.com>
 2023-09-04 10:01:04 +08:00
yihao.dai 7624c2b949
Improve rated log in insertBufferNode (#26788) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-04 09:59:10 +08:00
yah01 3349db4aa7
Refine errors to remove changes breaking design (#26521) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-04 09:57:09 +08:00
yihao.dai c6024a32f5
Add rate limit ratio monitoring (#26709) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-03 20:47:02 +08:00
MrPresent-Han a34a9d606c
fix panic due to empty traceID(#26754) (#26808) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-09-02 16:13:01 +08:00
yah01 941a383019
Fix failed to load collection with more than 128 partitions (#26763) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-02 00:09:01 +08:00
yihao.dai 64cf5eab18
Accelerate flush in flushAll (#26769) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-01 15:23:01 +08:00
congqixia e8f1b1736e
Remove log.Error(err.error())-style log (#26783) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-01 13:09:01 +08:00
wei liu 5602b22531
refine checker code style (#26759) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-09-01 11:57:01 +08:00
wei liu 949c320185
remove pull target from qc recover (#26775) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-09-01 11:17:01 +08:00
smellthemoon e56b0018e4
Remove merr mask logic to make it ok to compare resp.status with origin error (#26773) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-09-01 11:09:00 +08:00
XuanYang-cn ef75784715
Fix LoadSegmentLatency metric p99 (#26761) 
See also: #26743

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-01 11:07:07 +08:00
yah01 09218bfd3d
Optimize loading by reduce 1x copy while reading data (#26746) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-01 10:15:01 +08:00
sre-ci-robot b47da91f3c
Update knowhere commit (#26792) 
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2023-09-01 09:23:01 +08:00
Cai Yudong 8dc16b599b
Add binary metric types SUBSTRUCTURE/SUPERSTRUCTURE back (#26766) 
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2023-08-31 20:07:00 +08:00
cai.zhang 82b0e39abe
Fix bug for concurrent parsing expr with strings (#26721) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-08-31 17:39:02 +08:00
XuanYang-cn 8d54509e54
Fix CompactionLatency metrics (#26747) 
- Refine compactor logs

See also: #26743

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-08-31 17:35:03 +08:00
yah01 9004601817
Enable vector index mmap (#26750) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-31 17:31:11 +08:00
jaime d001133bd2
[skip e2e] Refine collection not exists error (#26707) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-08-31 17:29:03 +08:00
MrPresent-Han 8330c18dc9
add log for loading segment(#26564) (#26640) 
/kind improvement

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-08-31 12:03:00 +08:00
Xu Tong 5f08e3a72b
Fix field_name spell error (#26718) 
Signed-off-by: Writer-X <1256866856@qq.com>
 2023-08-31 11:39:02 +08:00
Jiquan Long 36e29ef859
Auto generate parser code (#26716) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-08-31 11:13:01 +08:00
Gao 7e36f819b4
Make autoindex config clean (#26732) 
Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-08-31 10:25:01 +08:00
Gao 41d37409e9
Remove unused search conf (#26689) 
Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-08-31 10:23:07 +08:00
smellthemoon 6069a7d42b
Log out memory in MB (#26262) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-08-30 17:13:00 +08:00
wei liu 2aab9b3234
fix proxy to querynode heartbeat failed counter logic (#26563) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-08-30 16:31:01 +08:00
wei liu 5af7559904
fix revoke ic session when stop dc (#26680) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-08-30 14:57:01 +08:00
SimFG 9311dc91ee
Clear error message in the delete request (#26656) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-08-30 14:47:00 +08:00
groot 1b1bafaff1
Bulkinsert read varchar by batch (#26199) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-08-30 14:23:06 +08:00
yah01 bfcc691129
Fix segment leaked if task canceled (#26685) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-30 14:17:03 +08:00
MrPresent-Han 7d5a4b2994
add more event for segcore search(#26277) (#26688) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-08-30 14:15:01 +08:00
congqixia 95dcf7fa06
Fix memory leak when Put duplicated segments (#26693) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-30 11:37:00 +08:00
XuanYang-cn 9598a8b236
Fix timeout task never release queue (#26593) 
See also: #26413, #26566

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-08-30 11:12:27 +08:00
smellthemoon 87ecaac703
Add dynamic schema check in upsert (#26644) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-08-30 10:52:26 +08:00
yah01 213db490bd
Use pointer receiver for large struct (#26668) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-30 10:24:29 +08:00
yah01 b475f25042
Remove invalid offset check while filling data (#26666) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-30 09:52:27 +08:00
wei liu cd7f811d40
fix rootcoord unstable ut (#26460) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-08-30 09:42:27 +08:00
congqixia 89fc9aad82
Improve sync target version logic (#26630) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-29 23:12:27 +08:00
congqixia 1cf6e00fa6
Improve segment manager interface (#26637) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-29 15:46:27 +08:00