Commit Graph

267 Commits (a0537156c02bbdc9b2043c8a30450e7a1edd1d80)

Author SHA1 Message Date
congqixia a68b32134a
fix: Verify sync task target segment and retry if not match (#30500)
See also #27675 #30469

For a sync task, the segment could be compacted during sync task. In
previous implementation, this sync task will hold only the old segment
id as KeyLock, in which case compaction on compacted to segment may run
in parallel with delta sync of this sync task.

This PR introduces sync target segment verification logic. It shall
check target segment lock it's holding beforing actually syncing logic.
If this check failed, sync task shall return`errTargetSegementNotMatch`
error and make manager re-fetch the current target segment id.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-05 11:33:43 +08:00
congqixia 5be909982d
enhance: add MockSerializer generation command into Makefile (#29713)
See also #27675

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-05 21:42:47 +08:00
XuanYang-cn 632d8b3743
enhance: Change DN channelmanger into interface (#29307)
See also: #28854

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-27 16:00:48 +08:00
congqixia 277849a915
enhance: separate serializer logic from sync task (#29413)
See also #27675

Since serialization segment buffer does not related to sync manager can
shall be done before submit into sync manager. So that the pk statistic
file could be more accurate and reduce complex logic inside sync
manager.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-26 10:40:47 +08:00
XuanYang-cn ae180d1628
enhance: Change ChannelManager to interface (#29300)
Rewrite cluster test
issue: #28854

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-25 19:24:46 +08:00
SimFG 67ab0e424b
fix: Clean the compaction plan info to avoid the object leak (#29365)
issue: #29296

Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-12-22 12:00:43 +08:00
wei liu e41fd6fbde
enhance: Move proxy client manager to util package (#28955)
issue:  #28898

This PR move the `ProxyClientManager` to util package, in case of
reusing it's implementation in querycoord

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-20 19:22:42 +08:00
zhagnlu a602171d06
enhance: Refactor runtime and expr framework (#28166)
#28165

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-12-18 12:04:42 +08:00
wei liu fe1eeae2aa
enhance: Use mockery to replace manual mock code (#29074)
issue: #29043
This PR remove mannul mock code for proxy and data coord

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-13 10:46:44 +08:00
aoiasd 3c32ba2407
enhance: pack datacoord Cluster and SessionManager with interface and mock them (#28869)
relate: https://github.com/milvus-io/milvus/issues/28861
https://github.com/milvus-io/milvus/issues/28854

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-11 17:52:37 +08:00
XuanYang-cn 5bac7f7897
fix: Fix L0 compaction in datacoord (#28814)
See also: #27606

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-05 18:44:37 +08:00
XuanYang-cn e62edb991a
enhance: Add FlowgraphManager interface (#28852)
- Change flowgraphManager to fgManagerImpl
- Change close to stop
- change execute to controlMemWaterLevel
- Change method name of fgManager for readability
- Add mockery for fgmanager

Issue: #28853

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-30 18:42:32 +08:00
XuanYang-cn aae7e62729
feat: Add levelzero compaction in DN (#28470)
See also: #27606

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-30 14:30:28 +08:00
cai.zhang f5f4f0872e
enhance: Support importing data with parquet file (#28608)
issue: #28272

Numpy does not support array type import. 
Array type data is imported through parquet.

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-11-29 20:52:27 +08:00
XuanYang-cn 9b371067d2
feat: Add Compaction views and triggers (#27906)
- Add Compaction l0 views
- Add Compaction scheduler
- Add Compaction triggerv2
- Add Compaction view manager

See also: #27606

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-23 17:30:25 +08:00
Bingyi Sun d7145e2c06
enhance: Update golangci_lint version (#28535)
Update golangci lint and fix some warnings

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-11-21 10:04:21 +08:00
Enwei Jiao 7445d3711c
feat: trigger compaction to handle index version (#28442)
issue: https://github.com/milvus-io/milvus/issues/28441

---------

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-11-21 09:26:22 +08:00
congqixia bed7467f20
enhance: Remove commented code and fix naming issue (#28450)
This PR removes all the commented code and files from PR #28320

For naming issue:
- Renaming `MinCheckpoint` to `EarliestPosition`, see #28320 comment
- Renaming `writebuffer.Mananger` to `BufferMananger`, see #27874
comment

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-16 00:22:20 +08:00
congqixia 0b905078e7
Use writebuffer, sync manager refactory in datanode (#28320)
See also #27675
This PR make previously merged refactory of datanode go online
- Use write node to replace insert/delete node
- Use write buffer manager to control all buffers
- Use sync manager to control sync tasks instead of flush manager

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-15 15:24:18 +08:00
wei liu 5b45a138b1
disable auto balance when old node exists (#28191)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-11-07 14:02:20 +08:00
congqixia bf2f62c1e7
Add `WriteBuffer` to provide abstraction for delta policy (#27874)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-04 12:10:17 +08:00
wei liu ecec5dfcfd
fix retry on offline node (#28079)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-11-03 10:14:16 +08:00
congqixia 233bf90c55
Add SyncManager to replace flush manager (#27873)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-10-31 02:30:16 +08:00
xige-16 bf46ffd6c4
Fix build with diskann failed (#27672)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-10-12 19:25:35 +08:00
Enwei Jiao 0f2f4a0a75
Remove useless parameters for Makefile (#27622)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-10-11 20:45:35 +08:00
congqixia cbb350c552
Add broker for datanode grpc operations (#27631)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-10-11 17:03:34 +08:00
foxspy 2b46bd1f08
fix diskann searchCache param compatibility (#27557)
Signed-off-by: xianliang <xianliang.li@zilliz.com>
2023-10-11 16:57:34 +08:00
PowderLi 8d3069b1db
update openssl to 3.1.2 (#27399)
deal with root path's normalization

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-10-08 19:17:31 +08:00
congqixia 8c59dba329
Refine queryHook mockery (#27394)
This PR move `QueryHook` interface to `optimizers` pkg
Update all mockery generated files to latest
Add makefile entry for `QueryHook`

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-28 10:01:26 +08:00
congqixia a3dd2756cf
Add predicates for TxnKV operations (#27365)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-27 10:21:26 +08:00
jaime 7119cb29ca
Fix kafka consumer connection leak (#27224)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-09-26 10:31:27 +08:00
congqixia 7bbeebc0d1
Fix lint-fix command may fail when .git has new ref (#27354)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-26 10:05:25 +08:00
jaime 7f7c71ea7d
Decoupling client and server API in types interface (#27186)
Co-authored-by:: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-09-26 09:57:25 +08:00
yah01 61ed1c7de3
Fix code check passed even violates the rule (#27341)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-25 18:05:27 +08:00
foxspy 370b6fde58
milvus support multi index engine (#27178)
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-09-22 09:59:26 +08:00
SimFG 26f06dd732
Format the code (#27275)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
jaime 3d7235b22a
[skip e2e] Add an build azure opt in Makefile (#27251)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-09-20 13:59:23 +08:00
yah01 a6c6d7301c
Enable ASAN for ut (#27016)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-12 14:01:17 +08:00
aoiasd e107d0794c
support complex delete expression (#25752)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-09-12 10:19:17 +08:00
Enwei Jiao c3f15c6b95
Refactor duplicate error class into one place (#26985)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-09-11 20:43:17 +08:00
XuanYang-cn 7f1ae35e72
Add timeout in dispatcher, AsConsumer and Seek (#26686)
See also: #25309

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-09-08 09:51:17 +08:00
congqixia 9ed5841730
Use specified mockery & golangci-lint version (#26845)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-05 17:39:48 +08:00
congqixia fe7f7ea237
Add cpp-build and setenv for querynode mockery regen command (#26843)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-05 14:13:49 +08:00
congqixia 1a8cf5c415
Organize all mockery generation commands in Makefile (#26826)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-04 21:19:48 +08:00
congqixia d343888f3c
Fix pkg codecov & static-check script (#26720)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-31 09:59:01 +08:00
yah01 74fb244bd7
Upgrade mockery (#26461)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-18 18:10:18 +08:00
wei liu b47a72bfcf
fix set dirty segment distribution to leader view (#26180)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-11 11:21:32 +08:00
yah01 300fef446b
Enable mmap for vector index (#25877)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-10 13:59:15 +08:00
cai.zhang 760a2d9aa7
Support AllocTimestamp api for Milvus (#25784)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-07-25 10:05:00 +08:00
zhagnlu 65cb52d06b
Support dynamic simd framework and using term expr as example (#25260)
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-07-13 16:22:30 +08:00