Commit Graph

9062 Commits (48898cefea89edd1865f234f08150d8c54f66079)

Author SHA1 Message Date
wayblink 48898cefea
fix: fix meta revert bug when clustering compaction task fail (#34604)
#34603

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-14 22:29:38 +08:00
zhagnlu bd9727a1f7
fix: fix bug that set incorrect info to columnbase (#34428)
#34427

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-14 22:27:46 +08:00
zhagnlu 18c83c6466
fix: fix auto merge error (#34661)
#33704

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-13 13:07:37 +08:00
cai.zhang 93a500676b
fix: Remove flushed segment in segment manager generated through import (#34649)
issue: #34648

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-12 23:15:22 +08:00
congqixia 1a248f2668
enhance: Add param item for segmentFlushInterval (#34629)
See also #28817

Add paramitem for segment flush interval

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-12 18:59:35 +08:00
SimFG d8e68cbb97
enhance: use the insert rate limit for the upsert request (#34615)
- issue: #34614

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-12 16:17:35 +08:00
congqixia 531092c031
enhance: Add lint rule to forbid gogo protobuf (#34594)
github.com/gogo/protobuf is deprecated and could be error prune after
upgrade protobuf message to v2.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-12 10:19:35 +08:00
Chun Han f00c529aea
feat: support group_size for search_group_by(#33544) (#33720)
related: #33544

mainly changes in three aspects:

1. enable setting group_size for group by function
2. separate normal reduce and group by reduce
3. eleminate uncessary padding in search result for reducing

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-07-12 10:17:36 +08:00
aoiasd 5bb0d21e32
fix: Restful API use deprecate error code cause access log panic. (#34576)
relate: https://github.com/milvus-io/milvus/issues/34578

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-07-12 10:13:35 +08:00
wayblink 358e9a10d2
enhance: Alter compactTo segments before compactFrom to avoid data loss if crash (#34513)
#34512

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-12 00:55:34 +08:00
jaime a08a0c831f
fix: encountering orphan channel-cp meta after DataCoord GC (#34612)
issue: #34545

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-11 23:01:35 +08:00
XuanYang-cn d7966f46ad
fix: Compaction task l0 state transfer wrong (#34597)
This bug caused failed L0 compaction tasks never end.

See also: #34460

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-11 21:39:36 +08:00
Patrick Weizhi Xu 104d0966b7
feat: support partition key isolation (#34336)
issue: #34332

---------

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-07-11 19:01:35 +08:00
XuanYang-cn d7a3697fb5
enhance: Add back compactionTaskNum metrics (#34583)
Fix L0 compaction task recover unable to set segment not isCompacting

See also: #34460

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-11 17:45:37 +08:00
wayblink fd3da90089
enhance: Refine clustering compaction latency metrics (#34528)
#34527

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-11 17:43:43 +08:00
smellthemoon 07b94b4615
enhance: support upsert autoid==true (#30342)
related with: #29258

---------

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-07-11 16:53:35 +08:00
XuanYang-cn e0b39d8bf4
fix: Milvus panic when compaction disabled and dropping a collection (#34103)
See also: #31059

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-11 14:44:52 +08:00
congqixia be92147723
enhance: Print reason when seal segments (#34575)
See also #34574

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-11 11:40:51 +08:00
jaime 3b62138c5c
fix: unstable UT for level0 deletion (#34524)
issue: #34533

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-11 10:02:56 +08:00
chyezh 1bc3c0b925
enhance: implement balancer at streaming coord (#34435)
issue: #33285

- add balancer implementation
- add channel count fair balance policy
- add channel assignment discover grpc service

Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-11 09:58:48 +08:00
jaime c332f69dec
enhance: skip orphan channel cp meta when checking cp lag (#34555)
issue: # #34545

Print warn log instead of check health fail if orphan channel cp meta is
found in health check request.

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-11 09:36:56 +08:00
SimFG b58a5617ef
enhance: add the seal segment when dispatch delete msgs (#34565)
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-10 19:14:51 +08:00
wei liu ebc68d2774
fix: Indexnode stuck at stopping progress cause by wrong lifetime control (#34558)
issue: #34557

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-10 15:12:13 +08:00
congqixia 4850336ca3
fix: Write padding at end of mmap file not chunk (#34529)
Related to #34508

The padding bytes shall be written only at the end of the mmap file not
the chunk of each field data file.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-10 11:12:14 +08:00
wei liu 9b37d3f517
enhance: Enable setting the replica number and resource group during collection creation (#34403)
issue: #30040

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-10 10:20:13 +08:00
congqixia d60e628aed
enhance: Avoid use concrete segment type in segments interfaces (#34521)
See also #34519

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-10 10:18:12 +08:00
wei liu 3c7046353a
fix: RootCoord panic during check health (#34541)
issue: #34540

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-10 09:51:44 +08:00
sre-ci-robot 0b3f317b14
[automated] Update Knowhere Commit (#34549)
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-07-10 01:26:14 +08:00
XuanYang-cn 314f4d995b
enhance: Tidy dc channel manager (#34515)
See also: #34518

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-09 18:26:12 +08:00
jaime 4365308241
enhance: support setting properties in create database request (#34510)
issue: #34493

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-09 18:16:18 +08:00
SimFG 5016038781
enhance: release the record in delete codec and add some log for compaction (#34454)
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-09 15:40:17 +08:00
Ted Xu eae4dfca7b
fix: reuse deserialize result to help improve memory management (#34507)
Fixed #33268
The original reuse is broken by #33359

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-07-09 14:12:10 +08:00
wei liu eeb03a0d6a
fix: Query may return deleted records (#34501)
issue: #34500
cause the sort in `GetLevel0Deletions` will broken the corresponed order
between pks and tss, then the pks and tss will be sorted in
segment.Delete() interface.

This PR remove this uncessary and incorrect sort progress to avoid query
may return deleted records.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-09 10:46:11 +08:00
wayblink 0c4d82c6d3
fix: fix compaction task startTime is 0 (#34496)
fix bug: #34497 #34498

---------

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-09 10:22:13 +08:00
congqixia 3333160b8d
enhance: Fix lint issues from recent PRs (#34482)
See also #34483
Some lint issues are introduced due to lack of static check run. This PR
fixes these problems.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-09 10:06:24 +08:00
chyezh dfe0416a70
enhance: implement streaming node server service (#34166)
issue: #33285

- implement producing and consuming server of message
- implement management operation for streaming node server

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-09 07:58:14 +08:00
yihao.dai e15ac2b472
fix: Fix incorrect segment num rows (#34441)
Repeated calls to UpdateStatistics, this PR correct it.

issue: https://github.com/milvus-io/milvus/issues/34440

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-08 20:00:14 +08:00
Chun Han 8af187f673
fix: lose partitionIDs when scalar pruning and refine segment prune ratio metrics(#30376) (#34477)
related: #30376
fix: paritionIDs lost when no setting paritions
enhance: refine metrics for segment prune

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-07-08 19:54:15 +08:00
Gao ae6d6f91e6
enhance: change autoindex default metric type (#34261)
issue: #34304 
cosine is more widely used in float vectors, and cosine and hamming
distance are 'metrics' which have good geometric properties

Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-07-08 19:52:24 +08:00
XuanYang-cn 467113deab
fix: Accidently exit the check loop (#34481)
See also: #34460

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-08 16:46:13 +08:00
wayblink efdaed4ac6
fix: fix clustering compaction can't enqueue when compaction queue is full (#34445)
#30633

---------

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-08 10:02:13 +08:00
yihao.dai 4e5f1d5f75
enhance: Pre-allocate ids for import (#33958)
The import is dependent on syncTask, which in turn relies on the
allocator. This PR pre-allocate the necessary IDs for import syncTask.

issue: https://github.com/milvus-io/milvus/issues/33957

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-07 21:26:14 +08:00
shaoting-huang f4dd7c7efb
enhance: add delta log stream new format reader and writer (#34116)
issue: #34123

Benchmark case: The benchmark run the go benchmark function
`BenchmarkDeltalogFormat` which is put in the Files changed. It tests
the performance of serializing and deserializing from two different data
formats under a 10 million delete log dataset.

Metrics: The benchmarks measure the average time taken per operation
(ns/op), memory allocated per operation (MB/op), and the number of
memory allocations per operation (allocs/op).
| Test Name | Avg Time (ns/op) | Time Comparison | Memory Allocation
(MB/op) | Memory Comparison | Allocation Count (allocs/op) | Allocation
Comparison |

|---------------------------------|------------------|-----------------|---------------------------|-------------------|------------------------------|------------------------|
| one_string_format_reader | 2,781,990,000 | Baseline | 2,422 | Baseline
| 20,336,539 | Baseline |
| pk_ts_separate_format_reader | 480,682,639 | -82.72% | 1,765 | -27.14%
| 20,396,958 | +0.30% |
| one_string_format_writer | 5,483,436,041 | Baseline | 13,900 |
Baseline | 70,057,473 | Baseline |
| pk_and_ts_separate_format_writer| 798,591,584 | -85.43% | 2,178 |
-84.34% | 30,270,488 | -56.78% |

Both read and write operations show significant improvements in both
speed and memory allocation.

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-07-06 09:08:09 +08:00
yihao.dai 43fd8d19c2
enhance: Check segment existence when FlushSegments and add some key logs (#34438)
Check if the segment exists during FlushSegments and add some key logs
in write path.

issue: https://github.com/milvus-io/milvus/issues/34255

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-06 08:50:11 +08:00
charleskakumanu 05df70973c
enhance: retry etcd server start with 3 default retries (#34208)
Added retry method and unit test cases for retrying etcd server start.
New to open source and Go language. Please reject if this is not the
requirement/ specify the modifications needed in code.
issue : #17569

Signed-off-by: Charles Kakumanu <charles_kakumanu@apple.com>
Co-authored-by: Charles Kakumanu <charles_kakumanu@apple.com>
2024-07-05 20:24:11 +08:00
jaime 60be454db0
enhance: add disk quota and max collections into db properties (#34368)
issue: #34385

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-05 18:22:17 +08:00
congqixia 6b4d977a10
fix: Write padding into mmap file in case of SIGBUS (#34443)
See also #34442

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-05 17:44:09 +08:00
aoiasd 233b8486ec
fix: generate gin logs failed cause access logs panic (#34142)
relate: https://github.com/milvus-io/milvus/issues/34141

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-07-05 16:30:10 +08:00
zhagnlu 18f2458385
fix: fix bitmap supported type (#34350)
#34314

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-05 15:50:10 +08:00
jaime 21fc5f5d46
enhance: Remove datanode reporting TT based on MQ implementation (#34421)
issue: #34420

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-05 15:48:09 +08:00