milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
20,969 Commits
141 Branches
117 Tags
549 MiB
Commit Graph

569 Commits (ef923958bbb797e0874196dee5d4f12e58b8d551)

Author SHA1 Message Date
congqixia 385dec3b69
fix: index attr caches wrong result variable (#30960) 
See also #30757 #30756

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-03-01 13:17:00 +08:00
aoiasd 3633923bb7
enhance: clean invalid pipline excluded segment info (#30429) 
relate: https://github.com/milvus-io/milvus/issues/30281

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-03-01 10:43:01 +08:00
MrPresent-Han 17a2fd048e
feat: support set up knowhere-build-pool-size on querynode(#29650) (#30922) 
related: #29650

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2024-02-29 18:15:00 +08:00
chyezh 0c7474d7e8
enhance: add graceful stop timeout to avoid node stop hang under extreme cases (#30317) 
1. add coordinator graceful stop timeout to 5s
2. change the order of datacoord component while stop
3. change querynode grace stop timeout to 900s, and we should
potentially change this to 600s when graceful stop is smooth

issue: #30310
also see pr: #30306

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-02-29 17:01:50 +08:00
congqixia 4082315bd0
enhance: Add `ParseCTraceContext` util function for tracing (#30883) 
See also #29803

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-02-28 18:59:00 +08:00
wei liu 382b7485a6
enhance: Correct misleading nodeID in GetComponentStates's log (#30731) 
This PR corrects the misleading nodeId in GetComponentStates's log

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-02-28 13:13:12 +08:00
Cai Yudong 8a219e0102
feat: Support knowhere trace using OpenTelemetry (#30750) 
Issue: #21508

Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2024-02-28 12:29:00 +08:00
smellthemoon a4f3e01a3a
fix: add the range search params check in proxy (#30423) 
if check in Segcore, will not do the it when not insert data.
so, check "radius" and "range_filter" in proxy.
related with #30365

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-02-28 11:24:58 +08:00
chyezh 77477d6340
fix: wrong context passing into NewClient, error handling lost in session_util (#30817) 
issue: #30799

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-02-28 10:40:09 +08:00
yah01 57397b1307
enhance: add new LRU cache impl (#30360) 
- remove  the unused LRU cache
- add new LRU cache impl which wraps github.com/karlseguin/ccache

related #30361

---------

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-02-27 20:58:40 +08:00
Xiaofan 0661d33879
enhance: change frequent log to debug (#30782) 
change the "pipeline fetch insert msg" log to debug

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
 2024-02-27 10:30:56 +08:00
congqixia 637dcffb6b
fix: Disk resource is not requested for index loaded with disk (#30757) 
See also #30756

This PR:
- Request disk resource when index type, version loaded with disk
- Add attribute cache for index utility
- Add `typeutil.Pair`

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-02-27 08:50:56 +08:00
congqixia 536e082f5a
enhance: Use defer to clean legacy growing segment (#30770) 
See also #30735 #30735

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-02-23 10:02:55 +08:00
congqixia 90e802e441
enhance: Add unit test for level-zero segment deltalog request resource (#30718) 
See also #30670 #30704

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-02-23 10:00:54 +08:00
congqixia f3b7a8892f
fix: Release loaded growing if WatchDmlChannel fail (#30735) 
See also #30734

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-02-21 22:12:53 +08:00
yiwangdr c6665c2a4c
test: support multiple data/querynodes in integration test (#30618) 
issue: https://github.com/milvus-io/milvus/issues/29507

Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2024-02-21 11:54:53 +08:00
congqixia 1346b57433
enhance: Add deltalog expansion rate in segment loader (#30704) 
See also #30191

It turned out that in auto id and batch delete scenario actual memory
size of deltalog maybe way larger than deltalog file size. This PR add a
configurable expansion rate for deltalog memory usage to prevent
out-of-memory panicking during loading deltalogs.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-02-21 11:26:52 +08:00
congqixia e5a16050ce
fix: Update disk usage metrics after segment released (#30702) 
See also #30701

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-02-20 22:14:53 +08:00
yah01 b74673c147
enhance: calculate the accuracy memory usage while loading segment (#30473) 
the old version Knowhere would copy the index data while loading, we
need to consider this to avoid OOM.

Knowhere provides a util function to indicate whether it will load the
index with disk, if not, we need to double the memory usage prediction
for index data

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-02-20 14:52:51 +08:00
congqixia ed754dc58c
enhance: Check Level-zero segment memory usage as well (#30670) 
Related to #30191

When loading segment, segment loader shall check memory usage for
current loading task. Previously l0 segment was ignored but level zero
segment may actually cost lots of memory.

This PR adds back memory resource check for Level zero segment loading.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-02-20 14:34:51 +08:00
zhagnlu e8a6f1ea2b
fix: erase pk empty check when pk index replace raw data (#30432) 
#30350

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-02-07 14:56:47 +08:00
congqixia 5143b8c793
enhance: Use dynamic pool for `NewLoadIndexInfo` (#30489) 
See also #30445

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-02-04 14:19:05 +08:00
xige-16 060c8603a3
fix: Support mvcc with hybrid serach (#30114) 
issue: https://github.com/milvus-io/milvus/issues/29656
/kind bug

Signed-off-by: xige-16 <xi.ge@zilliz.com>

---------

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2024-02-01 16:03:03 +08:00
yah01 878c4c9463
enhance: limit the max pool size to 16 (#30371) 
according to our benchmark, concurrency level 16 is enough to fully
utilize the object storage network bandwidth

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-31 14:13:06 +08:00
congqixia 7c086a4608
enhance: Set delete scope for LoadSegment streaming data (#30245) 
See also #29474

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-01-30 11:13:02 +08:00
PowderLi 8fc4ebfa11
fix: empty MetricType (#30216) 
issue: #30102 #30225

we should read MetricType from SearchResult,
because query node never
1. read metricType from LoadMeta
2. store to collection
3. set SearchRequest.MetricType

Signed-off-by: PowderLi <min.li@zilliz.com>
 2024-01-28 15:33:02 +08:00
xige-16 e9fdd2475d
fix: fix searchPlan metricType modified concurrently (#30227) 
issue: #30225
/kind bug
Signed-off-by: xige-16 <xi.ge@zilliz.com>

---------

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2024-01-26 14:03:09 +08:00
congqixia 405877c8cd
fix: Use correct pools for all CGO methods in segments pkg (#30274) 
See also #30273

This PR:
- Rename confusing `LoadIndexInfo` to `UpdateIndexInfo` for LocalSegment
- Use `DynamicPool` instead of `LoadPool` for `UpdateSealedSegmentIndex`
- Fix cgo call missing pool control

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-01-26 10:09:00 +08:00
yihao.dai c02fb64ad6
enhance: Allows proactive warming up of chunk cache (#30182) 
Allows proactive warming up of chunk cache. Original vector data will be
asynchronously loaded into the chunk cache during the load process. It
has the potential to significantly reduce query/search latency for a
certain duration after the load, albeit with a concurrent increase in
disk usage.

issue: https://github.com/milvus-io/milvus/issues/30181

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-01-25 19:55:39 +08:00
congqixia f2c0ead51a
enhance: Add trace span for scheduling read tasks in QueryNode (#30265) 
This PR adds a trace span for search/query task scheduling duration

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-01-25 16:59:00 +08:00
SimFG aa7014a360
enhance: move the cgo code in the pkg dir to interal dir (#30261) 
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-01-25 15:15:01 +08:00
SimFG ddccccbcab
enhance: add the bytes data type for merge data and format some code (#30105) 
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-01-18 22:18:55 +08:00
smellthemoon e52ce370b6
enhance:don't store logPath in meta to reduce memory (#28873) 
don't store logPath in meta to reduce memory, when service get
segmentinfo, generate logpath from logid.
#28885

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-01-18 22:06:31 +08:00
yah01 0d4e781f69
fix: the system rejects all queries and never recovers if enabled read rate limit (#30061) 
fix #30060

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-17 23:30:55 +08:00
yah01 9a3837212c
enhance: add index after load succeeded (#30015) 
this avoids a corner case: after load index failed, this index can be
never loaded as it has been added into the segment's index map

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-17 15:06:53 +08:00
yah01 8f083e45cb
enhance: enable converting segcore error to merr (#29914) 
this converts the segcore error to merr if possible

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-17 11:28:53 +08:00
chyezh d300bc7bcb
fix: querynode num entity metric is broken by illegal label (#29948) 
issue: #29766
also see pr: #29825

Signed-off-by: chyezh <ye.zhen@zilliz.com>
 2024-01-14 10:23:00 +08:00
Bingyi Sun e1258b8cad
feat: integrate storagev2 into loading segment (#29336) 
issue: #29335

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-01-12 18:10:51 +08:00
congqixia 10622698df
enhance: reduce delete detail log to delete range (#29916) 
Delete detail log will be large and hard to read when log level is
debug. This PR change the log to stringer and print only pk range,
number.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-01-12 14:40:59 +08:00
congqixia adf0c8885c
enhance: add trace span for wait tsafe (#29911) 
Add tracing span for search/query operation waiting tsafe duration

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-01-12 14:03:04 +08:00
jaime 90984a2cab
enhance: Support read hardware metrics for cgroupv2 (#29850) 
issue: https://github.com/milvus-io/milvus/issues/29846

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-01-12 10:30:51 +08:00
Xu Tong e429965f32
Add float16 approve for multi-type part (#28427) 
issue：https://github.com/milvus-io/milvus/issues/22837

Add bfloat16 vector, add the index part of float16 vector.

Signed-off-by: Writer-X <1256866856@qq.com>
 2024-01-11 15:48:51 +08:00
yah01 26e900180e
fix: the insert count is zero after set the pointer to nil (#29870) 
this leads to the EntitiesNum metric would be never reduced

fix: #29766

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-11 14:56:50 +08:00
yah01 44fe06f198
enhance: skip loading duplicated index (#29715) 
this protect the loading index from failure, and speed up the loading
progress

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-11 11:52:49 +08:00
congqixia d6429933a7
enhance: make Load process traceable in querynode & segcore (#29858) 
See also #29803

This PR:
- Add trace span for `LoadIndex` & `LoadFieldData` in segment loader
- Add `TraceCtx` parameter for `Index.Load` in segcore
- Add span for ReadFiles & Engine Load for Memory/Disk Vector index

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-01-10 21:58:51 +08:00
congqixia 93f87417fd
enhance: remove .git folder for unit test workflow (#29833) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-01-10 16:46:49 +08:00
yah01 d357139064
fix: the entities num metric may be contributed more than once (#29767) 
the growing segments contribute to this metric while inserting and
putting into the manager, but the current impl inserts data before
putting the segments into manager, which leads to double contributions

fix: #29766

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2024-01-10 10:00:51 +08:00
zhenshan.cao 60e88fb833
fix: Restore the MVCC functionality. (#29749) 
When the TimeTravel functionality was previously removed, it
inadvertently affected the MVCC functionality within the system. This PR
aims to reintroduce the internal MVCC functionality as follows:

1. Add MvccTimestamp to the requests of Search/Query and the results of
Search internally.
2. When the delegator receives a Query/Search request and there is no
MVCC timestamp set in the request, set the delegator's current tsafe as
the MVCC timestamp of the request. If the request already has an MVCC
timestamp, do not modify it.
3. When the Proxy handles Search and triggers the second phase ReQuery,
divide the ReQuery into different shards and pass the MVCC timestamp to
the corresponding Query requests.

issue: #29656

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-01-09 11:38:48 +08:00
xige-16 9702cef2b5
feat: Support multiple vector search (#29433) 
issue #25639 

Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2024-01-08 15:34:48 +08:00
congqixia fe47deebf3
fix: Set & Return correct SegmentLevel in querynode segment manager (#29740) 
See also #27349

The segment level label in querynode used `Legacy` before segment level
was correctly passed in Load request. Now this attribute is still using
legacy so the metrics does not look right.

This PR add paramter for `NewSegment` and passes corrent values for each
invocation.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-01-08 14:16:48 +08:00
Jiquan Long 20fb847521
enhance: load delta logs concurrently (#29623) 
This pr will make milvus load delta logs concurrently, which should
decrease the latency of loading a segment.
/kind improvement

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-01-07 20:22:48 +08:00
yah01 a0cec4047a
fix: make the entity num metric accurate (#29643) 
fix #29642

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-05 18:24:47 +08:00
smellthemoon 1c1f2a1371
enhance:change some logs (#29579) 
related #29588

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-01-05 16:12:48 +08:00
MrPresent-Han 9e2e7157e9
feat: support search_group_by for milvus(#25324) (#28983) 
related: #25324

Search GroupBy function, used to aggregate result entities based on a
specific scalar column.
several points to mention:

1. Temporarliy, the whole groupby is implemented separated from
iterative expr framework **for the first period**
2. In the long term, the groupBy operation will be incorporated into the
iterative expr framework:https://github.com/milvus-io/milvus/pull/28166
3. This pr includes some unrelated mocked interface regarding alterIndex
due to some unworth-to-mention reasons. All these un-associated content
will be removed before the final pr is merged. This version of pr is
only for review
4. All other related details were commented in the files comparison

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2024-01-05 15:50:47 +08:00
wei liu b45d08b47b
enhance: Add ctx for load index logs (#29686) 
This PR add ctx for load index logs

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-01-05 14:24:49 +08:00
yah01 9e0163e12f
enhance: use GPU pool for gpu tasks (#29678) 
- this much improve the performance for GPU index

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-04 17:50:46 +08:00
congqixia da7c3cbd88
enhance: make delegator delete buffer holding all delete from cp (#29626) 
See also #29625

This PR:
- Add a new implemention of `DeleteBuffer`: listDeleteBuffer
  - holds cacheBlock slice
  - `Put` method append new delete data into last block
  - when a block is full, append a new block into the list
- Add `TryDiscard` method for `DeleteBuffer` interface
  - For doubleCacheBuffer, do nothing
- For listDeleteBuffer, try to evict "old" blocks, which are blocks
before the first block whose start ts is behind provided ts
- Add checkpoint field for `UpdateVersion` sync action, which shall be
used to discard old cache delete block

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-01-04 17:02:46 +08:00
congqixia a8b7629315
fix: exclude insertData before growing checkpoint (#29558) 
Resolves: #29556
Refine exclude segment function signature
Add exclude growing before checkpoint logic

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-28 18:18:54 +08:00
MrPresent-Han ed644983e2
enhance: add param for bloomfilter(#29388) (#29490) 
related: #29388

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-12-28 18:10:46 +08:00
congqixia 6597c72992
fix: compose exclude info from flushed segment id (#29548) 
See also #29526

Previous PR removed flushed segment info from request, which causes
pipeline failing to exclude flushed segment info

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-28 14:02:54 +08:00
Gao 8a630f733a
enhance: add new optimize param for queryhook (#29495) 
add a flag to indicate if we use search param optimizations, default is
on

---------

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-12-28 10:04:46 +08:00
congqixia b251c3a682
enhance: add ctx for HandleCStatus and callers (#29517) 
See also #29516

Make `HandleCStatus` print trace id for better logging

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-27 16:10:47 +08:00
aoiasd a76e3b2813
Refine delete by expression for forbid proxy dml task scheduler hang (#29340) 
relate: https://github.com/milvus-io/milvus/issues/29146

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-12-26 19:52:48 +08:00
congqixia 02bc0d0dd5
fix: Add scope limit for querynode DeleteRequest (#29474) 
See also #27515

When Delegator processes delete data, it forwards delete data with only
segment id specified. When two segments has same segment id but one is
growing and the other is sealed, the delete will be applied to both
segments which causes delete data out of order when concurrent load
segment occurs.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-26 14:28:47 +08:00
yah01 b8318fcd7d
enhance: improve the handling for segcore error (#29471) 
- fix lost exception details in segcore
- improve the logs of handling errors from segcore

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-12-26 14:06:46 +08:00
cqy123456 4c979538a4
enhance: update cagra index params in config and add params check (#29045) 
issue:https://github.com/milvus-io/milvus/issues/29230
this pr do two things about cagra index:
 a.milvus yaml config support gpu memory settings

 b.add cagra-params check

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
Co-authored-by: yusheng.ma <yusheng.ma@zilliz.com>
 2023-12-26 11:04:47 +08:00
congqixia 13aa174b8a
enhance: add log when release segment created for load failure (#29464) 
Add log for releasing segment created during load process when load
error happens

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-26 00:46:46 +08:00
MrPresent-Han bd3bde82f0
fix iterator lose data for duplicted result(#29406) (#29451) 
related: #29406

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-12-25 23:28:47 +08:00
congqixia ac95c52171
enhance: change protection to RLock for loadStreamDelete (#29450) 
See also: #29332

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-25 23:27:02 +08:00
SimFG dd9c61831d
enhance: Support to get the param value in the runtime (#29297) 
/kind improvement
issue: #29299

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-12-22 18:36:44 +08:00
yah01 1997fd150c
enhance: use local path as mmap file path if no mmap path provided (#28992) 
the mmap mode isn't controlled by the config anymore, so we don't
require user to set it when they enabled mmap mode.

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-12-21 18:12:45 +08:00
yah01 a0e1a1eb31
feat: support enable/disable mmap for index (#29005) 
support enable/disable mmap for index, the user could alter the index's
mode by `AlterIndex` method
related: https://github.com/milvus-io/milvus/issues/21866

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-12-21 18:07:24 +08:00
wei liu cb0676153f
enhance: add metrics for stopping querynode balance progress (#29201) 
This PR add three metrics to track the stopping balance progress.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-12-21 10:14:52 +08:00
congqixia 1eacdc591b
fix: delegator may mark segment offline by mistake (#29343) 
See also #29332

The segment may be released before or during the request when delegator
tries to forward delete request to yet. Currently, these two situation
returns different error code.

In this particular case, `ErrSegmentNotLoaded` and `ErrSegmentNotFound`
shall both be ignored preventing return search service unavailable by
mistake.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-20 21:22:43 +08:00
yah01 61fc822207
fix: creating growing segments may introduce many threads (#29306) 
many growing segments may be created in a short time and there is no
restriction to the process, the CGO call will leave many threads

related: #29282

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-12-20 10:06:43 +08:00
zhagnlu a602171d06
enhance: Refactor runtime and expr framework (#28166) 
#28165

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-12-18 12:04:42 +08:00
congqixia 88b4b8b77c
enhance: make segments SQPool & LoadPool resizable (#29239) 
See also #29223

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-15 20:22:39 +08:00
yah01 c4dda3c50f
fix: panic caused by type assert LocalSegment on Segment (#29018) 
related #29017

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-12-08 01:50:38 +08:00
yah01 9b3e06ae86
enhance: add more metrics for level zero segments (#29029) 
- Add SegmentNum metric for level zero segments
- Add level zero segments size metirc

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-12-07 14:34:35 +08:00
congqixia dcb662d9ed
enhance: Refine C.NewSegment response and handle exception (#28952) 
See also #28795

Orignal `C.NewSegment` may panic if some condition is not met, this pr
changes response struct to `CNewSegmentResult`, which contains
`C.CStatus` and may return catched exception

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-07 13:34:35 +08:00
cai.zhang fb089cda8b
enhance: Load raw data while scalar index doesn't have raw data (#28888) 
issue: #28886

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-12-06 20:36:36 +08:00
aoiasd b4af6f8c40
fix: sync action load segment with lack collection index info list (#28788) 
relate: https://github.com/milvus-io/milvus/issues/28779
https://github.com/milvus-io/milvus/issues/28637

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-12-04 18:14:34 +08:00
yah01 bf633bb5d7
enhance: refine the retry error (#28573) 
return the last error but not combining all errors, to improve
readability and erorr handling

resolve: #28572

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-30 18:34:32 +08:00
yihao.dai b4353ca4ce
enhance: Remove vector chunk manager (#28569) 
We have implemented the chunkcache (in cpp) to retrieve vectors, hence
rendering the vectorchunkcache (in golang) obsolete.

issue: https://github.com/milvus-io/milvus/issues/28568

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-11-30 18:00:33 +08:00
yihao.dai 4b8bc2798e
enhance: Print nq (#28507) 
Log nq in search path.

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-11-30 10:42:27 +08:00
congqixia 1dc086496f
fix: schema->size() check logic with system field (#28802) 
Now segcore load system field info as well, the growing segment
assertion shall not pass with "+ 2" value
This will cause all growing segments load failure
Fix #28801
Related to #28478
See also #28524

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-29 22:40:28 +08:00
cqy123456 3b1b14dd78
fix: update binlog index memory uasge before loading segments (#28528) 
issue: #27678 
when interimIndex = true, memory predict should be update with the
memory usage of binlog index build process.

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2023-11-29 16:42:27 +08:00
wei liu 911a915798
feat: enable balance based on growing segment row count (#28623) 
issue: #28622 

query node with delegator will has more rows than other query node due
to delgator loads all growing rows.
This PR enable the balance segment which based on the num of growing
rows in leader view.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-27 14:58:26 +08:00
congqixia d344336a13
fix: Change schema to atomic.Pointer to avoid data race (#28739) 
Fix #28738

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-25 12:42:24 +08:00
congqixia 098c1c926d
fix: Add IndexList check for load segment request (#28601) 
See also #28022 #28034
The load segment may reaches before watch dml channel, so the index meta
may be empty as well

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-23 19:42:30 +08:00
Gao 3e77365de5
fix: correct autoindex segment num (#28387) 
Fix #28386 
Current code snippet 
```
// get delegator
sd, ok := node.delegators.Get(channel)
if !ok {
err := merr.WrapErrChannelNotFound(channel)
log.Warn("Query failed, failed to get shard delegator for search", zap.Error(err))
return nil, err
}
req, err = node.optimizeSearchParams(ctx, req, sd)
if err != nil {
log.Warn("failed to optimize search params", zap.Error(err))
return nil, err
}
// do search
results, err := sd.Search(searchCtx, req)
```

We could move these into `ShardDelegator`, and directly use sealed
segment num in `Search` methods, also segment num got outside could be
wrong when we specify partitions.

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-11-22 11:12:22 +08:00
smellthemoon 73f2bab454
enhance:add some log when create client and get component states (#28160) 
/kind improvement

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-11-22 09:12:22 +08:00
congqixia faf38ed8dd
fix: collection not `Unref` in local worker release (#28590) 
See also #28589

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-21 20:20:24 +08:00
congqixia efe3fe7b2e
fix: collection released when localWorker load segments (#28598) 
See also #28596
Increase ref for collection during load and unref after load completed.
Use the same logic protection from services.go `LoadSegments`

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-21 20:18:22 +08:00
yah01 cc952e0486
enhance: optimize forwarding level0 deletions by respecting partition (#28456) 
- Cache the level 0 deletions after loading level0 segments
- Divide the level 0 deletions by partition
related: #27349

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-21 18:24:22 +08:00
Bingyi Sun 0f985c9961
enhance: remove concurrency level in `checkSegmentSize` (#28533) 
`concurrency` is no longer needed in `checkSegmentSize`

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-11-21 15:00:23 +08:00
aoiasd 13a5b9f64a
fix: query l0 segment bugs (#28558) 
relate: https://github.com/milvus-io/milvus/issues/27675

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-11-20 17:26:23 +08:00
yah01 d2f53aefa5
enhance: improve load speed (#28518) 
This check rejects load request if running out the pool workers, but
small segment would be loaded soon, another segments would been loading
again after a check interval, which leads to slow loading for collection

Block the request by go pool

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-17 19:56:21 +08:00
MrPresent-Han 836f300536
support skip-index based on chunk-metrics to accelerate expr filter(#27925) (#28297) 
related: #27925

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-11-15 11:20:19 +08:00
yah01 d20ea061d6
Fix panic while forwarding empty deletions to growing segment (#28213) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-08 16:42:21 +08:00
SimFG e3b7fdac61
Delay the cancellation of ctx when stopping the node (#28247) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-11-08 03:20:17 +08:00
yah01 ece592a42f
Deliver L0 segments delete records (#27722) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-07 01:44:18 +08:00
yah01 90e2c63d9e
Fix getting incorrect CPU num (#28146) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-11-06 06:02:16 +08:00
wei liu 7485eeb689
fix sync distribution with wrong version (#28130) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-03 19:02:18 +08:00
wei liu ecec5dfcfd
fix retry on offline node (#28079) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-03 10:14:16 +08:00
yah01 dc89730a50
Support collection-level mmap control (#26901) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-02 23:52:16 +08:00
wei liu 9b737b77a7
fix fromShardLeader typo (#28061) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-02 15:50:16 +08:00
Enwei Jiao f8dd589755
Refactor collection's cgo call (#28055) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-11-02 13:02:13 +08:00
congqixia e4fdf5e68e
Refine offline segments logic in shard delegator (#28073) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-01 20:04:24 +08:00
cqy123456 4fbe3c9142
replace loaded binlog with binlog index for search performance (#27673) 
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2023-11-01 02:20:15 +08:00
yah01 1b9b8a31a8
Fix QueryNode panic while upgrading (#28034) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-31 15:50:13 +08:00
yah01 ab6dbf7659
Limit max thread num for pool (#28018) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-31 13:00:13 +08:00
yah01 9b6eeb46f1
Cache segment row num, size, and insert count to reduce CGO calls (#28007) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-30 17:54:14 +08:00
MrPresent-Han f93ad6471c
enable query reduce on querynodes for shortcut return(#27306) (#27837) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-10-30 15:08:11 +08:00
congqixia 36ddeae2cc
Refine log level when request resource fail for loading segments (#28004) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-30 14:20:14 +08:00
Filip Haltmayer 6b1a106a31
Moving etcd client into session (#27069) 
Signed-off-by: Filip Haltmayer <filip.haltmayer@zilliz.com>
 2023-10-27 07:36:12 +08:00
congqixia 13877a07ff
Add ctx parameter for tsafe pkg & NewDelegator method (#27877) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-26 19:14:10 +08:00
wei liu 8041fc3c75
avoid add empty growing segment to delegator distribution (#27930) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-10-26 10:10:10 +08:00
Gao c8415453a1
Support prepare config for index (#27920) 
Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-10-25 19:54:12 +08:00
aoiasd 9ba79a97e4
Add some log for delete task when it need query (#27317) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-10-24 14:54:12 +08:00
wei liu eadd48ef7c
fix grpc error judge logic (#27800) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-10-20 04:28:09 +08:00
MrPresent-Han af61a80f73
connect traceID for retrieve (#27804) 
/kind improvement

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-10-19 19:22:10 +08:00
zhenshan.cao 020ad9a6bc
Rectify wrong exception messages associated with Array datatype (#27769) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-10-19 17:24:07 +08:00
congqixia bcbe98aba1
Add querynode client wrapper and avoid grpc in standalone mode (#27781) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-19 11:10:07 +08:00
Enwei Jiao e98e56f75d
Fix SIGSEGV if run within gdb (#27736) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-10-18 02:16:11 +08:00
congqixia 1f2a76d04d
Handle legacy querynode `Delete` Unimplemented (#27749) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-17 19:06:06 +08:00
jaime ec1fe3549e
Add a stop hook to clean session (#27564) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-10-16 10:24:10 +08:00
yah01 50672f0d60
Skip empty segment while query/search (#27711) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-15 16:10:04 +08:00
yihao.dai d9431266bb
Fix panic at loadSegment during rolling upgrade (#27671) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-10-13 16:31:35 +08:00
yah01 be980fbc38
Refine state check (#27541) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-11 21:01:35 +08:00
wei liu 0695c88844
refine_log_of_update_target_version (#27572) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-10-11 06:51:33 +08:00
yihao.dai 7d0dd0047d
Use path.Join when init ChunkCache path (#27433) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-30 17:05:27 +08:00
zhenshan.cao dbdb9e15d8
Update Knowhere version (#27445) 
Signed-off-by: Li Liu <li.liu@zilliz.com>
Co-authored-by: Li Liu <li.liu@zilliz.com>
 2023-09-29 14:23:28 +08:00
Jiquan Long 370fdaf50d
Record engine version for segment index (#27384) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-28 18:03:28 +08:00
congqixia 6ff4ffa6c2
Fix scheduler may panic during close (#27431) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-28 16:53:26 +08:00
yihao.dai 106c17f304
Make read ahead policy in ChunkCache configurable (#27291) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-28 15:47:27 +08:00
congqixia 258e1ccd66
Refine querynode scheduler lifetime (#26915) 
This PR refines scheduler lifetime control:
- Move private tri-state into lifetime package
- Make scheduler block incoming "Add" task
- Make scheduler Stop wait until all previously accepted task done

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-28 10:21:26 +08:00
congqixia 8c59dba329
Refine queryHook mockery (#27394) 
This PR move `QueryHook` interface to `optimizers` pkg
Update all mockery generated files to latest
Add makefile entry for `QueryHook`

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-28 10:01:26 +08:00
yah01 2d6a968233
Refine QueryNode errors (#27380) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-26 22:09:28 +08:00
yah01 6539a5ae2c
Refine DataCoord status (#27262) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-26 17:15:27 +08:00
cai.zhang 30a6c6072e
Fix bug for loading array data panic (#27343) 
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-09-26 12:23:26 +08:00
jaime 7f7c71ea7d
Decoupling client and server API in types interface (#27186) 
Co-authored-by:: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-09-26 09:57:25 +08:00
foxspy 5db4a0489e
dynamic index version control (#27335) 
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-25 21:39:27 +08:00
xige-16 fa06265756
Fix cgo pointer param error (#27313) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-09-22 18:49:26 +08:00
yihao.dai 8f4aaa2da8
Remove ValidateIndexedFieldsData (#27254) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-22 18:31:25 +08:00
foxspy 370b6fde58
milvus support multi index engine (#27178) 
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-22 09:59:26 +08:00
SimFG 26f06dd732
Format the code (#27275) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-21 09:45:27 +08:00
yah01 0a750408d0
Skip delta logs have been applied (#26971) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-19 16:21:23 +08:00
MrPresent-Han 33e3e78937
add metrics for query reduce (#27201) 
/kind improvement
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-09-19 14:53:22 +08:00
congqixia cc9974979f
Add staticcheck linter and fix existing problems (#27174) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-19 10:05:22 +08:00
MrPresent-Han 7939f0e7d5
enable ctx traceId for assignsegment on dc(#26972) (#27108) (#27030) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-09-18 11:39:20 +08:00
Xiaofan e51cb739eb
Remove some logs in querynode delegator (#27156) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-09-18 10:57:20 +08:00
xige-16 488b423e1b
Fix high cpu usage caused by proto.size (#27054) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-09-17 20:55:21 +08:00
yihao.dai bb6711f28c
Add ChunkCache: support get vector from storage (#26142) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-15 10:21:20 +08:00
yah01 168e82ee10
Fix panic while handling with the nil status (#27040) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-15 10:09:21 +08:00
congqixia 394c898b4c
Discard SyncDistribution set action from legacy querycoord (#27027) 
Since Milvus in lower version (< 2.3.0), there is no load info in set action
which may corrupt data integrity and cause panicking

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-13 09:35:18 +08:00
yah01 00c65fa0d7
Refine QueryNode errors (#27013) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-12 16:07:18 +08:00
aoiasd e107d0794c
support complex delete expression (#25752) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-09-12 10:19:17 +08:00
congqixia ac45af585b
Make segment loaded successful put in manager even ctx done (#26992) 
Leave segment loaded in manager even wait other segment failed
See also #26908
Fix error case in distributed scenario

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-11 21:33:18 +08:00
congqixia 2a5d574a0d
Fix querynodev2 concurrent load logic (#26959) 
Fix logic error from #26926
function `waitSegmentLoadDone` shall return error when context is done

Make delegator control concurrency for each same segment
Related to #26908

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-10 07:41:18 +08:00
congqixia c6116d1819
Remove segment to LocalSegment type assertion (#26931) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-09 10:35:16 +08:00
congqixia c8f9f22c4a
Fix segment loader return false success (#26926) 
`waitSegmentLoadDone` did not check waitCh result is success or failure
after load return without error, delegator will assume all segments are loaded

This PR changes waitCh to loadResult with `sync.Cond` with `atomic.Int32` to represent status

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-08 16:41:16 +08:00
Xu Tong 9166011c4a
Add float16 vector (#25852) 
Signed-off-by: Writer-X <1256866856@qq.com>
 2023-09-08 10:03:16 +08:00
XuanYang-cn 7f1ae35e72
Add timeout in dispatcher, AsConsumer and Seek (#26686) 
See also: #25309

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-08 09:51:17 +08:00
congqixia af5c01082b
Refine delegator lifetime control (#26881) 
- Add SafeChan interface in lifetime package
- Embed SafeChan into  interface
- Replace private lifetime struct in delegator package with
- Refine delegator on-going task lifetime control and wait all accepted task done
- Fix potential goroutine leakage from  if delegator closed concurrently

/kind improvement

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-07 10:11:15 +08:00
SimFG 28681276e2
Improve the retry of the rpc client (#26795) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-06 17:43:14 +08:00
MrPresent-Han 528948559f
fix false load failure for long unserviable period(#26813) (#26818) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-09-06 12:57:15 +08:00
Enwei Jiao fb0705df1b
Decouple basetable and componentparam (#26725) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-05 10:31:48 +08:00
congqixia 4b58c71908
Add ctx parameter for organizeTask and GetWorker method (#26835) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-05 10:05:48 +08:00
congqixia 1a8cf5c415
Organize all mockery generation commands in Makefile (#26826) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-04 21:19:48 +08:00
yah01 3349db4aa7
Refine errors to remove changes breaking design (#26521) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-04 09:57:09 +08:00
congqixia e8f1b1736e
Remove log.Error(err.error())-style log (#26783) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-01 13:09:01 +08:00
XuanYang-cn ef75784715
Fix LoadSegmentLatency metric p99 (#26761) 
See also: #26743

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-01 11:07:07 +08:00
MrPresent-Han 8330c18dc9
add log for loading segment(#26564) (#26640) 
/kind improvement

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-08-31 12:03:00 +08:00
Gao 7e36f819b4
Make autoindex config clean (#26732) 
Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-08-31 10:25:01 +08:00
yah01 bfcc691129
Fix segment leaked if task canceled (#26685) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-30 14:17:03 +08:00
congqixia 95dcf7fa06
Fix memory leak when Put duplicated segments (#26693) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-30 11:37:00 +08:00
congqixia 89fc9aad82
Improve sync target version logic (#26630) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-29 23:12:27 +08:00
congqixia 1cf6e00fa6
Improve segment manager interface (#26637) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-29 15:46:27 +08:00
wei liu 7af0f7d90c
avoid concurrent sub/unsub on same channel (#26454) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-08-23 10:10:22 +08:00
Xiaofan 7b3a1dc93f
Fix load index panic (#26457) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-08-21 09:56:20 +08:00
wei liu 3b11ad2695
fix dead lock in concurrent search/release segment (#26435) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-08-18 15:28:18 +08:00
cqy123456 fd37860e57
init knowhere build/search thread pool; (#26449) 
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2023-08-18 14:00:19 +08:00
yihao.dai 63b86b32a6
Add server id validation interceptor (#26395) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-08-17 20:20:20 +08:00
congqixia 123ad921e0
Fix querynodev2 search/query segments return unexpected grpc err (#26341) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-17 18:24:18 +08:00
Xiaofan ba806429fa
Remove frequent log on search growing segment (#26315) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-08-17 15:44:53 +08:00
congqixia 855a04539a
Set delegator lastUpdateTs to startTs (#26416) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-17 12:24:18 +08:00
yah01 5dae6a654a
Protect segment from being released while query/search (#26322) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-16 18:38:17 +08:00
yah01 3d8871b30e
Fix data race while updating segment version (#26387) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-16 16:30:18 +08:00
MrPresent-Han d30a920226
add log trace for segcore(#26277) (#26339) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-08-16 11:41:33 +08:00
wei liu 23baecd70f
set sealed segment to unreadable before sync target version (#26338) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-08-15 17:27:35 +08:00
congqixia ec65a4e048
Use float64 for resource estimation log (#26335) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-15 14:25:33 +08:00
Enwei Jiao 7d61355ab0
Refactor log for Query (#26310) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-08-14 18:57:32 +08:00
yah01 a173486d2e
Fix calculation of memory usage prediction for mmap mode (#26264) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-12 17:19:31 +08:00
Gao 9ca8f28005
Fix missing totalChannelNum bug (#26295) 
Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-08-11 18:35:29 +08:00
yah01 48422dd4c5
Fix spawn too many threads (#26293) 
- Low the thread pool cap
- Limit CGO calls concurrency

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-11 18:29:29 +08:00
Gao b6fcbb0998
Support ScaNN index (#26099) 
Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-08-11 14:21:29 +08:00
wei liu b47a72bfcf
fix set dirty segment distribution to leader view (#26180) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-08-11 11:21:32 +08:00
xige-16 1971d98897
Add disk metric info (#25675) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-08-11 10:35:42 +08:00
Enwei Jiao ca1349708b
Remove time travel ralted testcase (#26119) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-08-10 18:53:17 +08:00
xige-16 1055c90456
Add default retrieve limit (#24782) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-08-10 14:11:15 +08:00
yah01 300fef446b
Enable mmap for vector index (#25877) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-10 13:59:15 +08:00
yah01 889424b3f9
Fix load index with empty file list (#26236) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-09 18:39:16 +08:00
congqixia eea9197306
Perform alter distribution under mutex protection (#26229) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-09 13:05:15 +08:00
wei liu f33a89387f
enable config shard level cost metrics whether contains worker's cost (#26132) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-08-09 11:37:15 +08:00
yah01 99c633954a
Downgrade the log level in search path (#26188) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-08 18:35:14 +08:00
Bingyi Sun 54c0e64059
Fix search on empty segments set bug (#26136) 
Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-08-08 11:17:08 +08:00
MrPresent-Han 68ecf49ed5
fix lost parameter for threadCoefficient(#25781) (#26109) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-08-03 12:05:06 +08:00
yah01 d9074bcb06
Remove unused code (#26100) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-08-03 09:45:05 +08:00
MrPresent-Han 5634ba777d
add new threadpool with various priority to avoid deadlock(#25781) (#26028) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-08-03 09:31:07 +08:00
smellthemoon 3fc18c53a3
Fix wrong log (#26022) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-08-02 11:01:07 +08:00
yah01 9c55a7f422
Add worker num as one of load resource (#26045) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-01 21:47:06 +08:00
Jiquan Long 5c1f79dc54
Push down the limit operator to segcore (#25959) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-08-01 20:29:05 +08:00
aoiasd 7fec0d61cc
Refine test chunkmanager of querynode v2 (#25888) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-08-01 10:33:04 +08:00
congqixia 8b5e276361
Use different cgo pool for SQ and other operations (#26021) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-01 09:19:05 +08:00
yah01 39b00b97a6
Add more logs for committed resources (#26026) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-31 19:27:03 +08:00
yah01 8245e078c0
Add max segment size log (#26015) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-31 14:37:04 +08:00
zhagnlu 833674c1cb
add glog configurable function and redirect aws log to segcore log (#25664) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-07-27 19:49:02 +08:00
congqixia 5fbe6e99fd
Remove debug log from segment loader (#25937) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-26 18:51:01 +08:00
wei liu 302897f866
refine look aside balance logic (#25837) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-25 18:51:01 +08:00
yah01 3735a097e6
Refine the memeory usage prediction (#25889) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-25 17:41:01 +08:00
congqixia 0bc03ede0d
Add eventlog pkg and support grpc streaming event observation (#25812) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-25 17:23:01 +08:00
wei liu a132122370
fix time unit in insert/delete cost metrics value (#25893) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-25 15:07:01 +08:00
aoiasd b533c68632
Forbid update checkpoint without msgID (#25694) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-07-25 10:43:04 +08:00
wei liu 0e26486abc
fix segment num in log (#25876) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-25 09:13:00 +08:00
yah01 168f346620
Remove deprecated interface (#25864) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-07-24 17:47:00 +08:00
wei liu fc19b85a40
fix count(*）retrieve redundant growing segment (#25825) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-24 14:09:00 +08:00
wei liu 32827f538a
add metrics for delegator insert/delete cost (#25733) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-21 15:30:59 +08:00
yah01 361db51337
Create dir if mmap enabled for validating path (#25750) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-19 18:48:58 +08:00
Cai Yudong 9a4761dcc7
Remove binary metrics TANIMOTO/SUPERSTRUCTURE/SUBSTRUCTURE (#25708) 
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2023-07-19 16:16:58 +08:00
yah01 079cd9dc70
Add caller skip to improve the log for segcore error (#25710) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-18 19:54:57 +08:00
congqixia cc71e065e3
Support load index after segment loaded (#25567) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-18 10:51:19 +08:00
congqixia c490b30db7
Fix segment loader skip wait if all segments are loading (#25643) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-17 19:51:18 +08:00
MrPresent-Han 49655d2f13
fix panic due to lack of indexInfo on querynode(#25362) (#25613) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-07-17 19:23:19 +08:00
yah01 948d1f1f4a
Handle errors by merr for QueryCoord (#24926) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-17 14:59:34 +08:00
Zhihong Yu a7023ab3cc
Narrow the scope of deleteMut lock in loadStreamDelete (#25436) 
Signed-off-by: Ted Yu <yuzhihong@gmail.com>
 2023-07-17 09:30:34 +08:00
wei liu b62c82af22
fix set target version in loading sealed segment (#25603) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-14 20:00:31 +08:00
congqixia f2b14b43de
Add log field for local worker call (#25594) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-14 17:34:31 +08:00
congqixia a7e6f08183
Fix DeleteCollection not protected (#25599) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-14 17:04:31 +08:00
Enwei Jiao 66fdc71479
Refactor logs in DataCoord & DataNode (#25574) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-07-14 15:56:31 +08:00
yah01 d216f9abda
Clear collection meta after all channels/segments released (#25486) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-14 10:28:30 +08:00
yiwangdr b9189b9f41
Organize mocks from types.go (#25466) 
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2023-07-14 10:12:31 +08:00
wei liu b72809db15
fix release sealed segment on unwatch channel (#25555) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-13 14:30:29 +08:00
yah01 205a7c430a
Fix QueryNode panic while inserting to released segment (#25495) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-12 19:48:29 +08:00
SimFG f9e2d00f91
Prevent `exclusive consumer` exception in pulsar (#25376) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-07-12 17:26:30 +08:00
yihao.dai 4c93495587
Add segment size metric in querynode (#25406) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-07-12 14:26:28 +08:00
congqixia bcd3abf644
Separate segments/shards reduce latency metrics (#25475) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-11 18:18:28 +08:00
congqixia 5aec6036dc
Add index info in GetDataDistribution response (#25444) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-11 11:22:29 +08:00
cai.zhang 81278f4aea
Fix bug of missing JSON type when sorting retrieve results (#25412) (#25455) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-07-10 20:02:28 +08:00
congqixia 9e3a591bda
Fix segment released early in case of balancing (#25453) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-10 18:58:28 +08:00