Commit Graph

21 Commits (22bddde5ffd396529482d4646be1931220f0ca60)

Author SHA1 Message Date
jaime ba625835bc
enhance: Add metrics for segment index files size (#32979)
issue:#32980

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-05-13 17:59:32 +08:00
Patrick Weizhi Xu 52ae47c850
enhance: gather materialized view search info once per request (#31996)
issue: #29892 

This PR:
1. Move the process of gathering materialized search info to when the
search plan is created, before it goes to each segment, to avoid
repeated work and access the plan node under multi-threaded
circumstances.
2. Enforce the supported MV type to `VARCHAR`
3. Add integration test

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-04-11 15:21:19 +08:00
Patrick Weizhi Xu 982dd2834b
enhance: add materialized view search info (#30888)
issue: #29892 

This PR
1. Pass Materialized View (MV) search information obtained from the
expression parsing planning procedure to Knowhere. It only performs when
MV is enabled and the partition key is involved in the expression. The
search information includes:
1. Touched field_id and the count of related categories in the
expression. E.g., `color == red && color == blue` yields `field_id ->
2`.
2. Whether the expression only includes AND (&&) logical operator,
default `true`.
    3. Whether the expression has NOT (!) operator, default `false`.
4. Store if turning on MV on the proxy to eliminate reading from
paramtable for every search request.
5. Renames to MV.

## Rebuttals
1. Did not write in `ExtractInfoPlanNodeVisitor` since the new scalar
framework was introduced and this part might be removed in the future.
2. Currently only interested in `==` and `in` expression, `string` data
type, anything else is a bonus.
3. Leave handling expressions like `F == A || F == A` for future works
of the optimizer.

## Detailed MV Info

![image](https://github.com/milvus-io/milvus/assets/6563846/b27c08a0-9fd3-4474-8897-30a3d6d6b36f)

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-03-21 11:19:07 +08:00
jaime 4b0c3dd377
enhance: index meta use independent rather than global meta lock (#30869)
issue: https://github.com/milvus-io/milvus/issues/30837

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-03-04 16:56:59 +08:00
Patrick Weizhi Xu 0907d76253
enhance: pass partition key scalar info if enabled when build vector index (#29931)
issue: #29892 

Pass optional scalar IVF offsets to Cardinal

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-01-24 00:04:55 +08:00
Bingyi Sun ad866d2889
feat: integrate storagev2 into index build process (#28995)
issue: https://github.com/milvus-io/milvus/issues/28994

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-12-13 17:24:38 +08:00
cai.zhang fd68837f84
Fix bug for setting index state when indexnode connecting failed (#28220)
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-11-07 23:34:26 +08:00
yah01 be980fbc38
Refine state check (#27541)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-11 21:01:35 +08:00
Jiquan Long 370fdaf50d
Record engine version for segment index (#27384)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-09-28 18:03:28 +08:00
jaime 7f7c71ea7d
Decoupling client and server API in types interface (#27186)
Co-authored-by:: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-09-26 09:57:25 +08:00
SimFG 26f06dd732
Format the code (#27275)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
yah01 b4f86ea55e
Construct all success status with merr (#27226)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-20 10:57:23 +08:00
Enwei Jiao fb0705df1b
Decouple basetable and componentparam (#26725)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-09-05 10:31:48 +08:00
aoiasd 4cb6351f87
update isDelete of index task meta when delete task from index builder (#25717)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-07-21 17:15:00 +08:00
congqixia 41af0a98fa
Use go-api/v2 for milvus-proto (#24770)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
congqixia 73a181d226
Fix get vector it timeout and improve some string const usage (#24141)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-05-16 17:41:22 +08:00
jaime c9d0c157ec
Move some modules from internal to public package (#22572)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
Enwei Jiao 697dedac7e
Use cockroachdb/errors to replace other error pkg (#22390)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-02-26 11:31:49 +08:00
congqixia 5986106037
Make paramtable init only once (#21782)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-01-19 14:53:44 +08:00
jaime 73d8509600
Improve ListSegments performance for datacoord catalog (#21477)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-01-06 14:33:36 +08:00
cai.zhang e5f408dceb
Merge IndexCoord and DataCoord (#21267)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-01-04 19:37:36 +08:00