b0bd290a6e
enhance: Use internal json(sonic) to replace std json lib ( #37708 )
...
Related to #35020
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-18 10:46:31 +08:00
5a23c80f20
fix: Change memoryCheck write lock to read lock ( #37525 )
...
See also: milvus-io#37493
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-11-15 10:44:31 +08:00
1d06d4324b
fix: Int64 overflow in JSON encoding ( #37657 )
...
issue: ##36621
- For simple types in a struct, add "string" to the JSON tag for
automatic string conversion during JSON encoding.
- For complex types in a struct, replace "int64" with "string."
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-11-14 22:52:30 +08:00
31a8d08bdd
fix: Correct varchar primarykey size calculation ( #37617 )
...
See also: #37582
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-11-14 14:16:38 +08:00
cd181e4c6d
fix: make NewDataSyncService idempotent of dispatcher ( #37576 )
...
issue: #37547
Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-14 11:00:31 +08:00
1e8ea4a7e7
feat: add segment/channel/task/slow query render ( #37561 )
...
issue: #36621
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-11-12 17:44:29 +08:00
12951f0abb
enhance: rename tokenizer to analyzer and check analyzer params ( #37478 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-11-10 16:12:26 +08:00
49657c4690
enhance: add create segment message, enable empty segment flush ( #37407 )
...
issue: #37172
- add redo interceptor to implement append context refresh. (make new
timetick)
- add create segment handler for flusher.
- make empty segment flushable and directly change it into dropped.
- add create segment message into wal when creating new growing segment.
- make the insert operation into following seq: createSegment -> insert
-> insert -> flushSegment.
- make manual flush into following seq: flushTs -> flushsegment ->
flushsegment -> manualflush.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-08 10:16:34 +08:00
81879425e1
enhance: Optimize the performance of stats task ( #37374 )
...
1. Increase the writer's `batchSize` to avoid multiple serialization
operations.
2. Perform asynchronous upload of binlog files to prevent blocking the
data processing flow.
3. Reduce multiple calls to `writer.Flush()`.
issue: https://github.com/milvus-io/milvus/issues/37373
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-11-08 10:08:27 +08:00
f348bd9441
feat: add segment,pipeline, replica and resourcegroup api for WebUI ( #37344 )
...
issue: #36621
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-11-07 11:52:25 +08:00
cae9e1c732
fix: drop collection failed if enable streaming service ( #37444 )
...
issue: #36858
- Start channel manager on datacoord, but with empty assign policy in
streaming service.
- Make collection at dropping state can be recovered by flusher to make
sure that
milvus consume the dropCollection message.
- Add backoff for flusher lifetime.
- remove the proxy watcher from timetick at rootcoord in streaming
service.
Also see the better fixup: #37176
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-07 10:26:26 +08:00
9d16b972ea
feat: add tasks page into management WebUI ( #37002 )
...
issue: #36621
1. Add API to access task runtime metrics, including:
- build index task
- compaction task
- import task
- balance (including load/release of segments/channels and some leader
tasks on querycoord)
- sync task
2. Add a debug model to the webpage by using debug=true or debug=false
in the URL query parameters to enable or disable debug mode.
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-10-28 10:13:29 +08:00
fd72151037
fix: merge datanode bm25 error after reload growing segment with no data ( #37154 )
...
Segment with numrow 0 don't init bm25 stats, cause flush with bm25 stats
failed.
relate: https://github.com/milvus-io/milvus/issues/37150
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-10-26 07:43:28 +08:00
20750c061b
fix: loss small bm25 binlogs ( #36942 )
...
Sync task don't flush small bm25 logs, cause growing segment bm25 stats
loss.
relate: https://github.com/milvus-io/milvus/issues/36805
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-10-17 14:49:23 +08:00
b172ea1093
fix: Remove enableLevelZeroSegment config ( #36535 )
...
See also: #36504
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-10-17 11:59:24 +08:00
8905b042f1
fix: add proportion for capacity seal policy in streaming flusher ( #36761 )
...
issue: #36760
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-10-14 14:47:22 +08:00
794e3ab7e5
fix: fail to init fg clears flushTs so that slows flush ( #36740 )
...
See also: #36709
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-10-11 17:37:04 +08:00
db34572c56
feat: support load and query with bm25 metric ( #36071 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-10-11 10:23:20 +08:00
f7b811450d
feat: add enable_tokenizer params to VarChar field ( #36480 )
...
issue: #35922
add an enable_tokenizer param to varchar field: must be set to true so
that a varchar field can enable_match or used as input of BM25 function
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-10-10 20:33:21 +08:00
a47abb2f2b
fix: stats log lost after disable stats log loading on flush ( #36592 )
...
issue: #36555
Signed-off-by: chyezh <chyezh@outlook.com>
2024-09-29 16:53:15 +08:00
80f25d497f
enhance: Add metrics to monitor import throughput and imported rows ( #36519 )
...
issue: https://github.com/milvus-io/milvus/issues/36518
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-28 17:31:15 +08:00
9e8cafcbe2
enhance: Skip loading bf in datanode ( #36367 )
...
Skip loading bf in datanode:
1. When watching vchannels, skip loading bloom filters for segments.
2. Bypass bloom filter checks for delete messages, directly writing to
L0 segments.
3. Remove flushed segments proactively after flush.
issue: https://github.com/milvus-io/milvus/issues/34585
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-26 10:11:15 +08:00
139787371e
feat: support embedding bm25 sparse vector and flush bm25 stats log ( #36036 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-09-19 10:57:12 +08:00
c6eb6c7cb2
enhance: Add error handler for write buffer ( #36216 )
...
Related to #36215
This PR add error handler setting option providing the possibility to
change error handling behavior other than panicking.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-13 10:11:09 +08:00
9871841fa0
fix: datacoord broker use querycoord broker config and get segment info timeout ( #36026 )
...
relate: https://github.com/milvus-io/milvus/issues/36025
Will case channel with large segment num watch failed.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-09-06 08:15:05 +08:00
6130a85444
enhance: Remove bf from streaming node ( #35902 )
...
Remove bf from streaming node:
1. When watching vchannels, skip loading bloom filters for segments.
2. Bypass bloom filter checks for delete messages, directly writing to
L0 segments.
3. Remove flushed segments proactively after flush.
issue: https://github.com/milvus-io/milvus/issues/33285 ,
https://github.com/milvus-io/milvus/issues/34585
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-03 14:17:02 +08:00
99dff06391
enhance: using streaming service in insert/upsert/flush/delete/querynode ( #35406 )
...
issue: #33285
- using streaming service in insert/upsert/flush/delete/querynode
- fixup flusher bugs and refactor the flush operation
- enable streaming service for dml and ddl
- pass the e2e when enabling streaming service
- pass the integration tst when enabling streaming service
---------
Signed-off-by: chyezh <chyezh@outlook.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-29 10:03:08 +08:00
ab532ae199
enhance: Add back BF lazy load logic for datanode watch channel ( #35646 )
...
Add back lazy loading statslog when watch dml channel on datanode.
Related to #22994 #27675
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-22 19:42:57 +08:00
41646c8439
feat: integrate new deltalog format ( #35522 )
...
See #34123
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-08-20 19:06:56 +08:00
80a7c78f28
enhance: import supports null in parquet and json formats ( #35558 )
...
#31728
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-08-20 16:50:55 +08:00
c0200eec39
enhance: limit getSegmentInfo batch size to avoid excced grpc message limit ( #35394 )
...
issue: #35395
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-15 19:17:00 +08:00
36d59867f3
fix: err cannot be captured in defer in data_sync_service ( #35448 )
...
https://github.com/milvus-io/milvus/issues/35447
Signed-off-by: fengjun2016 <jornfeng@gmail.com>
2024-08-14 10:26:52 +08:00
a690e6e004
fix: panic in timetick_sender ( #35451 )
...
https://github.com/milvus-io/milvus/issues/35450
Signed-off-by: fengjun2016 <jornfeng@gmail.com>
2024-08-14 10:10:53 +08:00
143502e708
enhance: Fix progress always zero due to integer divide ( #35374 )
...
See also #35183
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-08 22:26:16 +08:00
a4439cc911
enhance: Implement flusher in streamingNode ( #34942 )
...
- Implement flusher to:
- Manage the pipelines (creation, deletion, etc.)
- Manage the segment write buffer
- Manage sync operation (including receive flushMsg and execute flush)
- Add a new `GetChannelRecoveryInfo` RPC in DataCoord.
- Reorganize packages: `flushcommon` and `datanode`.
issue: https://github.com/milvus-io/milvus/issues/33285
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-02 18:30:23 +08:00
aa247f192d
enhance: remove unused code for StorageV2 ( #35132 )
...
issue: https://github.com/milvus-io/milvus/issues/34168
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-08-01 12:08:13 +08:00
de8a266d8a
enhance: Enable linux code checker ( #35084 )
...
See also #34483
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-30 15:53:51 +08:00
c45f38aa61
enhance: Update protobuf-go to protobuf-go v2 ( #34394 )
...
issue: #34252
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-29 11:31:51 +08:00
39c7e06bc5
enhance: add message and msgstream msgpack adaptor ( #34874 )
...
issue: #33285
- make message builder and message conversion type safe
- add adaptor type and function to adapt old msgstream msgpack and
message interface
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-22 20:59:42 +08:00
8aab6cbfac
enhance: Organize the common modules of streamingNode and dataNode ( #34773 )
...
1. Move the common modules of streamingNode and dataNode to flushcommon
2. Add new GetVChannels interface for rootcoord
issue: https://github.com/milvus-io/milvus/issues/33285
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-22 11:33:51 +08:00
congqixia
XuanYang-cn
jaime
Zhen Ye
aoiasd
yihao.dai
Buqian Zheng
Ted Xu
smellthemoon
wei liu
CharlesFeng
zhenshan.cao