Grow slice & map.growWork may cause a lot when segment number is large
for big K query. This PR pre-allocate space for reduce methods to avoid
this cost.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
add prometheus dependency for monitor module. Or else Some compilers may
report a compilation failure.
issue: #35077
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
1.fix compaction task not be cleaned correctly
2.add a new parameter to control compaction gc loop interval
3.remove some useless configs of clustering compaction
bug: #34764
Signed-off-by: wayblink <anyang.wang@zilliz.com>
after manual stop component by management restful api, `healthz` may
return unhealthy state. k8s may restart the pod to save the unhealthy
sate, and the manual stop operation will got unexpected result.
to solve this, we make `healthz` API skip the manual stopped component.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
#34778#34849
fix two problems:
1. count(*) incorrect, if growing insert duplicated (pk, timestamp)
pairs that pk and timestamp all same, need to keep just one pair.
2. count(*) may core dump, if get_real_count interface get snapshot and
do mvcc at not consistency status, mainly happens under concurrency.
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
issue: #33583
the old policy permit datanode has at most 2 more channels than other
datanode. so if milvus has 2 datanode and 2 channels, both 2 channels
will be assign to 1 datanode, left another datanode empty.
This PR refine the balance policy to solve channel unbalance on datanode
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
See also #34746
This PR add segment level field in response of
`GetPersistentSegmentInfo` and `GetQuerySegmentInfo`
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #33285
- register streaming coord service into datacoord.
- add new streaming node role.
- add global static switch to enable streaming service or not.
Signed-off-by: chyezh <chyezh@outlook.com>
issue: #33285
- add specialized mutable and immutable message, make type safe.
- add version based constructor and type.
Signed-off-by: chyezh <chyezh@outlook.com>
issue: #34972
fix string type data use memcpy to fill cause segv for not malloc enough
memory in advance.
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
issue: #34798
after we remove the task priority on query coord, to avoid load/release
segment blocked by too much balance task, we limit the balance task size
in each round. at same time, we reduce the balance interval to trigger
balance more frequently.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
1. support read and write null in segcore
will store valid_data(use uint8_t type to save memory) in fieldData.
2. support load null
binlog reader read and write data into column(sealed segment),
insertRecord(growing segment). In sealed segment, store valid_data
directly. In growing segment, considering prior implementation and easy
code reading, it covert uint8_t to fbvector<bool>, which may optimize in
future.
3. retrieve valid_data.
parse valid_data in search/query.
#31728
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
issue: #34595
pr#34596 to we add an overloaded factor to segment in delegator, which
cause same segment got different score in delegator and worker. which
may cause segment bounce between delegator and worker.
This PR use average score to compute the delegator overloaded factor, to
avoid segment bounce between delegator and worker.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue: #34685
knowhere needs a new json param `range_search_k` for RangeSearch to
early terminate the iterator.
Signed-off-by: min.tian <min.tian.cn@gmail.com>
issue: #33285
- make message builder and message conversion type safe
- add adaptor type and function to adapt old msgstream msgpack and
message interface
---------
Signed-off-by: chyezh <chyezh@outlook.com>
Patrick Weizhi Xu
congqixia
foxspy
wayblink
zhagnlu
wei liu
chyezh
Chun Han
XuanYang-cn
PowderLi
cai.zhang
jaime
Aldrin
sre-ci-robot
Jiquan Long
smellthemoon
Min Tian