issue: https://github.com/milvus-io/milvus/issues/43917
1. fix ngrma index to be mistakenly used for unsopported operation
2. fix potential uaf problem
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
Ref https://github.com/milvus-io/milvus/issues/42148
This PR supports create index for vector array (now, only for
`DataType.FLOAT_VECTOR`) and search on it.
The index type supported in this PR is `EMB_LIST_HNSW` and the metric
type is `MAX_SIM` only.
The way to use it:
```python
milvus_client = MilvusClient("xxx:19530")
schema = milvus_client.create_schema(enable_dynamic_field=True, auto_id=True)
...
struct_schema = milvus_client.create_struct_array_field_schema("struct_array_field")
...
struct_schema.add_field("struct_float_vec", DataType.ARRAY_OF_VECTOR, element_type=DataType.FLOAT_VECTOR, dim=128, max_capacity=1000)
...
schema.add_struct_array_field(struct_schema)
index_params = milvus_client.prepare_index_params()
index_params.add_index(field_name="struct_float_vec", index_type="EMB_LIST_HNSW", metric_type="MAX_SIM", index_params={"nlist": 128})
...
milvus_client.create_index(COLLECTION_NAME, schema=schema, index_params=index_params)
```
Note: This PR uses `Lims` to convey offsets of the vector array to
knowhere where vectors of multiple vector arrays are concatenated and we
need offsets to specify which vectors belong to which vector array.
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>
Related to #43936
This PR:
- Use `folly::SharedMutex` instead of `std::shared_mutex` preventing
starvation
- Use `folly::SharedMutex::WriteHolder/ReadHolder` instead of
std::shared_lock and std::unique_lock to get better performance
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Enables the compilation of SVE code for the bitset library if a C++
compiler supports it.
There are two conditions for enabling the SVE code
* a C++ compiler needs to have a `-march=armv8-a+sve`
* `arm_sve.h` header must be available
AFAIK, `gcc 7 does not support SVE`, `gcc 8` and `gcc 9` support SVE,
but have no `arm_sve.h` file, and only `gcc 10` has both.
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
The Out of Memory (OOM) error occurs because a handler retains the
entire ImportRecordBatch in memory. Consequently, even when child arrays
within the batch are flushed, the memory for the complete batch is not
released. We temporarily fixed by deep copying record batch in #43724.
The proposed fix is to split the RecordBatch into smaller sub-batches by
column group. These sub-batches will be transferred via CGO, then
reassembled before being written to storage using the Storage V2 API.
Thus we can achieve zero-copy and only transferring references in CGO.
related: #43310
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
Related to #43230
This PR
- Move segcore setup function to `initcore` package to remove cgo
dependency from pkg
- Register core callback only for components depends on segcore
- Rectify `UpdateLogLevel` implementation
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #43725
This patch add assertion preventing segment reloading same field column.
Also improve the message info when pk already exists.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #43660
This patch reduces the unwanted offset&ts entries having same timestamp
of delete record. Under large amount of upsert, this false hit could
increase large amount of memory usage while applying delete.
The next step could be passing a callback to `search_pk_func_` to handle
hit entry streamingly.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #43655
This patch add a padding when writing mmap file for ScalarSortedIndex in
case of mmap falure due to 0 mmap length.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #43626
Similar to previous pr #43321, null bitmap could be dislocated if the
bitset ptr does not count the offset of array
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #43592
When delete records are large, search pk one by one will result into
many `Pincells` call which creates lots of futures.
This patch make search pk execute in batch to reduce this cost.
Also add `GetAllChunks` API to utilize `PinAllCells` to reduce pins.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #43584
There might be concurrent calls on `translator.get_cells`. The channel
cannot be shared among these calls, otherwise the logic will break.
This patch create new channel for each `get_cells` invocation in case of
data race.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #43584
Directly accessing `fields_` in `get_raw_data` may have race if load vec
index happens concurrently during getting raw data.
This PR make `bulk_subscript` hold shared_ptr of field column prevent
field column being release during reading it.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
relate: https://github.com/milvus-io/milvus/issues/42792
Add char group tokenizer which support use costum char group or use some
build-in char group as delimiters.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Related to #43584
When `LoadWithStrategy` throw exception, the ex was wrapped in the
returned future. If the future is not handled, this exception would be
ignored.
This patch add `future.get()` to get exception if any.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Spade A
zhagnlu
congqixia
Alexander Guzhva
liliu-z
aoiasd
Gao
yihao.dai
foxspy
sthuang
sparknack
Chun Han
Buqian Zheng
Bingyi Sun
Xianhui Lin
Ted Xu