milvus/internal/core/src/segcore
smellthemoon 5616b7e8d2
enhance: support null in c data_datacodec and load null value (#32183)
1. support read and write null in segcore
    will store valid_data(use uint8_t type to save memory) in fieldData.
2. support load null
binlog reader read and write data into column(sealed segment),
insertRecord(growing segment). In sealed segment, store valid_data
directly. In growing segment, considering prior implementation and easy
code reading, it covert uint8_t to fbvector<bool>, which may optimize in
future.
3.  retrieve valid_data.
    parse valid_data in search/query.
#31728

---------

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-07-23 16:07:51 +08:00
..
reduce enhance: optimize search reduce perf(#32507) (#34607) 2024-07-22 20:57:43 +08:00
AckResponder.h enhance: add new LRU cache impl (#30360) 2024-02-27 20:58:40 +08:00
CMakeLists.txt feat: support group_size for search_group_by(#33544) (#33720) 2024-07-12 10:17:36 +08:00
Collection.cpp enhance: improve the segcore logs (#29372) 2023-12-23 21:52:43 +08:00
Collection.h fix: fix searchPlan metricType modified concurrently (#30227) 2024-01-26 14:03:09 +08:00
ConcurrentVector.cpp feat: [Sparse Float Vector] segcore basics and index building (#30357) 2024-03-11 14:45:02 +08:00
ConcurrentVector.h enhance: support null in c data_datacodec and load null value (#32183) 2024-07-23 16:07:51 +08:00
DeletedRecord.h enhance: mark duplicated pk as deleted (#34586) 2024-07-16 14:25:39 +08:00
FieldIndexing.cpp enhance: growing segment support mmap (#32633) 2024-06-18 14:42:00 +08:00
FieldIndexing.h enhance: support null in c data_datacodec and load null value (#32183) 2024-07-23 16:07:51 +08:00
IndexConfigGenerator.cpp enhance: support sparse cardinal hnsw index (#33656) 2024-06-12 16:57:55 +08:00
IndexConfigGenerator.h enhance: support sparse cardinal hnsw index (#33656) 2024-06-12 16:57:55 +08:00
InsertRecord.h enhance: support null in c data_datacodec and load null value (#32183) 2024-07-23 16:07:51 +08:00
Record.h Fix deleted data is still visible (#24849) 2023-06-16 17:16:41 +08:00
ReduceStructure.h fix:possible out-of-bound due to groupby when reduing(#30711) (#31200) 2024-03-14 13:07:03 +08:00
ReduceUtils.cpp feat: LRU cache implementation (#32567) 2024-05-06 20:29:30 +08:00
ReduceUtils.h feat: LRU cache implementation (#32567) 2024-05-06 20:29:30 +08:00
SealedIndexingRecord.h enhance: add new LRU cache impl (#30360) 2024-02-27 20:58:40 +08:00
SegcoreConfig.cpp Add code for PanicInfo (#27364) 2023-09-27 12:01:28 +08:00
SegcoreConfig.h replace loaded binlog with binlog index for search performance (#27673) 2023-11-01 02:20:15 +08:00
SegmentGrowing.h enhance: mark duplicated pk as deleted (#34586) 2024-07-16 14:25:39 +08:00
SegmentGrowingImpl.cpp enhance: support null in c data_datacodec and load null value (#32183) 2024-07-23 16:07:51 +08:00
SegmentGrowingImpl.h enhance: mark duplicated pk as deleted (#34586) 2024-07-16 14:25:39 +08:00
SegmentInterface.cpp enhance: mark duplicated pk as deleted (#34586) 2024-07-16 14:25:39 +08:00
SegmentInterface.h enhance: mark duplicated pk as deleted (#34586) 2024-07-16 14:25:39 +08:00
SegmentSealed.h enhance: mark duplicated pk as deleted (#34586) 2024-07-16 14:25:39 +08:00
SegmentSealedImpl.cpp enhance: support null in c data_datacodec and load null value (#32183) 2024-07-23 16:07:51 +08:00
SegmentSealedImpl.h enhance: mark duplicated pk as deleted (#34586) 2024-07-16 14:25:39 +08:00
TimestampIndex.cpp Re-format cpp code (#22513) 2023-03-02 15:55:49 +08:00
TimestampIndex.h Re-format cpp code (#22513) 2023-03-02 15:55:49 +08:00
Types.h feat: support inverted index for array (#33452) 2024-05-31 09:47:47 +08:00
Utils.cpp enhance: support null in c data_datacodec and load null value (#32183) 2024-07-23 16:07:51 +08:00
Utils.h enhance: support null in c data_datacodec and load null value (#32183) 2024-07-23 16:07:51 +08:00
check_vec_index_c.cpp enhance: check index with data type (#33880) 2024-06-19 10:15:59 +08:00
check_vec_index_c.h enhance: check index with data type (#33880) 2024-06-19 10:15:59 +08:00
collection_c.cpp Refactor collection's cgo call (#28055) 2023-11-02 13:02:13 +08:00
collection_c.h Refactor collection's cgo call (#28055) 2023-11-02 13:02:13 +08:00
load_field_data_c.cpp enhance: Enhance and correct exception module (#33705) 2024-06-23 21:22:01 +08:00
load_field_data_c.h feat: integrate storagev2 in building index of segcore (#28768) 2023-12-05 16:48:54 +08:00
load_index_c.cpp feat: support inverted index for array (#33452) 2024-05-31 09:47:47 +08:00
load_index_c.h feat: support inverted index for array (#33452) 2024-05-31 09:47:47 +08:00
metrics_c.cpp Fill trailing \0 for c metrics value (#26578) 2023-08-23 18:42:24 +08:00
metrics_c.h Support show knowhere prometheus metrics (#23102) 2023-04-07 15:34:29 +08:00
milvus_segcore.pc.in refine complie configuration (#17502) 2022-06-24 21:12:15 +08:00
pkVisitor.h feat: LRU cache implementation (#32567) 2024-05-06 20:29:30 +08:00
plan_c.cpp enhance: retrieve output fields after local reduce (#32346) 2024-04-25 09:49:26 +08:00
plan_c.h enhance: retrieve output fields after local reduce (#32346) 2024-04-25 09:49:26 +08:00
reduce_c.cpp feat: support group_size for search_group_by(#33544) (#33720) 2024-07-12 10:17:36 +08:00
reduce_c.h enhance: add more trace for search & query (#32734) 2024-05-07 13:03:29 +08:00
segcore_init_c.cpp enhance: add score compute consistency config for knowhere (#32997) 2024-05-13 14:21:31 +08:00
segcore_init_c.h enhance: add score compute consistency config for knowhere (#32997) 2024-05-13 14:21:31 +08:00
segment_c.cpp enhance: support null in c data_datacodec and load null value (#32183) 2024-07-23 16:07:51 +08:00
segment_c.h enhance: mark duplicated pk as deleted (#34586) 2024-07-16 14:25:39 +08:00