milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
milvus/internal/core/src/storage
History
Buqian Zheng 070dfc77bf
feat: [Sparse Float Vector] segcore basics and index building (#30357) 
This commit adds sparse float vector support to segcore with the
following:

1. data type enum declarations
2. Adds corresponding data structures for handling sparse float vectors
in various scenarios, including:
* FieldData as a bridge between the binlog and the in memory data
structures
* mmap::Column as the in memory representation of a sparse float vector
column of a sealed segment;
* ConcurrentVector as the in memory representation of a sparse float
vector of a growing segment which supports inserts.
3. Adds logic in payload reader/writer to serialize/deserialize from/to
binlog
4. Adds the ability to allow the index node to build sparse float vector
index
5. Adds the ability to allow the query node to build growing index for
growing segment and temp index for sealed segment without index built

This commit also includes some code cleanness, comment improvement, and
some unit tests for sparse vector.

https://github.com/milvus-io/milvus/issues/29419

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 		2024-03-11 14:45:02 +08:00
..
azure-blob-storage enhance: restful support import jobs (#30343) 2024-01-31 17:57:04 +08:00
AliyunCredentialsProvider.cpp Migrate the ability to upload and download binlog to cpp (#22984) 2023-06-25 14:38:44 +08:00
AliyunCredentialsProvider.h Migrate the ability to upload and download binlog to cpp (#22984) 2023-06-25 14:38:44 +08:00
AliyunSTSClient.cpp Migrate the ability to upload and download binlog to cpp (#22984) 2023-06-25 14:38:44 +08:00
AliyunSTSClient.h Migrate the ability to upload and download binlog to cpp (#22984) 2023-06-25 14:38:44 +08:00
AzureChunkManager.cpp enhance: improve the segcore logs (#29372) 2023-12-23 21:52:43 +08:00
AzureChunkManager.h print azure sdk log (#28240) 2023-11-08 17:50:18 +08:00
BinlogReader.cpp Refactor duplicate error class into one place (#26985) 2023-09-11 20:43:17 +08:00
BinlogReader.h Refactor duplicate error class into one place (#26985) 2023-09-11 20:43:17 +08:00
CMakeLists.txt enhance: change opendal as compile configurable (#30384) 2024-02-20 19:16:52 +08:00
ChunkCache.cpp feat: [Sparse Float Vector] segcore basics and index building (#30357) 2024-03-11 14:45:02 +08:00
ChunkCache.h fix: Prevent ChunkCache use absolute path in All-in-one mode (#30666) 2024-02-19 20:58:51 +08:00
ChunkCacheSingleton.h Make read ahead policy in ChunkCache configurable (#27291) 2023-09-28 15:47:27 +08:00
ChunkManager.cpp feat: Support tencent cloud object storage for milvus (#30163) 2024-01-23 11:28:56 +08:00
ChunkManager.h feat: [Sparse Float Vector] segcore basics and index building (#30357) 2024-03-11 14:45:02 +08:00
DataCodec.cpp Add code for PanicInfo (#27364) 2023-09-27 12:01:28 +08:00
DataCodec.h enhance: Refactor runtime and expr framework (#28166) 2023-12-18 12:04:42 +08:00
DiskFileManagerImpl.cpp enhance: change opendal as compile configurable (#30384) 2024-02-20 19:16:52 +08:00
DiskFileManagerImpl.h enhance: pass partition key scalar info if enabled when build vector index (#29931) 2024-01-24 00:04:55 +08:00
Event.cpp feat: [Sparse Float Vector] segcore basics and index building (#30357) 2024-03-11 14:45:02 +08:00
Event.h enhance: Refactor runtime and expr framework (#28166) 2023-12-18 12:04:42 +08:00
FileManager.h enhance: improve the segcore logs (#29372) 2023-12-23 21:52:43 +08:00
IndexData.cpp Add code for PanicInfo (#27364) 2023-09-27 12:01:28 +08:00
IndexData.h [Cherry-Pick] Remove arrow uasge in FieldData (#22726) 2023-03-20 10:41:56 +08:00
InsertData.cpp Add code for PanicInfo (#27364) 2023-09-27 12:01:28 +08:00
InsertData.h [Cherry-Pick] Remove arrow uasge in FieldData (#22726) 2023-03-20 10:41:56 +08:00
LocalChunkManager.cpp enhance: Refactor runtime and expr framework (#28166) 2023-12-18 12:04:42 +08:00
LocalChunkManager.h Migrate the ability to upload and download binlog to cpp (#22984) 2023-06-25 14:38:44 +08:00
LocalChunkManagerSingleton.h Add ChunkCache: support get vector from storage (#26142) 2023-09-15 10:21:20 +08:00
MemFileManagerImpl.cpp enhance: convert the `GetObject` util to async (#30166) 2024-01-22 19:20:57 +08:00
MemFileManagerImpl.h enhance: Refactor runtime and expr framework (#28166) 2023-12-18 12:04:42 +08:00
MinioChunkManager.cpp feat: Support tencent cloud object storage for milvus (#30163) 2024-01-23 11:28:56 +08:00
MinioChunkManager.h feat: Support tencent cloud object storage for milvus (#30163) 2024-01-23 11:28:56 +08:00
OpenDALChunkManager.cpp enhance: improve the segcore logs (#29372) 2023-12-23 21:52:43 +08:00
OpenDALChunkManager.h feat: support inverted index (#28783) 2023-12-31 19:50:47 +08:00
PayloadReader.cpp feat: [Sparse Float Vector] segcore basics and index building (#30357) 2024-03-11 14:45:02 +08:00
PayloadReader.h enhance: Refactor runtime and expr framework (#28166) 2023-12-18 12:04:42 +08:00
PayloadStream.cpp Refactor duplicate error class into one place (#26985) 2023-09-11 20:43:17 +08:00
PayloadStream.h Migrate the ability to upload and download binlog to cpp (#22984) 2023-06-25 14:38:44 +08:00
PayloadWriter.cpp feat: [Sparse Float Vector] segcore basics and index building (#30357) 2024-03-11 14:45:02 +08:00
PayloadWriter.h Support json & array types (#23408) 2023-04-20 11:32:31 +08:00
RemoteChunkManagerSingleton.h enhance: change opendal as compile configurable (#30384) 2024-02-20 19:16:52 +08:00
SafeQueue.h Migrate the ability to upload and download binlog to cpp (#22984) 2023-06-25 14:38:44 +08:00
TencentCloudCredentialsProvider.cpp feat: Support tencent cloud object storage for milvus (#30163) 2024-01-23 11:28:56 +08:00
TencentCloudCredentialsProvider.h feat: Support tencent cloud object storage for milvus (#30163) 2024-01-23 11:28:56 +08:00
TencentCloudSTSClient.cpp feat: Support tencent cloud object storage for milvus (#30163) 2024-01-23 11:28:56 +08:00
TencentCloudSTSClient.h feat: Support tencent cloud object storage for milvus (#30163) 2024-01-23 11:28:56 +08:00
ThreadPool.cpp enhance: improve the segcore logs (#29372) 2023-12-23 21:52:43 +08:00
ThreadPool.h enhance: limit the max pool size to 16 (#30371) 2024-01-31 14:13:06 +08:00
ThreadPools.cpp enhance: improve the segcore logs (#29372) 2023-12-23 21:52:43 +08:00
ThreadPools.h enhance: improve the segcore logs (#29372) 2023-12-23 21:52:43 +08:00
Types.h feat: integrate storagev2 in building index of segcore (#28768) 2023-12-05 16:48:54 +08:00
Util.cpp feat: [Sparse Float Vector] segcore basics and index building (#30357) 2024-03-11 14:45:02 +08:00
Util.h enhance: convert the `GetObject` util to async (#30166) 2024-01-22 19:20:57 +08:00
milvus_storage.pc.in refine complie configuration (#17502) 2022-06-24 21:12:15 +08:00
parquet_c.cpp enhance: Refactor runtime and expr framework (#28166) 2023-12-18 12:04:42 +08:00
parquet_c.h feat: [Sparse Float Vector] segcore basics and index building (#30357) 2024-03-11 14:45:02 +08:00
prometheus_client.cpp add internal storage metrics (#28278) 2023-11-19 17:22:25 +08:00
prometheus_client.h add internal storage metrics (#28278) 2023-11-19 17:22:25 +08:00
storage_c.cpp add internal storage metrics (#28278) 2023-11-19 17:22:25 +08:00
storage_c.h add internal storage metrics (#28278) 2023-11-19 17:22:25 +08:00