milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #323 from milvus-io/0.5.3 

Merge from 0.5.3
pull/336/head
Jin Hai 2019-11-13 18:25:07 +08:00 committed by GitHub
parent 7c658f852a 240e616150
commit 69fdd13303
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
55 changed files with 1383 additions and 1865 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

17
CHANGELOG.md
View File

@ -1,6 +1,21 @@
# Changelog
Please mark all change in change log and use the ticket from JIRA.
# Milvus 0.5.3 (TODO)
## Bug
- \#258 - Bytes type in proto cause big-endian/little-endian problem
## Feature
## Improvement
- \#204 - improve grpc performance in search
- \#207 - Add more unittest for config set/get
- \#208 - optimize unittest to support run single test more easily
- \#284 - Change C++ SDK to shared library
- \#260 - C++ SDK README
## Task
# Milvus 0.5.2 (2019-11-07)
@ -10,7 +25,7 @@ Please mark all change in change log and use the ticket from JIRA.
## Feature
## Improvement
- \#190 - Update default config:use_blas_threshold to 1100 and server version printout to 0.52
- \#190 - Update default config:use_blas_threshold to 1100 and server version printout to 0.5.2
## Task

4
README.md
View File

@ -8,7 +8,7 @@
![Release](https://img.shields.io/badge/release-v0.5.2-yellowgreen)
![Release_date](https://img.shields.io/badge/release%20date-November-yellowgreen)
[中文版](README_CN.md)
[中文版](README_CN.md) | [日本語版](README_JP.md)
## What is Milvus
@ -16,7 +16,7 @@ Milvus is the world's fastest similarity search engine for massive-scale feature
For more detailed introduction of Milvus and its architecture, see [Milvus overview](https://www.milvus.io/docs/en/aboutmilvus/overview/).
Milvus provides stable [Python](https://github.com/milvus-io/pymilvus), [Java](https://github.com/milvus-io/milvus-sdk-java) and C++ APIs.
Milvus provides stable [Python](https://github.com/milvus-io/pymilvus), [Java](https://github.com/milvus-io/milvus-sdk-java) and [C++](https://github.com/milvus-io/milvus/tree/master/core/src/sdk) APIs.
Keep up-to-date with newest releases and latest updates by reading Milvus [release notes](https://www.milvus.io/docs/en/release/v0.5.2/).

153
README_JP.md
View File

@ -13,142 +13,24 @@
## 概要
Milvus は大規模な特徴ベクトルにむかう類似性検索エンジンです。不均質な計算アーキテクチャーに基づいて効率を最大化出来ます。数十億のベクタの中に目標を検索できるまで数ミリ秒しかかからず、最低限の計算資源だけが必要です。
Milvusは世界中一番早い特徴ベクトルにむかう類似性検索エンジンです。不均質な計算アーキテクチャーに基づいて効率を最大化出来ます。数十億のベクタの中に目標を検索できるまで数ミリ秒しかかからず、最低限の計算資源だけが必要です。
Milvus は安定的な Python、Java 又は C++ APIsを提供します。
Milvusは安定的なPython、Java又は C++ APIsを提供します。
Milvus [リリースノート](https://milvus.io/docs/en/release/v0.5.2/)を読んで最新バージョンや更新情報を手に入れます。
- 不均質な計算アーキテクチャー
Milvusは不均質な計算アーキテクチャーに基づいて効率を最大化出来ます。
- 様々な索引方法
Milvusはさまざまな索引方法を支えます。量子化、 木、 グラフに基づいて索引を作成できます。
- 知能てきな資源管理
Milvusはデータセットと利用できる資源を基づいて、自動的に検索アルゴリズムと索引作成方法を選びます。
- 水平拡張
Milvusはオンラインとオフラインに記憶域と計算を簡単な命令で拡張できます。
- 高い可用性
MilvusはKubernetes枠組みと統合するので、 単一障害点を避けられます。
- 高い互換性
Milvusはほぼ全ての深層学習モデルと主要なプログラミング言語と互換性があります。
- やすい使い方
Milvusは簡単にインストールできます。ベクタ以外のことを心配する必要がありますん。
- 視覚化てきな監視モード
Prometheusに基づいてGUIでシステムの性能を監視出来ます。
## アーキテクチャー
![Milvus_arch](https://github.com/milvus-io/docs/blob/master/assets/milvus_arch.png)
## はじめに
### ハードウェア要件
DockerでMilvusをインストールすることは簡単です。[Milvusインストール案内](https://milvus.io/docs/en/userguide/install_milvus/) を参考してください。ソースからMilvusを構築するために、[ソースから構築する](install.md)を参考してください。
| コンポーネント | お勧めの配置 |
| --------- | ----------------------------------- |
| CPU | Intel CPU Haswell 以上 |
| GPU | NVIDIA Pascal series 以上 |
| RAM メモリ | 8 GB 以上 (データ規模に関わる) |
| ハードディスク | SATA 3.0 SSD 以上 |
Milvusをコンフィグするために、[Milvusコンフィグ](https://github.com/milvus-io/docs/blob/master/reference/milvus_config.md)を読んでください。
### Dockerでインストールする
### 初めてのMilvusプログラムを試す
DockerでMilvusをインストールすることは簡単です。 [Milvusインストール案内](https://milvus.io/docs/en/userguide/install_milvus/) を参考してください。
[Python](https://www.milvus.io/docs/en/userguide/example_code/)や[Java](https://github.com/milvus-io/milvus-sdk-java/tree/master/examples)などのサンプルコードを使ってMilvusプログラムを試す。
### ソースから構築する
#### ソフトウェア要件
- Ubuntu 18.04 以上
- CMake 3.14 以上
- CUDA 10.0 以上
- NVIDIA driver 418 以上
#### コンパイル
##### 1 依存コンポーネントをインストールする
```shell
$ cd [Milvus sourcecode path]/core
./ubuntu_build_deps.sh
```
##### 2 構築する
```shell
$ cd [Milvus sourcecode path]/core
$ ./build.sh -t Debug
or
$ ./build.sh -t Release
```
構築が完成するとき、 Milvusを実行するために必要なものは全てこのディレクトリにあります： `[Milvus root path]/core/milvus`。
#### Milvusサーバーを実行する
```shell
$ cd [Milvus root path]/core/milvus
```
`lib/`　ディレクトリを　`LD_LIBRARY_PATH`　に添付する。
```shell
$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/milvus/lib
```
Milvusサーバーを実行する。
```shell
$ cd scripts
$ ./start_server.sh
```
Milvusサーバーを止めるために、次のコードを実行する：
```shell
$ ./stop_server.sh
```
`conf/server_config.yaml` と `conf/log_config.conf` でMilvusをコンフィグするために、 [Milvusコンフィグ](https://github.com/milvus-io/docs/blob/master/reference/milvus_config.md)を読んでください。
### 初めてのMilvusプログラムを実行する
#### Pythonサンプルコードを実行する
[Python 3.5](https://www.python.org/downloads/)以上のバージョンがインストールされていることを確かめてください。
Milvus Python SDK　をインストールする。
```shell
# Install Milvus Python SDK
$ pip install pymilvus==0.2.3
```
新しいファイル　`example.py`　を作成し、 [Pythonサンプルコード]( https://github.com/milvus-io/pymilvus/blob/master/examples/advanced_example.py)を添付してください。
サンプルコードを実行する。
```shell
# Run Milvus Python example
$ python3 example.py
```
#### C++サンプルコードを実行する
C++サンプルコードを実行するために、次のコマンドをつかってください。
```shell
# Run Milvus C++ example
@ -156,11 +38,9 @@ $ python3 example.py
$ ./sdk_simple
```
#### Javaサンプルコードを実行する
## Milvusロードマップ
Java 8以上のバージョンがインストールされていることを確かめてください。
[このリンク](https://github.com/milvus-io/milvus-sdk-java/tree/master/examples)でサンプルコードを手に入れます。
[ロードマップ](https://milvus.io/docs/en/roadmap/)を読んで、追加する予定の特性が分かります。
## 貢献規約
@ -172,23 +52,20 @@ Java 8以上のバージョンがインストールされていることを確
他の貢献者と交流したい場合は、Milvusの [slackチャンネル](https://join.slack.com/t/milvusio/shared_invite/enQtNzY1OTQ0NDI3NjMzLWNmYmM1NmNjOTQ5MGI5NDhhYmRhMGU5M2NhNzhhMDMzY2MzNDdlYjM5ODQ5MmE3ODFlYzU3YjJkNmVlNDQ2ZTk)に参加してください。
## Milvusロードマップ
[ロードマップ](https://milvus.io/docs/en/roadmap/)を読んで、追加する予定の特性が分かります。
## 参考情報
[Milvus公式サイト](https://www.milvus.io)
- [Milvus.io](https://www.milvus.io)
[Milvusドキュメント](https://www.milvus.io/docs/en/userguide/install_milvus/)
- [Milvus](https://github.com/milvus-io/bootcamp)
[Milvusトレーニングセンター](https://github.com/milvus-io/bootcamp)
- [Milvus Medium](https://medium.com/@milvusio)
[Milvusブロック](https://www.milvus.io/blog/)
- [Milvus CSDN](https://zilliz.blog.csdn.net/)
[Milvus CSDN](https://zilliz.blog.csdn.net/)
- [Milvus ツイッター](https://twitter.com/milvusio)
[Milvusロードマップ](https://milvus.io/docs/en/roadmap/)
- [Milvus Facebook](https://www.facebook.com/io.milvus.5)
## ライセンス

2
ci/jenkins/Jenkinsfile vendored
View File

@ -75,7 +75,7 @@ pipeline {
steps {
container('milvus-build-env') {
script {
load "${env.WORKSPACE}/ci/jenkins/step/package.groovy"
load "${env.WORKSPACE}/ci/jenkins/step/packaging.groovy"
}
}
}

13
ci/jenkins/scripts/packaging.sh Executable file
View File

@ -0,0 +1,13 @@
#!/usr/bin/env bash
set -ex
pip3 install -r requirements.txt
./yaml_processor.py merge -f /opt/milvus/conf/server_config.yaml -m ../yaml/update_server_config.yaml -i && \
rm /opt/milvus/conf/server_config.yaml.bak
if [ -d "/opt/milvus/unittest" ]; then
rm -rf "/opt/milvus/unittest"
fi
tar -zcvf ./${PROJECT_NAME}-${PACKAGE_VERSION}.tar.gz -C /opt/ milvus

2
ci/jenkins/step/deploySingle2Dev.groovy
View File

@ -1,7 +1,7 @@
sh 'helm init --client-only --skip-refresh --stable-repo-url https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts'
sh 'helm repo update'
dir ('milvus-helm') {
checkout([$class: 'GitSCM', branches: [[name: "0.5.0"]], userRemoteConfigs: [[url: "https://github.com/milvus-io/milvus-helm.git", name: 'origin', refspec: "+refs/heads/0.5.0:refs/remotes/origin/0.5.0"]]])
checkout([$class: 'GitSCM', branches: [[name: "0.5.3"]], userRemoteConfigs: [[url: "https://github.com/milvus-io/milvus-helm.git", name: 'origin', refspec: "+refs/heads/0.5.3:refs/remotes/origin/0.5.3"]]])
dir ("milvus-gpu") {
sh "helm install --wait --timeout 300 --set engine.image.tag=${DOCKER_VERSION} --set expose.type=clusterIP --name ${env.PIPELINE_NAME}-${env.BUILD_NUMBER}-single-gpu -f ci/db_backend/sqlite_values.yaml -f ci/filebeat/values.yaml --namespace milvus ."
}

13
ci/jenkins/step/package.groovy
View File

@ -1,13 +0,0 @@
timeout(time: 5, unit: 'MINUTES') {
dir ("ci/jenkins/scripts") {
sh "pip3 install -r requirements.txt"
sh "./yaml_processor.py merge -f /opt/milvus/conf/server_config.yaml -m ../yaml/update_server_config.yaml -i && rm /opt/milvus/conf/server_config.yaml.bak"
}
sh "tar -zcvf ./${PROJECT_NAME}-${PACKAGE_VERSION}.tar.gz -C /opt/ milvus"
withCredentials([usernamePassword(credentialsId: "${params.JFROG_CREDENTIALS_ID}", usernameVariable: 'JFROG_USERNAME', passwordVariable: 'JFROG_PASSWORD')]) {
def uploadStatus = sh(returnStatus: true, script: "curl -u${JFROG_USERNAME}:${JFROG_PASSWORD} -T ./${PROJECT_NAME}-${PACKAGE_VERSION}.tar.gz ${params.JFROG_ARTFACTORY_URL}/milvus/package/${PROJECT_NAME}-${PACKAGE_VERSION}.tar.gz")
if (uploadStatus != 0) {
error("\" ${PROJECT_NAME}-${PACKAGE_VERSION}.tar.gz \" upload to \" ${params.JFROG_ARTFACTORY_URL}/milvus/package/${PROJECT_NAME}-${PACKAGE_VERSION}.tar.gz \" failed!")
}
}
}

11
ci/jenkins/step/packaging.groovy Normal file
View File

@ -0,0 +1,11 @@
timeout(time: 5, unit: 'MINUTES') {
dir ("ci/jenkins/scripts") {
sh "./packaging.sh"
withCredentials([usernamePassword(credentialsId: "${params.JFROG_CREDENTIALS_ID}", usernameVariable: 'JFROG_USERNAME', passwordVariable: 'JFROG_PASSWORD')]) {
def uploadStatus = sh(returnStatus: true, script: "curl -u${JFROG_USERNAME}:${JFROG_PASSWORD} -T ./${PROJECT_NAME}-${PACKAGE_VERSION}.tar.gz ${params.JFROG_ARTFACTORY_URL}/milvus/package/${PROJECT_NAME}-${PACKAGE_VERSION}.tar.gz")
if (uploadStatus != 0) {
error("\" ${PROJECT_NAME}-${PACKAGE_VERSION}.tar.gz \" upload to \" ${params.JFROG_ARTFACTORY_URL}/milvus/package/${PROJECT_NAME}-${PACKAGE_VERSION}.tar.gz \" failed!")
}
}
}
}

2
ci/jenkins/step/singleDevNightlyTest.groovy
View File

@ -8,7 +8,7 @@ timeout(time: 90, unit: 'MINUTES') {
if (!fileExists('milvus-helm')) {
dir ("milvus-helm") {
checkout([$class: 'GitSCM', branches: [[name: "0.5.0"]], userRemoteConfigs: [[url: "https://github.com/milvus-io/milvus-helm.git", name: 'origin', refspec: "+refs/heads/0.5.0:refs/remotes/origin/0.5.0"]]])
checkout([$class: 'GitSCM', branches: [[name: "0.5.3"]], userRemoteConfigs: [[url: "https://github.com/milvus-io/milvus-helm.git", name: 'origin', refspec: "+refs/heads/0.5.3:refs/remotes/origin/0.5.3"]]])
}
}
dir ("milvus-helm") {

2
ci/jenkins/step/singleDevTest.groovy
View File

@ -10,7 +10,7 @@ timeout(time: 60, unit: 'MINUTES') {
// if (!fileExists('milvus-helm')) {
// dir ("milvus-helm") {
// checkout([$class: 'GitSCM', branches: [[name: "0.5.0"]], userRemoteConfigs: [[url: "https://github.com/milvus-io/milvus-helm.git", name: 'origin', refspec: "+refs/heads/0.5.0:refs/remotes/origin/0.5.0"]]])
// checkout([$class: 'GitSCM', branches: [[name: "0.5.3"]], userRemoteConfigs: [[url: "https://github.com/milvus-io/milvus-helm.git", name: 'origin', refspec: "+refs/heads/0.5.3:refs/remotes/origin/0.5.3"]]])
// }
// }
// dir ("milvus-helm") {

3
ci/jenkinsfile/packaged_milvus.groovy
View File

@ -5,6 +5,9 @@ container('milvus-build-env') {
gitlabCommitStatus(name: 'Packaged Engine') {
if (fileExists('milvus')) {
try {
if (fileExists('milvus/unittest')) {
sh "rm -rf ./milvus/unittest"
}
sh "tar -zcvf ./${PROJECT_NAME}-engine-${PACKAGE_VERSION}.tar.gz ./milvus"
def fileTransfer = load "${env.WORKSPACE}/ci/function/file_transfer.groovy"
fileTransfer.FileTransfer("${PROJECT_NAME}-engine-${PACKAGE_VERSION}.tar.gz", "${PROJECT_NAME}/engine/${JOB_NAME}-${BUILD_ID}", 'nas storage')

2
core/CMakeLists.txt
View File

@ -71,7 +71,7 @@ if(MILVUS_VERSION_MAJOR STREQUAL ""
OR MILVUS_VERSION_MINOR STREQUAL ""
OR MILVUS_VERSION_PATCH STREQUAL "")
message(WARNING "Failed to determine Milvus version from git branch name")
set(MILVUS_VERSION "0.5.2")
set(MILVUS_VERSION "0.5.3")
endif()
message(STATUS "Build version = ${MILVUS_VERSION}")

6
core/src/grpc/README.md Normal file
View File

@ -0,0 +1,6 @@
We manually change two APIs in "milvus.pd.h":
add_vector_data()
add_row_id_array()
add_ids()
add_distances()
If proto files need be generated again, remember to re-change above APIs.

44
core/src/grpc/gen-milvus/milvus.grpc.pb.cc
View File

@ -201,60 +201,60 @@ void MilvusService::Stub::experimental_async::Insert(::grpc::ClientContext* cont
return ::grpc_impl::internal::ClientAsyncResponseReaderFactory< ::milvus::grpc::VectorIds>::Create(channel_.get(), cq, rpcmethod_Insert_, context, request, false);
}
::grpc::Status MilvusService::Stub::Search(::grpc::ClientContext* context, const ::milvus::grpc::SearchParam& request, ::milvus::grpc::TopKQueryResultList* response) {
::grpc::Status MilvusService::Stub::Search(::grpc::ClientContext* context, const ::milvus::grpc::SearchParam& request, ::milvus::grpc::TopKQueryResult* response) {
return ::grpc::internal::BlockingUnaryCall(channel_.get(), rpcmethod_Search_, context, request, response);
}
void MilvusService::Stub::experimental_async::Search(::grpc::ClientContext* context, const ::milvus::grpc::SearchParam* request, ::milvus::grpc::TopKQueryResultList* response, std::function<void(::grpc::Status)> f) {
void MilvusService::Stub::experimental_async::Search(::grpc::ClientContext* context, const ::milvus::grpc::SearchParam* request, ::milvus::grpc::TopKQueryResult* response, std::function<void(::grpc::Status)> f) {
::grpc_impl::internal::CallbackUnaryCall(stub_->channel_.get(), stub_->rpcmethod_Search_, context, request, response, std::move(f));
}
void MilvusService::Stub::experimental_async::Search(::grpc::ClientContext* context, const ::grpc::ByteBuffer* request, ::milvus::grpc::TopKQueryResultList* response, std::function<void(::grpc::Status)> f) {
void MilvusService::Stub::experimental_async::Search(::grpc::ClientContext* context, const ::grpc::ByteBuffer* request, ::milvus::grpc::TopKQueryResult* response, std::function<void(::grpc::Status)> f) {
::grpc_impl::internal::CallbackUnaryCall(stub_->channel_.get(), stub_->rpcmethod_Search_, context, request, response, std::move(f));
}
void MilvusService::Stub::experimental_async::Search(::grpc::ClientContext* context, const ::milvus::grpc::SearchParam* request, ::milvus::grpc::TopKQueryResultList* response, ::grpc::experimental::ClientUnaryReactor* reactor) {
void MilvusService::Stub::experimental_async::Search(::grpc::ClientContext* context, const ::milvus::grpc::SearchParam* request, ::milvus::grpc::TopKQueryResult* response, ::grpc::experimental::ClientUnaryReactor* reactor) {
::grpc_impl::internal::ClientCallbackUnaryFactory::Create(stub_->channel_.get(), stub_->rpcmethod_Search_, context, request, response, reactor);
}
void MilvusService::Stub::experimental_async::Search(::grpc::ClientContext* context, const ::grpc::ByteBuffer* request, ::milvus::grpc::TopKQueryResultList* response, ::grpc::experimental::ClientUnaryReactor* reactor) {
void MilvusService::Stub::experimental_async::Search(::grpc::ClientContext* context, const ::grpc::ByteBuffer* request, ::milvus::grpc::TopKQueryResult* response, ::grpc::experimental::ClientUnaryReactor* reactor) {
::grpc_impl::internal::ClientCallbackUnaryFactory::Create(stub_->channel_.get(), stub_->rpcmethod_Search_, context, request, response, reactor);
}
::grpc::ClientAsyncResponseReader< ::milvus::grpc::TopKQueryResultList>* MilvusService::Stub::AsyncSearchRaw(::grpc::ClientContext* context, const ::milvus::grpc::SearchParam& request, ::grpc::CompletionQueue* cq) {
return ::grpc_impl::internal::ClientAsyncResponseReaderFactory< ::milvus::grpc::TopKQueryResultList>::Create(channel_.get(), cq, rpcmethod_Search_, context, request, true);
::grpc::ClientAsyncResponseReader< ::milvus::grpc::TopKQueryResult>* MilvusService::Stub::AsyncSearchRaw(::grpc::ClientContext* context, const ::milvus::grpc::SearchParam& request, ::grpc::CompletionQueue* cq) {
return ::grpc_impl::internal::ClientAsyncResponseReaderFactory< ::milvus::grpc::TopKQueryResult>::Create(channel_.get(), cq, rpcmethod_Search_, context, request, true);
}
::grpc::ClientAsyncResponseReader< ::milvus::grpc::TopKQueryResultList>* MilvusService::Stub::PrepareAsyncSearchRaw(::grpc::ClientContext* context, const ::milvus::grpc::SearchParam& request, ::grpc::CompletionQueue* cq) {
return ::grpc_impl::internal::ClientAsyncResponseReaderFactory< ::milvus::grpc::TopKQueryResultList>::Create(channel_.get(), cq, rpcmethod_Search_, context, request, false);
::grpc::ClientAsyncResponseReader< ::milvus::grpc::TopKQueryResult>* MilvusService::Stub::PrepareAsyncSearchRaw(::grpc::ClientContext* context, const ::milvus::grpc::SearchParam& request, ::grpc::CompletionQueue* cq) {
return ::grpc_impl::internal::ClientAsyncResponseReaderFactory< ::milvus::grpc::TopKQueryResult>::Create(channel_.get(), cq, rpcmethod_Search_, context, request, false);
}
::grpc::Status MilvusService::Stub::SearchInFiles(::grpc::ClientContext* context, const ::milvus::grpc::SearchInFilesParam& request, ::milvus::grpc::TopKQueryResultList* response) {
::grpc::Status MilvusService::Stub::SearchInFiles(::grpc::ClientContext* context, const ::milvus::grpc::SearchInFilesParam& request, ::milvus::grpc::TopKQueryResult* response) {
return ::grpc::internal::BlockingUnaryCall(channel_.get(), rpcmethod_SearchInFiles_, context, request, response);
}
void MilvusService::Stub::experimental_async::SearchInFiles(::grpc::ClientContext* context, const ::milvus::grpc::SearchInFilesParam* request, ::milvus::grpc::TopKQueryResultList* response, std::function<void(::grpc::Status)> f) {
void MilvusService::Stub::experimental_async::SearchInFiles(::grpc::ClientContext* context, const ::milvus::grpc::SearchInFilesParam* request, ::milvus::grpc::TopKQueryResult* response, std::function<void(::grpc::Status)> f) {
::grpc_impl::internal::CallbackUnaryCall(stub_->channel_.get(), stub_->rpcmethod_SearchInFiles_, context, request, response, std::move(f));
}
void MilvusService::Stub::experimental_async::SearchInFiles(::grpc::ClientContext* context, const ::grpc::ByteBuffer* request, ::milvus::grpc::TopKQueryResultList* response, std::function<void(::grpc::Status)> f) {
void MilvusService::Stub::experimental_async::SearchInFiles(::grpc::ClientContext* context, const ::grpc::ByteBuffer* request, ::milvus::grpc::TopKQueryResult* response, std::function<void(::grpc::Status)> f) {
::grpc_impl::internal::CallbackUnaryCall(stub_->channel_.get(), stub_->rpcmethod_SearchInFiles_, context, request, response, std::move(f));
}
void MilvusService::Stub::experimental_async::SearchInFiles(::grpc::ClientContext* context, const ::milvus::grpc::SearchInFilesParam* request, ::milvus::grpc::TopKQueryResultList* response, ::grpc::experimental::ClientUnaryReactor* reactor) {
void MilvusService::Stub::experimental_async::SearchInFiles(::grpc::ClientContext* context, const ::milvus::grpc::SearchInFilesParam* request, ::milvus::grpc::TopKQueryResult* response, ::grpc::experimental::ClientUnaryReactor* reactor) {
::grpc_impl::internal::ClientCallbackUnaryFactory::Create(stub_->channel_.get(), stub_->rpcmethod_SearchInFiles_, context, request, response, reactor);
}
void MilvusService::Stub::experimental_async::SearchInFiles(::grpc::ClientContext* context, const ::grpc::ByteBuffer* request, ::milvus::grpc::TopKQueryResultList* response, ::grpc::experimental::ClientUnaryReactor* reactor) {
void MilvusService::Stub::experimental_async::SearchInFiles(::grpc::ClientContext* context, const ::grpc::ByteBuffer* request, ::milvus::grpc::TopKQueryResult* response, ::grpc::experimental::ClientUnaryReactor* reactor) {
::grpc_impl::internal::ClientCallbackUnaryFactory::Create(stub_->channel_.get(), stub_->rpcmethod_SearchInFiles_, context, request, response, reactor);
}
::grpc::ClientAsyncResponseReader< ::milvus::grpc::TopKQueryResultList>* MilvusService::Stub::AsyncSearchInFilesRaw(::grpc::ClientContext* context, const ::milvus::grpc::SearchInFilesParam& request, ::grpc::CompletionQueue* cq) {
return ::grpc_impl::internal::ClientAsyncResponseReaderFactory< ::milvus::grpc::TopKQueryResultList>::Create(channel_.get(), cq, rpcmethod_SearchInFiles_, context, request, true);
::grpc::ClientAsyncResponseReader< ::milvus::grpc::TopKQueryResult>* MilvusService::Stub::AsyncSearchInFilesRaw(::grpc::ClientContext* context, const ::milvus::grpc::SearchInFilesParam& request, ::grpc::CompletionQueue* cq) {
return ::grpc_impl::internal::ClientAsyncResponseReaderFactory< ::milvus::grpc::TopKQueryResult>::Create(channel_.get(), cq, rpcmethod_SearchInFiles_, context, request, true);
}
::grpc::ClientAsyncResponseReader< ::milvus::grpc::TopKQueryResultList>* MilvusService::Stub::PrepareAsyncSearchInFilesRaw(::grpc::ClientContext* context, const ::milvus::grpc::SearchInFilesParam& request, ::grpc::CompletionQueue* cq) {
return ::grpc_impl::internal::ClientAsyncResponseReaderFactory< ::milvus::grpc::TopKQueryResultList>::Create(channel_.get(), cq, rpcmethod_SearchInFiles_, context, request, false);
::grpc::ClientAsyncResponseReader< ::milvus::grpc::TopKQueryResult>* MilvusService::Stub::PrepareAsyncSearchInFilesRaw(::grpc::ClientContext* context, const ::milvus::grpc::SearchInFilesParam& request, ::grpc::CompletionQueue* cq) {
return ::grpc_impl::internal::ClientAsyncResponseReaderFactory< ::milvus::grpc::TopKQueryResult>::Create(channel_.get(), cq, rpcmethod_SearchInFiles_, context, request, false);
}
::grpc::Status MilvusService::Stub::DescribeTable(::grpc::ClientContext* context, const ::milvus::grpc::TableName& request, ::milvus::grpc::TableSchema* response) {
@ -510,12 +510,12 @@ MilvusService::Service::Service() {
AddMethod(new ::grpc::internal::RpcServiceMethod(
MilvusService_method_names[5],
::grpc::internal::RpcMethod::NORMAL_RPC,
new ::grpc::internal::RpcMethodHandler< MilvusService::Service, ::milvus::grpc::SearchParam, ::milvus::grpc::TopKQueryResultList>(
new ::grpc::internal::RpcMethodHandler< MilvusService::Service, ::milvus::grpc::SearchParam, ::milvus::grpc::TopKQueryResult>(
std::mem_fn(&MilvusService::Service::Search), this)));
AddMethod(new ::grpc::internal::RpcServiceMethod(
MilvusService_method_names[6],
::grpc::internal::RpcMethod::NORMAL_RPC,
new ::grpc::internal::RpcMethodHandler< MilvusService::Service, ::milvus::grpc::SearchInFilesParam, ::milvus::grpc::TopKQueryResultList>(
new ::grpc::internal::RpcMethodHandler< MilvusService::Service, ::milvus::grpc::SearchInFilesParam, ::milvus::grpc::TopKQueryResult>(
std::mem_fn(&MilvusService::Service::SearchInFiles), this)));
AddMethod(new ::grpc::internal::RpcServiceMethod(
MilvusService_method_names[7],
@ -597,14 +597,14 @@ MilvusService::Service::~Service() {
return ::grpc::Status(::grpc::StatusCode::UNIMPLEMENTED, "");
}
::grpc::Status MilvusService::Service::Search(::grpc::ServerContext* context, const ::milvus::grpc::SearchParam* request, ::milvus::grpc::TopKQueryResultList* response) {
::grpc::Status MilvusService::Service::Search(::grpc::ServerContext* context, const ::milvus::grpc::SearchParam* request, ::milvus::grpc::TopKQueryResult* response) {
(void) context;
(void) request;
(void) response;
return ::grpc::Status(::grpc::StatusCode::UNIMPLEMENTED, "");
}
::grpc::Status MilvusService::Service::SearchInFiles(::grpc::ServerContext* context, const ::milvus::grpc::SearchInFilesParam* request, ::milvus::grpc::TopKQueryResultList* response) {
::grpc::Status MilvusService::Service::SearchInFiles(::grpc::ServerContext* context, const ::milvus::grpc::SearchInFilesParam* request, ::milvus::grpc::TopKQueryResult* response) {
(void) context;
(void) request;
(void) response;

540
core/src/grpc/gen-milvus/milvus.grpc.pb.h
View File

File diff suppressed because it is too large Load Diff

1058
core/src/grpc/gen-milvus/milvus.pb.cc
View File

File diff suppressed because it is too large Load Diff

566
core/src/grpc/gen-milvus/milvus.pb.h
View File

@ -48,7 +48,7 @@ struct TableStruct_milvus_2eproto {
PROTOBUF_SECTION_VARIABLE(protodesc_cold);
static const ::PROTOBUF_NAMESPACE_ID::internal::AuxillaryParseTableField aux[]
PROTOBUF_SECTION_VARIABLE(protodesc_cold);
static const ::PROTOBUF_NAMESPACE_ID::internal::ParseTable schema[19]
static const ::PROTOBUF_NAMESPACE_ID::internal::ParseTable schema[17]
PROTOBUF_SECTION_VARIABLE(protodesc_cold);
static const ::PROTOBUF_NAMESPACE_ID::internal::FieldMetadata field_metadata[];
static const ::PROTOBUF_NAMESPACE_ID::internal::SerializationTable serialization_table[];
@ -75,9 +75,6 @@ extern IndexParamDefaultTypeInternal _IndexParam_default_instance_;
class InsertParam;
class InsertParamDefaultTypeInternal;
extern InsertParamDefaultTypeInternal _InsertParam_default_instance_;
class QueryResult;
class QueryResultDefaultTypeInternal;
extern QueryResultDefaultTypeInternal _QueryResult_default_instance_;
class Range;
class RangeDefaultTypeInternal;
extern RangeDefaultTypeInternal _Range_default_instance_;
@ -108,9 +105,6 @@ extern TableSchemaDefaultTypeInternal _TableSchema_default_instance_;
class TopKQueryResult;
class TopKQueryResultDefaultTypeInternal;
extern TopKQueryResultDefaultTypeInternal _TopKQueryResult_default_instance_;
class TopKQueryResultList;
class TopKQueryResultListDefaultTypeInternal;
extern TopKQueryResultListDefaultTypeInternal _TopKQueryResultList_default_instance_;
class VectorIds;
class VectorIdsDefaultTypeInternal;
extern VectorIdsDefaultTypeInternal _VectorIds_default_instance_;
@ -123,7 +117,6 @@ template<> ::milvus::grpc::DeleteByRangeParam* Arena::CreateMaybeMessage<::milvu
template<> ::milvus::grpc::Index* Arena::CreateMaybeMessage<::milvus::grpc::Index>(Arena*);
template<> ::milvus::grpc::IndexParam* Arena::CreateMaybeMessage<::milvus::grpc::IndexParam>(Arena*);
template<> ::milvus::grpc::InsertParam* Arena::CreateMaybeMessage<::milvus::grpc::InsertParam>(Arena*);
template<> ::milvus::grpc::QueryResult* Arena::CreateMaybeMessage<::milvus::grpc::QueryResult>(Arena*);
template<> ::milvus::grpc::Range* Arena::CreateMaybeMessage<::milvus::grpc::Range>(Arena*);
template<> ::milvus::grpc::RowRecord* Arena::CreateMaybeMessage<::milvus::grpc::RowRecord>(Arena*);
template<> ::milvus::grpc::SearchInFilesParam* Arena::CreateMaybeMessage<::milvus::grpc::SearchInFilesParam>(Arena*);
@ -134,7 +127,6 @@ template<> ::milvus::grpc::TableNameList* Arena::CreateMaybeMessage<::milvus::gr
template<> ::milvus::grpc::TableRowCount* Arena::CreateMaybeMessage<::milvus::grpc::TableRowCount>(Arena*);
template<> ::milvus::grpc::TableSchema* Arena::CreateMaybeMessage<::milvus::grpc::TableSchema>(Arena*);
template<> ::milvus::grpc::TopKQueryResult* Arena::CreateMaybeMessage<::milvus::grpc::TopKQueryResult>(Arena*);
template<> ::milvus::grpc::TopKQueryResultList* Arena::CreateMaybeMessage<::milvus::grpc::TopKQueryResultList>(Arena*);
template<> ::milvus::grpc::VectorIds* Arena::CreateMaybeMessage<::milvus::grpc::VectorIds>(Arena*);
PROTOBUF_NAMESPACE_CLOSE
namespace milvus {
@ -870,7 +862,8 @@ class RowRecord :
void clear_vector_data();
float vector_data(int index) const;
void set_vector_data(int index, float value);
void add_vector_data(float value);
//void add_vector_data(float value);
void add_vector_data(std::vector<float>::const_iterator begin, std::vector<float>::const_iterator end);
const ::PROTOBUF_NAMESPACE_ID::RepeatedField< float >&
vector_data() const;
::PROTOBUF_NAMESPACE_ID::RepeatedField< float >*
@ -1021,7 +1014,9 @@ class InsertParam :
void clear_row_id_array();
::PROTOBUF_NAMESPACE_ID::int64 row_id_array(int index) const;
void set_row_id_array(int index, ::PROTOBUF_NAMESPACE_ID::int64 value);
void add_row_id_array(::PROTOBUF_NAMESPACE_ID::int64 value);
//void add_row_id_array(::PROTOBUF_NAMESPACE_ID::int64 value);
void add_row_id_array(std::vector<::PROTOBUF_NAMESPACE_ID::int64>::const_iterator begin,
std::vector<::PROTOBUF_NAMESPACE_ID::int64>::const_iterator end);
const ::PROTOBUF_NAMESPACE_ID::RepeatedField< ::PROTOBUF_NAMESPACE_ID::int64 >&
row_id_array() const;
::PROTOBUF_NAMESPACE_ID::RepeatedField< ::PROTOBUF_NAMESPACE_ID::int64 >*
@ -1530,144 +1525,6 @@ class SearchInFilesParam :
};
// -------------------------------------------------------------------
class QueryResult :
public ::PROTOBUF_NAMESPACE_ID::Message /* @@protoc_insertion_point(class_definition:milvus.grpc.QueryResult) */ {
public:
QueryResult();
virtual ~QueryResult();
QueryResult(const QueryResult& from);
QueryResult(QueryResult&& from) noexcept
: QueryResult() {
*this = ::std::move(from);
}
inline QueryResult& operator=(const QueryResult& from) {
CopyFrom(from);
return *this;
}
inline QueryResult& operator=(QueryResult&& from) noexcept {
if (GetArenaNoVirtual() == from.GetArenaNoVirtual()) {
if (this != &from) InternalSwap(&from);
} else {
CopyFrom(from);
}
return *this;
}
static const ::PROTOBUF_NAMESPACE_ID::Descriptor* descriptor() {
return GetDescriptor();
}
static const ::PROTOBUF_NAMESPACE_ID::Descriptor* GetDescriptor() {
return GetMetadataStatic().descriptor;
}
static const ::PROTOBUF_NAMESPACE_ID::Reflection* GetReflection() {
return GetMetadataStatic().reflection;
}
static const QueryResult& default_instance();
static void InitAsDefaultInstance(); // FOR INTERNAL USE ONLY
static inline const QueryResult* internal_default_instance() {
return reinterpret_cast<const QueryResult*>(
&_QueryResult_default_instance_);
}
static constexpr int kIndexInFileMessages =
9;
friend void swap(QueryResult& a, QueryResult& b) {
a.Swap(&b);
}
inline void Swap(QueryResult* other) {
if (other == this) return;
InternalSwap(other);
}
// implements Message ----------------------------------------------
inline QueryResult* New() const final {
return CreateMaybeMessage<QueryResult>(nullptr);
}
QueryResult* New(::PROTOBUF_NAMESPACE_ID::Arena* arena) const final {
return CreateMaybeMessage<QueryResult>(arena);
}
void CopyFrom(const ::PROTOBUF_NAMESPACE_ID::Message& from) final;
void MergeFrom(const ::PROTOBUF_NAMESPACE_ID::Message& from) final;
void CopyFrom(const QueryResult& from);
void MergeFrom(const QueryResult& from);
PROTOBUF_ATTRIBUTE_REINITIALIZES void Clear() final;
bool IsInitialized() const final;
size_t ByteSizeLong() const final;
#if GOOGLE_PROTOBUF_ENABLE_EXPERIMENTAL_PARSER
const char* _InternalParse(const char* ptr, ::PROTOBUF_NAMESPACE_ID::internal::ParseContext* ctx) final;
#else
bool MergePartialFromCodedStream(
::PROTOBUF_NAMESPACE_ID::io::CodedInputStream* input) final;
#endif // GOOGLE_PROTOBUF_ENABLE_EXPERIMENTAL_PARSER
void SerializeWithCachedSizes(
::PROTOBUF_NAMESPACE_ID::io::CodedOutputStream* output) const final;
::PROTOBUF_NAMESPACE_ID::uint8* InternalSerializeWithCachedSizesToArray(
::PROTOBUF_NAMESPACE_ID::uint8* target) const final;
int GetCachedSize() const final { return _cached_size_.Get(); }
private:
inline void SharedCtor();
inline void SharedDtor();
void SetCachedSize(int size) const final;
void InternalSwap(QueryResult* other);
friend class ::PROTOBUF_NAMESPACE_ID::internal::AnyMetadata;
static ::PROTOBUF_NAMESPACE_ID::StringPiece FullMessageName() {
return "milvus.grpc.QueryResult";
}
private:
inline ::PROTOBUF_NAMESPACE_ID::Arena* GetArenaNoVirtual() const {
return nullptr;
}
inline void* MaybeArenaPtr() const {
return nullptr;
}
public:
::PROTOBUF_NAMESPACE_ID::Metadata GetMetadata() const final;
private:
static ::PROTOBUF_NAMESPACE_ID::Metadata GetMetadataStatic() {
::PROTOBUF_NAMESPACE_ID::internal::AssignDescriptors(&::descriptor_table_milvus_2eproto);
return ::descriptor_table_milvus_2eproto.file_level_metadata[kIndexInFileMessages];
}
public:
// nested types ----------------------------------------------------
// accessors -------------------------------------------------------
enum : int {
kIdFieldNumber = 1,
kDistanceFieldNumber = 2,
};
// int64 id = 1;
void clear_id();
::PROTOBUF_NAMESPACE_ID::int64 id() const;
void set_id(::PROTOBUF_NAMESPACE_ID::int64 value);
// double distance = 2;
void clear_distance();
double distance() const;
void set_distance(double value);
// @@protoc_insertion_point(class_scope:milvus.grpc.QueryResult)
private:
class _Internal;
::PROTOBUF_NAMESPACE_ID::internal::InternalMetadataWithArena _internal_metadata_;
::PROTOBUF_NAMESPACE_ID::int64 id_;
double distance_;
mutable ::PROTOBUF_NAMESPACE_ID::internal::CachedSize _cached_size_;
friend struct ::TableStruct_milvus_2eproto;
};
// -------------------------------------------------------------------
class TopKQueryResult :
public ::PROTOBUF_NAMESPACE_ID::Message /* @@protoc_insertion_point(class_definition:milvus.grpc.TopKQueryResult) */ {
public:
@ -1710,7 +1567,7 @@ class TopKQueryResult :
&_TopKQueryResult_default_instance_);
}
static constexpr int kIndexInFileMessages =
10;
9;
friend void swap(TopKQueryResult& a, TopKQueryResult& b) {
a.Swap(&b);
@ -1781,156 +1638,35 @@ class TopKQueryResult :
// accessors -------------------------------------------------------
enum : int {
kQueryResultArraysFieldNumber = 1,
};
// repeated .milvus.grpc.QueryResult query_result_arrays = 1;
int query_result_arrays_size() const;
void clear_query_result_arrays();
::milvus::grpc::QueryResult* mutable_query_result_arrays(int index);
::PROTOBUF_NAMESPACE_ID::RepeatedPtrField< ::milvus::grpc::QueryResult >*
mutable_query_result_arrays();
const ::milvus::grpc::QueryResult& query_result_arrays(int index) const;
::milvus::grpc::QueryResult* add_query_result_arrays();
const ::PROTOBUF_NAMESPACE_ID::RepeatedPtrField< ::milvus::grpc::QueryResult >&
query_result_arrays() const;
// @@protoc_insertion_point(class_scope:milvus.grpc.TopKQueryResult)
private:
class _Internal;
::PROTOBUF_NAMESPACE_ID::internal::InternalMetadataWithArena _internal_metadata_;
::PROTOBUF_NAMESPACE_ID::RepeatedPtrField< ::milvus::grpc::QueryResult > query_result_arrays_;
mutable ::PROTOBUF_NAMESPACE_ID::internal::CachedSize _cached_size_;
friend struct ::TableStruct_milvus_2eproto;
};
// -------------------------------------------------------------------
class TopKQueryResultList :
public ::PROTOBUF_NAMESPACE_ID::Message /* @@protoc_insertion_point(class_definition:milvus.grpc.TopKQueryResultList) */ {
public:
TopKQueryResultList();
virtual ~TopKQueryResultList();
TopKQueryResultList(const TopKQueryResultList& from);
TopKQueryResultList(TopKQueryResultList&& from) noexcept
: TopKQueryResultList() {
*this = ::std::move(from);
}
inline TopKQueryResultList& operator=(const TopKQueryResultList& from) {
CopyFrom(from);
return *this;
}
inline TopKQueryResultList& operator=(TopKQueryResultList&& from) noexcept {
if (GetArenaNoVirtual() == from.GetArenaNoVirtual()) {
if (this != &from) InternalSwap(&from);
} else {
CopyFrom(from);
}
return *this;
}
static const ::PROTOBUF_NAMESPACE_ID::Descriptor* descriptor() {
return GetDescriptor();
}
static const ::PROTOBUF_NAMESPACE_ID::Descriptor* GetDescriptor() {
return GetMetadataStatic().descriptor;
}
static const ::PROTOBUF_NAMESPACE_ID::Reflection* GetReflection() {
return GetMetadataStatic().reflection;
}
static const TopKQueryResultList& default_instance();
static void InitAsDefaultInstance(); // FOR INTERNAL USE ONLY
static inline const TopKQueryResultList* internal_default_instance() {
return reinterpret_cast<const TopKQueryResultList*>(
&_TopKQueryResultList_default_instance_);
}
static constexpr int kIndexInFileMessages =
11;
friend void swap(TopKQueryResultList& a, TopKQueryResultList& b) {
a.Swap(&b);
}
inline void Swap(TopKQueryResultList* other) {
if (other == this) return;
InternalSwap(other);
}
// implements Message ----------------------------------------------
inline TopKQueryResultList* New() const final {
return CreateMaybeMessage<TopKQueryResultList>(nullptr);
}
TopKQueryResultList* New(::PROTOBUF_NAMESPACE_ID::Arena* arena) const final {
return CreateMaybeMessage<TopKQueryResultList>(arena);
}
void CopyFrom(const ::PROTOBUF_NAMESPACE_ID::Message& from) final;
void MergeFrom(const ::PROTOBUF_NAMESPACE_ID::Message& from) final;
void CopyFrom(const TopKQueryResultList& from);
void MergeFrom(const TopKQueryResultList& from);
PROTOBUF_ATTRIBUTE_REINITIALIZES void Clear() final;
bool IsInitialized() const final;
size_t ByteSizeLong() const final;
#if GOOGLE_PROTOBUF_ENABLE_EXPERIMENTAL_PARSER
const char* _InternalParse(const char* ptr, ::PROTOBUF_NAMESPACE_ID::internal::ParseContext* ctx) final;
#else
bool MergePartialFromCodedStream(
::PROTOBUF_NAMESPACE_ID::io::CodedInputStream* input) final;
#endif // GOOGLE_PROTOBUF_ENABLE_EXPERIMENTAL_PARSER
void SerializeWithCachedSizes(
::PROTOBUF_NAMESPACE_ID::io::CodedOutputStream* output) const final;
::PROTOBUF_NAMESPACE_ID::uint8* InternalSerializeWithCachedSizesToArray(
::PROTOBUF_NAMESPACE_ID::uint8* target) const final;
int GetCachedSize() const final { return _cached_size_.Get(); }
private:
inline void SharedCtor();
inline void SharedDtor();
void SetCachedSize(int size) const final;
void InternalSwap(TopKQueryResultList* other);
friend class ::PROTOBUF_NAMESPACE_ID::internal::AnyMetadata;
static ::PROTOBUF_NAMESPACE_ID::StringPiece FullMessageName() {
return "milvus.grpc.TopKQueryResultList";
}
private:
inline ::PROTOBUF_NAMESPACE_ID::Arena* GetArenaNoVirtual() const {
return nullptr;
}
inline void* MaybeArenaPtr() const {
return nullptr;
}
public:
::PROTOBUF_NAMESPACE_ID::Metadata GetMetadata() const final;
private:
static ::PROTOBUF_NAMESPACE_ID::Metadata GetMetadataStatic() {
::PROTOBUF_NAMESPACE_ID::internal::AssignDescriptors(&::descriptor_table_milvus_2eproto);
return ::descriptor_table_milvus_2eproto.file_level_metadata[kIndexInFileMessages];
}
public:
// nested types ----------------------------------------------------
// accessors -------------------------------------------------------
enum : int {
kTopkQueryResultFieldNumber = 2,
kIdsFieldNumber = 3,
kDistancesFieldNumber = 4,
kStatusFieldNumber = 1,
kRowNumFieldNumber = 2,
};
// repeated .milvus.grpc.TopKQueryResult topk_query_result = 2;
int topk_query_result_size() const;
void clear_topk_query_result();
::milvus::grpc::TopKQueryResult* mutable_topk_query_result(int index);
::PROTOBUF_NAMESPACE_ID::RepeatedPtrField< ::milvus::grpc::TopKQueryResult >*
mutable_topk_query_result();
const ::milvus::grpc::TopKQueryResult& topk_query_result(int index) const;
::milvus::grpc::TopKQueryResult* add_topk_query_result();
const ::PROTOBUF_NAMESPACE_ID::RepeatedPtrField< ::milvus::grpc::TopKQueryResult >&
topk_query_result() const;
// repeated int64 ids = 3;
int ids_size() const;
void clear_ids();
::PROTOBUF_NAMESPACE_ID::int64 ids(int index) const;
void set_ids(int index, ::PROTOBUF_NAMESPACE_ID::int64 value);
// void add_ids(::PROTOBUF_NAMESPACE_ID::int64 value);
void add_ids(std::vector<::PROTOBUF_NAMESPACE_ID::int64>::const_iterator begin,
std::vector<::PROTOBUF_NAMESPACE_ID::int64>::const_iterator end);
const ::PROTOBUF_NAMESPACE_ID::RepeatedField< ::PROTOBUF_NAMESPACE_ID::int64 >&
ids() const;
::PROTOBUF_NAMESPACE_ID::RepeatedField< ::PROTOBUF_NAMESPACE_ID::int64 >*
mutable_ids();
// repeated float distances = 4;
int distances_size() const;
void clear_distances();
float distances(int index) const;
void set_distances(int index, float value);
// void add_distances(float value);
void add_distances(std::vector<float>::const_iterator begin, std::vector<float>::const_iterator end);
const ::PROTOBUF_NAMESPACE_ID::RepeatedField< float >&
distances() const;
::PROTOBUF_NAMESPACE_ID::RepeatedField< float >*
mutable_distances();
// .milvus.grpc.Status status = 1;
bool has_status() const;
@ -1940,13 +1676,22 @@ class TopKQueryResultList :
::milvus::grpc::Status* mutable_status();
void set_allocated_status(::milvus::grpc::Status* status);
// @@protoc_insertion_point(class_scope:milvus.grpc.TopKQueryResultList)
// int64 row_num = 2;
void clear_row_num();
::PROTOBUF_NAMESPACE_ID::int64 row_num() const;
void set_row_num(::PROTOBUF_NAMESPACE_ID::int64 value);
// @@protoc_insertion_point(class_scope:milvus.grpc.TopKQueryResult)
private:
class _Internal;
::PROTOBUF_NAMESPACE_ID::internal::InternalMetadataWithArena _internal_metadata_;
::PROTOBUF_NAMESPACE_ID::RepeatedPtrField< ::milvus::grpc::TopKQueryResult > topk_query_result_;
::PROTOBUF_NAMESPACE_ID::RepeatedField< ::PROTOBUF_NAMESPACE_ID::int64 > ids_;
mutable std::atomic<int> _ids_cached_byte_size_;
::PROTOBUF_NAMESPACE_ID::RepeatedField< float > distances_;
mutable std::atomic<int> _distances_cached_byte_size_;
::milvus::grpc::Status* status_;
::PROTOBUF_NAMESPACE_ID::int64 row_num_;
mutable ::PROTOBUF_NAMESPACE_ID::internal::CachedSize _cached_size_;
friend struct ::TableStruct_milvus_2eproto;
};
@ -1994,7 +1739,7 @@ class StringReply :
&_StringReply_default_instance_);
}
static constexpr int kIndexInFileMessages =
12;
10;
friend void swap(StringReply& a, StringReply& b) {
a.Swap(&b);
@ -2141,7 +1886,7 @@ class BoolReply :
&_BoolReply_default_instance_);
}
static constexpr int kIndexInFileMessages =
13;
11;
friend void swap(BoolReply& a, BoolReply& b) {
a.Swap(&b);
@ -2282,7 +2027,7 @@ class TableRowCount :
&_TableRowCount_default_instance_);
}
static constexpr int kIndexInFileMessages =
14;
12;
friend void swap(TableRowCount& a, TableRowCount& b) {
a.Swap(&b);
@ -2423,7 +2168,7 @@ class Command :
&_Command_default_instance_);
}
static constexpr int kIndexInFileMessages =
15;
13;
friend void swap(Command& a, Command& b) {
a.Swap(&b);
@ -2560,7 +2305,7 @@ class Index :
&_Index_default_instance_);
}
static constexpr int kIndexInFileMessages =
16;
14;
friend void swap(Index& a, Index& b) {
a.Swap(&b);
@ -2698,7 +2443,7 @@ class IndexParam :
&_IndexParam_default_instance_);
}
static constexpr int kIndexInFileMessages =
17;
15;
friend void swap(IndexParam& a, IndexParam& b) {
a.Swap(&b);
@ -2855,7 +2600,7 @@ class DeleteByRangeParam :
&_DeleteByRangeParam_default_instance_);
}
static constexpr int kIndexInFileMessages =
18;
16;
friend void swap(DeleteByRangeParam& a, DeleteByRangeParam& b) {
a.Swap(&b);
@ -3401,9 +3146,14 @@ inline void RowRecord::set_vector_data(int index, float value) {
vector_data_.Set(index, value);
// @@protoc_insertion_point(field_set:milvus.grpc.RowRecord.vector_data)
}
inline void RowRecord::add_vector_data(float value) {
vector_data_.Add(value);
// @@protoc_insertion_point(field_add:milvus.grpc.RowRecord.vector_data)
//inline void RowRecord::add_vector_data(float value) {
// vector_data_.Add(value);
// // @@protoc_insertion_point(field_add:milvus.grpc.RowRecord.vector_data)
//}
inline void RowRecord::add_vector_data(std::vector<float>::const_iterator begin,
std::vector<float>::const_iterator end) {
vector_data_.Add(begin, end);
// @@protoc_insertion_point(field_add:milvus.grpc.RowRecord.vector_data)
}
inline const ::PROTOBUF_NAMESPACE_ID::RepeatedField< float >&
RowRecord::vector_data() const {
@ -3516,9 +3266,14 @@ inline void InsertParam::set_row_id_array(int index, ::PROTOBUF_NAMESPACE_ID::in
row_id_array_.Set(index, value);
// @@protoc_insertion_point(field_set:milvus.grpc.InsertParam.row_id_array)
}
inline void InsertParam::add_row_id_array(::PROTOBUF_NAMESPACE_ID::int64 value) {
row_id_array_.Add(value);
// @@protoc_insertion_point(field_add:milvus.grpc.InsertParam.row_id_array)
//inline void InsertParam::add_row_id_array(::PROTOBUF_NAMESPACE_ID::int64 value) {
// row_id_array_.Add(value);
// // @@protoc_insertion_point(field_add:milvus.grpc.InsertParam.row_id_array)
//}
inline void InsertParam::add_row_id_array(std::vector<::PROTOBUF_NAMESPACE_ID::int64>::const_iterator begin,
std::vector<::PROTOBUF_NAMESPACE_ID::int64>::const_iterator end) {
row_id_array_.Add(begin, end);
// @@protoc_insertion_point(field_add:milvus.grpc.InsertParam.row_id_array)
}
inline const ::PROTOBUF_NAMESPACE_ID::RepeatedField< ::PROTOBUF_NAMESPACE_ID::int64 >&
InsertParam::row_id_array() const {
@ -3875,101 +3630,35 @@ inline void SearchInFilesParam::set_allocated_search_param(::milvus::grpc::Searc
// -------------------------------------------------------------------
// QueryResult
// int64 id = 1;
inline void QueryResult::clear_id() {
id_ = PROTOBUF_LONGLONG(0);
}
inline ::PROTOBUF_NAMESPACE_ID::int64 QueryResult::id() const {
// @@protoc_insertion_point(field_get:milvus.grpc.QueryResult.id)
return id_;
}
inline void QueryResult::set_id(::PROTOBUF_NAMESPACE_ID::int64 value) {
id_ = value;
// @@protoc_insertion_point(field_set:milvus.grpc.QueryResult.id)
}
// double distance = 2;
inline void QueryResult::clear_distance() {
distance_ = 0;
}
inline double QueryResult::distance() const {
// @@protoc_insertion_point(field_get:milvus.grpc.QueryResult.distance)
return distance_;
}
inline void QueryResult::set_distance(double value) {
distance_ = value;
// @@protoc_insertion_point(field_set:milvus.grpc.QueryResult.distance)
}
// -------------------------------------------------------------------
// TopKQueryResult
// repeated .milvus.grpc.QueryResult query_result_arrays = 1;
inline int TopKQueryResult::query_result_arrays_size() const {
return query_result_arrays_.size();
}
inline void TopKQueryResult::clear_query_result_arrays() {
query_result_arrays_.Clear();
}
inline ::milvus::grpc::QueryResult* TopKQueryResult::mutable_query_result_arrays(int index) {
// @@protoc_insertion_point(field_mutable:milvus.grpc.TopKQueryResult.query_result_arrays)
return query_result_arrays_.Mutable(index);
}
inline ::PROTOBUF_NAMESPACE_ID::RepeatedPtrField< ::milvus::grpc::QueryResult >*
TopKQueryResult::mutable_query_result_arrays() {
// @@protoc_insertion_point(field_mutable_list:milvus.grpc.TopKQueryResult.query_result_arrays)
return &query_result_arrays_;
}
inline const ::milvus::grpc::QueryResult& TopKQueryResult::query_result_arrays(int index) const {
// @@protoc_insertion_point(field_get:milvus.grpc.TopKQueryResult.query_result_arrays)
return query_result_arrays_.Get(index);
}
inline ::milvus::grpc::QueryResult* TopKQueryResult::add_query_result_arrays() {
// @@protoc_insertion_point(field_add:milvus.grpc.TopKQueryResult.query_result_arrays)
return query_result_arrays_.Add();
}
inline const ::PROTOBUF_NAMESPACE_ID::RepeatedPtrField< ::milvus::grpc::QueryResult >&
TopKQueryResult::query_result_arrays() const {
// @@protoc_insertion_point(field_list:milvus.grpc.TopKQueryResult.query_result_arrays)
return query_result_arrays_;
}
// -------------------------------------------------------------------
// TopKQueryResultList
// .milvus.grpc.Status status = 1;
inline bool TopKQueryResultList::has_status() const {
inline bool TopKQueryResult::has_status() const {
return this != internal_default_instance() && status_ != nullptr;
}
inline const ::milvus::grpc::Status& TopKQueryResultList::status() const {
inline const ::milvus::grpc::Status& TopKQueryResult::status() const {
const ::milvus::grpc::Status* p = status_;
// @@protoc_insertion_point(field_get:milvus.grpc.TopKQueryResultList.status)
// @@protoc_insertion_point(field_get:milvus.grpc.TopKQueryResult.status)
return p != nullptr ? *p : *reinterpret_cast<const ::milvus::grpc::Status*>(
&::milvus::grpc::_Status_default_instance_);
}
inline ::milvus::grpc::Status* TopKQueryResultList::release_status() {
// @@protoc_insertion_point(field_release:milvus.grpc.TopKQueryResultList.status)
inline ::milvus::grpc::Status* TopKQueryResult::release_status() {
// @@protoc_insertion_point(field_release:milvus.grpc.TopKQueryResult.status)
::milvus::grpc::Status* temp = status_;
status_ = nullptr;
return temp;
}
inline ::milvus::grpc::Status* TopKQueryResultList::mutable_status() {
inline ::milvus::grpc::Status* TopKQueryResult::mutable_status() {
if (status_ == nullptr) {
auto* p = CreateMaybeMessage<::milvus::grpc::Status>(GetArenaNoVirtual());
status_ = p;
}
// @@protoc_insertion_point(field_mutable:milvus.grpc.TopKQueryResultList.status)
// @@protoc_insertion_point(field_mutable:milvus.grpc.TopKQueryResult.status)
return status_;
}
inline void TopKQueryResultList::set_allocated_status(::milvus::grpc::Status* status) {
inline void TopKQueryResult::set_allocated_status(::milvus::grpc::Status* status) {
::PROTOBUF_NAMESPACE_ID::Arena* message_arena = GetArenaNoVirtual();
if (message_arena == nullptr) {
delete reinterpret_cast< ::PROTOBUF_NAMESPACE_ID::MessageLite*>(status_);
@ -3985,37 +3674,90 @@ inline void TopKQueryResultList::set_allocated_status(::milvus::grpc::Status* st
}
status_ = status;
// @@protoc_insertion_point(field_set_allocated:milvus.grpc.TopKQueryResultList.status)
// @@protoc_insertion_point(field_set_allocated:milvus.grpc.TopKQueryResult.status)
}
// repeated .milvus.grpc.TopKQueryResult topk_query_result = 2;
inline int TopKQueryResultList::topk_query_result_size() const {
return topk_query_result_.size();
// int64 row_num = 2;
inline void TopKQueryResult::clear_row_num() {
row_num_ = PROTOBUF_LONGLONG(0);
}
inline void TopKQueryResultList::clear_topk_query_result() {
topk_query_result_.Clear();
inline ::PROTOBUF_NAMESPACE_ID::int64 TopKQueryResult::row_num() const {
// @@protoc_insertion_point(field_get:milvus.grpc.TopKQueryResult.row_num)
return row_num_;
}
inline ::milvus::grpc::TopKQueryResult* TopKQueryResultList::mutable_topk_query_result(int index) {
// @@protoc_insertion_point(field_mutable:milvus.grpc.TopKQueryResultList.topk_query_result)
return topk_query_result_.Mutable(index);
inline void TopKQueryResult::set_row_num(::PROTOBUF_NAMESPACE_ID::int64 value) {
row_num_ = value;
// @@protoc_insertion_point(field_set:milvus.grpc.TopKQueryResult.row_num)
}
inline ::PROTOBUF_NAMESPACE_ID::RepeatedPtrField< ::milvus::grpc::TopKQueryResult >*
TopKQueryResultList::mutable_topk_query_result() {
// @@protoc_insertion_point(field_mutable_list:milvus.grpc.TopKQueryResultList.topk_query_result)
return &topk_query_result_;
// repeated int64 ids = 3;
inline int TopKQueryResult::ids_size() const {
return ids_.size();
}
inline const ::milvus::grpc::TopKQueryResult& TopKQueryResultList::topk_query_result(int index) const {
// @@protoc_insertion_point(field_get:milvus.grpc.TopKQueryResultList.topk_query_result)
return topk_query_result_.Get(index);
inline void TopKQueryResult::clear_ids() {
ids_.Clear();
}
inline ::milvus::grpc::TopKQueryResult* TopKQueryResultList::add_topk_query_result() {
// @@protoc_insertion_point(field_add:milvus.grpc.TopKQueryResultList.topk_query_result)
return topk_query_result_.Add();
inline ::PROTOBUF_NAMESPACE_ID::int64 TopKQueryResult::ids(int index) const {
// @@protoc_insertion_point(field_get:milvus.grpc.TopKQueryResult.ids)
return ids_.Get(index);
}
inline const ::PROTOBUF_NAMESPACE_ID::RepeatedPtrField< ::milvus::grpc::TopKQueryResult >&
TopKQueryResultList::topk_query_result() const {
// @@protoc_insertion_point(field_list:milvus.grpc.TopKQueryResultList.topk_query_result)
return topk_query_result_;
inline void TopKQueryResult::set_ids(int index, ::PROTOBUF_NAMESPACE_ID::int64 value) {
ids_.Set(index, value);
// @@protoc_insertion_point(field_set:milvus.grpc.TopKQueryResult.ids)
}
//inline void TopKQueryResult::add_ids(::PROTOBUF_NAMESPACE_ID::int64 value) {
// ids_.Add(value);
// // @@protoc_insertion_point(field_add:milvus.grpc.TopKQueryResult.ids)
//}
inline void TopKQueryResult::add_ids(std::vector<::PROTOBUF_NAMESPACE_ID::int64>::const_iterator begin,
std::vector<::PROTOBUF_NAMESPACE_ID::int64>::const_iterator end) {
ids_.Add(begin,end);
// @@protoc_insertion_point(field_add:milvus.grpc.TopKQueryResult.ids)
}
inline const ::PROTOBUF_NAMESPACE_ID::RepeatedField< ::PROTOBUF_NAMESPACE_ID::int64 >&
TopKQueryResult::ids() const {
// @@protoc_insertion_point(field_list:milvus.grpc.TopKQueryResult.ids)
return ids_;
}
inline ::PROTOBUF_NAMESPACE_ID::RepeatedField< ::PROTOBUF_NAMESPACE_ID::int64 >*
TopKQueryResult::mutable_ids() {
// @@protoc_insertion_point(field_mutable_list:milvus.grpc.TopKQueryResult.ids)
return &ids_;
}
// repeated float distances = 4;
inline int TopKQueryResult::distances_size() const {
return distances_.size();
}
inline void TopKQueryResult::clear_distances() {
distances_.Clear();
}
inline float TopKQueryResult::distances(int index) const {
// @@protoc_insertion_point(field_get:milvus.grpc.TopKQueryResult.distances)
return distances_.Get(index);
}
inline void TopKQueryResult::set_distances(int index, float value) {
distances_.Set(index, value);
// @@protoc_insertion_point(field_set:milvus.grpc.TopKQueryResult.distances)
}
//inline void TopKQueryResult::add_distances(float value) {
// distances_.Add(value);
// // @@protoc_insertion_point(field_add:milvus.grpc.TopKQueryResult.distances)
//}
inline void TopKQueryResult::add_distances(std::vector<float>::const_iterator begin, std::vector<float>::const_iterator end) {
distances_.Add(begin, end);
// @@protoc_insertion_point(field_add:milvus.grpc.TopKQueryResult.distances)
}
inline const ::PROTOBUF_NAMESPACE_ID::RepeatedField< float >&
TopKQueryResult::distances() const {
// @@protoc_insertion_point(field_list:milvus.grpc.TopKQueryResult.distances)
return distances_;
}
inline ::PROTOBUF_NAMESPACE_ID::RepeatedField< float >*
TopKQueryResult::mutable_distances() {
// @@protoc_insertion_point(field_mutable_list:milvus.grpc.TopKQueryResult.distances)
return &distances_;
}
// -------------------------------------------------------------------
@ -4623,10 +4365,6 @@ inline void DeleteByRangeParam::set_allocated_table_name(std::string* table_name
// -------------------------------------------------------------------
// -------------------------------------------------------------------
// -------------------------------------------------------------------
// @@protoc_insertion_point(namespace_scope)

23
core/src/grpc/milvus.proto
View File

@ -84,24 +84,11 @@ message SearchInFilesParam {
/**
* @brief Query result params
*/
message QueryResult {
int64 id = 1;
double distance = 2;
}
/**
* @brief TopK query result
*/
message TopKQueryResult {
repeated QueryResult query_result_arrays = 1;
}
/**
* @brief List of topK query result
*/
message TopKQueryResultList {
Status status = 1;
repeated TopKQueryResult topk_query_result = 2;
int64 row_num = 2;
repeated int64 ids = 3;
repeated float distances = 4;
}
/**
@ -227,7 +214,7 @@ service MilvusService {
*
* @return query result array.
*/
rpc Search(SearchParam) returns (TopKQueryResultList) {}
rpc Search(SearchParam) returns (TopKQueryResult) {}
/**
* @brief Internal use query interface
@ -241,7 +228,7 @@ service MilvusService {
*
* @return query result array.
*/
rpc SearchInFiles(SearchInFilesParam) returns (TopKQueryResultList) {}
rpc SearchInFiles(SearchInFilesParam) returns (TopKQueryResult) {}
/**
* @brief Get table schema

3
core/src/index/cmake/ThirdPartyPackagesCore.cmake
View File

@ -247,7 +247,8 @@ if(CUSTOMIZATION)
# set(FAISS_MD5 "87fdd86351ffcaf3f80dc26ade63c44b") # commit-id 841a156e67e8e22cd8088e1b58c00afbf2efc30b branch-0.2.1
# set(FAISS_MD5 "f3b2ce3364c3fa7febd3aa7fdd0fe380") # commit-id 694e03458e6b69ce8a62502f71f69a614af5af8f branch-0.3.0
# set(FAISS_MD5 "bb30722c22390ce5f6759ccb216c1b2a") # commit-id d324db297475286afe107847c7fb7a0f9dc7e90e branch-0.3.0
set(FAISS_MD5 "2293cdb209c3718e3b19f3edae8b32b3") # commit-id a13c1205dc52977a9ad3b33a14efa958604a8bff branch-0.3.0
# set(FAISS_MD5 "2293cdb209c3718e3b19f3edae8b32b3") # commit-id a13c1205dc52977a9ad3b33a14efa958604a8bff branch-0.3.0
set(FAISS_MD5 "3632192f06b70e5f3fac8e2f08ced503") # commit-id fbd5b7566466c5665bb646f2b2f911c8544090f5 branch-0.3.0
endif()
else()
set(FAISS_SOURCE_URL "https://github.com/JinHai-CN/faiss/archive/1.6.0.tar.gz")

101
core/src/index/unittest/faiss_benchmark/faiss_benchmark_test.cpp
View File

@ -202,14 +202,14 @@ parse_ann_test_name(const std::string& ann_test_name, size_t& dim, faiss::Metric
int32_t
GetResultHitCount(const faiss::Index::idx_t* ground_index, const faiss::Index::idx_t* index, size_t ground_k, size_t k,
size_t nq, int32_t index_add_loops) {
assert(ground_k <= k);
size_t min_k = std::min(ground_k, k);
int hit = 0;
for (int i = 0; i < nq; i++) {
// count the num of results exist in ground truth result set
// each result replicates INDEX_ADD_LOOPS times
for (int j_c = 0; j_c < k; j_c++) {
int r_c = index[i * k + j_c];
for (int j_g = 0; j_g < ground_k / index_add_loops; j_g++) {
for (int j_g = 0; j_g < min_k / index_add_loops; j_g++) {
if (ground_index[i * ground_k + j_g] == r_c) {
hit++;
continue;
@ -283,7 +283,11 @@ load_base_data(faiss::Index*& index, const std::string& ann_test_name, const std
// add index multiple times to get ~1G data set
for (int i = 0; i < index_add_loops; i++) {
printf("[%.3f s] No.%d Indexing database, size %ld*%ld\n", elapsed() - t0, i, nb, d);
gpu_index->add(nb, xb);
std::vector<faiss::Index::idx_t> xids(nb);
for (int t = 0; t < nb; t++) {
xids[t] = i * nb + t;
}
gpu_index->add_with_ids(nb, xb, xids.data());
}
printf("[%.3f s] Coping GPU index to CPU\n", elapsed() - t0);
@ -359,13 +363,14 @@ test_with_nprobes(const std::string& ann_test_name, const std::string& index_key
const int32_t search_loops) {
double t0 = elapsed();
const size_t NQ = 1000, NQ_START = 10, NQ_STEP = 10;
const size_t K = 1000, K_START = 100, K_STEP = 10;
const std::vector<size_t> NQ = {10, 100};
const std::vector<size_t> K = {10, 100, 1000};
const size_t GK = 100; // topk of ground truth
std::unordered_map<size_t, std::string> mode_str_map = {
{MODE_CPU, "MODE_CPU"}, {MODE_MIX, "MODE_MIX"}, {MODE_GPU, "MODE_GPU"}};
double copy_time = 0.0;
faiss::Index *gpu_index, *index;
if (query_mode != MODE_CPU) {
faiss::gpu::GpuClonerOptions option;
@ -375,7 +380,6 @@ test_with_nprobes(const std::string& ann_test_name, const std::string& index_key
index_composition.index = cpu_index;
index_composition.quantizer = nullptr;
double copy_time;
switch (query_mode) {
case MODE_MIX: {
index_composition.mode = 1; // 0: all data, 1: copy quantizer, 2: copy data
@ -420,34 +424,39 @@ test_with_nprobes(const std::string& ann_test_name, const std::string& index_key
}
for (auto nprobe : nprobes) {
switch (query_mode) {
case MODE_CPU:
case MODE_MIX: {
faiss::ParameterSpace params;
std::string nprobe_str = "nprobe=" + std::to_string(nprobe);
params.set_index_parameters(index, nprobe_str.c_str());
break;
}
case MODE_GPU: {
faiss::gpu::GpuIndexIVF* gpu_index_ivf = dynamic_cast<faiss::gpu::GpuIndexIVF*>(index);
gpu_index_ivf->setNumProbes(nprobe);
// brute-force need not set nprobe
if (index_key.find("IDMap") == std::string::npos) {
switch (query_mode) {
case MODE_CPU:
case MODE_MIX: {
faiss::ParameterSpace params;
std::string nprobe_str = "nprobe=" + std::to_string(nprobe);
params.set_index_parameters(index, nprobe_str.c_str());
break;
}
case MODE_GPU: {
faiss::gpu::GpuIndexIVF* gpu_index_ivf = dynamic_cast<faiss::gpu::GpuIndexIVF*>(index);
gpu_index_ivf->setNumProbes(nprobe);
}
}
}
// output buffers
faiss::Index::idx_t* I = new faiss::Index::idx_t[NQ * K];
faiss::Index::distance_t* D = new faiss::Index::distance_t[NQ * K];
faiss::Index::idx_t* I = new faiss::Index::idx_t[NQ.back() * K.back()];
faiss::Index::distance_t* D = new faiss::Index::distance_t[NQ.back() * K.back()];
printf("\n%s | %s - %s | nprobe=%lu\n", ann_test_name.c_str(), index_key.c_str(),
mode_str_map[query_mode].c_str(), nprobe);
printf("======================================================================================\n");
for (size_t t_nq = NQ_START; t_nq <= NQ; t_nq *= NQ_STEP) { // nq = {10, 100, 1000}
for (size_t t_k = K_START; t_k <= K; t_k *= K_STEP) { // k = {100, 1000}
for (size_t j = 0; j < K.size(); j++) {
size_t t_k = K[j];
for (size_t i = 0; i < NQ.size(); i++) {
size_t t_nq = NQ[i];
faiss::indexIVF_stats.quantization_time = 0.0;
faiss::indexIVF_stats.search_time = 0.0;
double t_start = elapsed(), t_end;
for (int i = 0; i < search_loops; i++) {
for (int s = 0; s < search_loops; s++) {
index->search(t_nq, xq, t_k, D, I);
}
t_end = elapsed();
@ -466,7 +475,7 @@ test_with_nprobes(const std::string& ann_test_name, const std::string& index_key
printf("nq = %4ld, k = %4ld, elapse = %.4fs (quant = %.4fs, search = %.4fs), R@ = %.4f\n", t_nq, t_k,
(t_end - t_start) / search_loops, faiss::indexIVF_stats.quantization_time / 1000 / search_loops,
faiss::indexIVF_stats.search_time / 1000 / search_loops,
(hit / float(t_nq * GK / index_add_loops)));
(hit / float(t_nq * std::min(GK, t_k) / index_add_loops)));
}
}
printf("======================================================================================\n");
@ -479,8 +488,9 @@ test_with_nprobes(const std::string& ann_test_name, const std::string& index_key
}
void
test_ann_hdf5(const std::string& ann_test_name, const std::string& index_type, const QueryMode query_mode,
int32_t index_add_loops, const std::vector<size_t>& nprobes, int32_t search_loops) {
test_ann_hdf5(const std::string& ann_test_name, const std::string& cluster_type, const std::string& index_type,
const QueryMode query_mode, int32_t index_add_loops, const std::vector<size_t>& nprobes,
int32_t search_loops) {
double t0 = elapsed();
faiss::gpu::StandardGpuResources res;
@ -493,7 +503,7 @@ test_ann_hdf5(const std::string& ann_test_name, const std::string& index_type, c
return;
}
std::string index_key = "IVF16384," + index_type;
std::string index_key = cluster_type + "," + index_type;
if (!parse_ann_test_name(ann_test_name, dim, metric_type)) {
printf("Invalid ann test name: %s\n", ann_test_name.c_str());
@ -526,7 +536,7 @@ test_ann_hdf5(const std::string& ann_test_name, const std::string& index_type, c
*
* Dataset Dimensions Train_size Test_size Neighbors Distance Download
* Fashion-
* MNIST 784 60,000 10,000 100 Euclidean HDF5 (217MB)
MNIST 784 60,000 10,000 100 Euclidean HDF5 (217MB)
* GIST 960 1,000,000 1,000 100 Euclidean HDF5 (3.6GB)
* GloVe 100 1,183,514 10,000 100 Angular HDF5 (463MB)
* GloVe 200 1,183,514 10,000 100 Angular HDF5 (918MB)
@ -542,30 +552,39 @@ TEST(FAISSTEST, BENCHMARK) {
///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
const int32_t SIFT_INSERT_LOOPS = 2; // insert twice to get ~1G data set
test_ann_hdf5("sift-128-euclidean", "Flat", MODE_CPU, SIFT_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
test_ann_hdf5("sift-128-euclidean", "Flat", MODE_GPU, SIFT_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
test_ann_hdf5("sift-128-euclidean", "IDMap", "Flat", MODE_CPU, SIFT_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
test_ann_hdf5("sift-128-euclidean", "IDMap", "Flat", MODE_GPU, SIFT_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
test_ann_hdf5("sift-128-euclidean", "SQ8", MODE_CPU, SIFT_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
test_ann_hdf5("sift-128-euclidean", "SQ8", MODE_GPU, SIFT_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
test_ann_hdf5("sift-128-euclidean", "IVF16384", "Flat", MODE_CPU, SIFT_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
test_ann_hdf5("sift-128-euclidean", "IVF16384", "Flat", MODE_GPU, SIFT_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
test_ann_hdf5("sift-128-euclidean", "IVF16384", "SQ8", MODE_CPU, SIFT_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
test_ann_hdf5("sift-128-euclidean", "IVF16384", "SQ8", MODE_GPU, SIFT_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
#ifdef CUSTOMIZATION
test_ann_hdf5("sift-128-euclidean", "SQ8Hybrid", MODE_CPU, SIFT_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
test_ann_hdf5("sift-128-euclidean", "SQ8Hybrid", MODE_MIX, SIFT_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
test_ann_hdf5("sift-128-euclidean", "SQ8Hybrid", MODE_GPU, SIFT_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
test_ann_hdf5("sift-128-euclidean", "IVF16384", "SQ8Hybrid", MODE_CPU, SIFT_INSERT_LOOPS, param_nprobes,
SEARCH_LOOPS);
test_ann_hdf5("sift-128-euclidean", "IVF16384", "SQ8Hybrid", MODE_MIX, SIFT_INSERT_LOOPS, param_nprobes,
SEARCH_LOOPS);
test_ann_hdf5("sift-128-euclidean", "IVF16384", "SQ8Hybrid", MODE_GPU, SIFT_INSERT_LOOPS, param_nprobes,
SEARCH_LOOPS);
#endif
///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
const int32_t GLOVE_INSERT_LOOPS = 1;
test_ann_hdf5("glove-200-angular", "Flat", MODE_CPU, GLOVE_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
test_ann_hdf5("glove-200-angular", "Flat", MODE_GPU, GLOVE_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
test_ann_hdf5("glove-200-angular", "IVF16384", "Flat", MODE_CPU, GLOVE_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
test_ann_hdf5("glove-200-angular", "IVF16384", "Flat", MODE_GPU, GLOVE_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
test_ann_hdf5("glove-200-angular", "SQ8", MODE_CPU, GLOVE_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
test_ann_hdf5("glove-200-angular", "SQ8", MODE_GPU, GLOVE_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
test_ann_hdf5("glove-200-angular", "IVF16384", "SQ8", MODE_CPU, GLOVE_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
test_ann_hdf5("glove-200-angular", "IVF16384", "SQ8", MODE_GPU, GLOVE_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
#ifdef CUSTOMIZATION
test_ann_hdf5("glove-200-angular", "SQ8Hybrid", MODE_CPU, GLOVE_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
test_ann_hdf5("glove-200-angular", "SQ8Hybrid", MODE_MIX, GLOVE_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
test_ann_hdf5("glove-200-angular", "SQ8Hybrid", MODE_GPU, GLOVE_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
test_ann_hdf5("glove-200-angular", "IVF16384", "SQ8Hybrid", MODE_CPU, GLOVE_INSERT_LOOPS, param_nprobes,
SEARCH_LOOPS);
test_ann_hdf5("glove-200-angular", "IVF16384", "SQ8Hybrid", MODE_MIX, GLOVE_INSERT_LOOPS, param_nprobes,
SEARCH_LOOPS);
test_ann_hdf5("glove-200-angular", "IVF16384", "SQ8Hybrid", MODE_GPU, GLOVE_INSERT_LOOPS, param_nprobes,
SEARCH_LOOPS);
#endif
}

2
core/src/sdk/CMakeLists.txt
View File

@ -22,7 +22,7 @@ include_directories(include)
aux_source_directory(interface interface_files)
aux_source_directory(grpc grpc_client_files)
add_library(milvus_sdk STATIC
add_library(milvus_sdk SHARED
${interface_files}
${grpc_client_files}
${grpc_service_files}

103
core/src/sdk/README.md Normal file
View File

@ -0,0 +1,103 @@
### Build C++ SDK
The C++ SDK source code is under milvus/core/src/sdk. Build entire milvus project will also build the C++ SDK project.
If you don't want to build the entire milvus project, follow below steps:
```shell
# generate make files
$ cd [Milvus root path]/core
$ ./build.sh -l
# build C++ SDK project
$ cd [Milvus root path]/core/cmake_build
$ make -C src/sdk
```
### Try C++ example
Firstly, you need to start a Milvus server.
If you've already built the entire milvus project, just start Milvus server with the following command:
```shell
# start milvus server
$ cd [Milvus root path]/core
$ ./start_server.sh
```
You can also use Docker to start Milvus server:
```shell
# pull milvus docker image and start milvus server
$ docker pull milvusdb/milvus:latest
$ docker run --runtime=nvidia -p 19530:19530 -d milvusdb/milvus:latest
```
Run C++ example:
```shell
# run milvus C++ example
$ cd [Milvus root path]/core/cmake_build/src/sdk/examples/simple
$ ./sdk_simple
```
### Make your own C++ client project
Create a folder for the project, and copy C++ SDK header and library files into it.
```shell
# create project folder
$ mkdir MyMilvusClient
$ cd MyMilvusClient
# copy necessary files
$ cp [Milvus root path]/core/cmake_build/src/sdk/libmilvus_sdk.so .
$ cp [Milvus root path]/core/src/sdk/include/MilvusApi.h .
$ cp [Milvus root path]/core/src/sdk/include/Status.h .
```
Create file main.cpp in the project folder, and copy the following code into it:
```shell
#include "./MilvusApi.h"
#include "./Status.h"
int main() {
// connect to milvus server
std::shared_ptr<milvus::Connection> conn = milvus::Connection::Create();
milvus::ConnectParam param = {"127.0.0.1", "19530"};
conn->Connect(param);
// put your client code here
milvus::Connection::Destroy(conn);
return 0;
}
```
Create file CMakeList.txt in the project folder, and copy the following code into it:
```shell
cmake_minimum_required(VERSION 3.14)
project(test)
set(CMAKE_CXX_STANDARD 14)
add_executable(milvus_client main.cpp)
target_link_libraries(milvus_client
${PROJECT_SOURCE_DIR}/libmilvus_sdk.so)
```
Now there are 5 files in your project:
```shell
MyMilvusClient
|-CMakeList.txt
|-main.cpp
|-libmilvus_sdk.so
|-MilvusApi.h
|-Status.h
```
Build the project:
```shell
$ mkdir cmake_build
$ cd cmake_build
$ cmake ..
$ make
```
Run your client program:
```shell
$ ./milvus_client
```

35
core/src/sdk/examples/grpcsimple/src/ClientTest.cpp
View File

@ -57,22 +57,22 @@ PrintTableSchema(const milvus::TableSchema& tb_schema) {
void
PrintSearchResult(const std::vector<std::pair<int64_t, milvus::RowRecord>>& search_record_array,
const std::vector<milvus::TopKQueryResult>& topk_query_result_array) {
const milvus::TopKQueryResult& topk_query_result) {
BLOCK_SPLITER
std::cout << "Returned result count: " << topk_query_result_array.size() << std::endl;
size_t nq = topk_query_result.row_num;
size_t topk = topk_query_result.ids.size() / nq;
std::cout << "Returned result count: " << nq * topk << std::endl;
int32_t index = 0;
for (auto& result : topk_query_result_array) {
for (size_t i = 0; i < nq; i++) {
auto search_id = search_record_array[index].first;
index++;
std::cout << "No." << std::to_string(index) << " vector " << std::to_string(search_id) << " top "
<< std::to_string(result.query_result_arrays.size()) << " search result:" << std::endl;
for (auto& item : result.query_result_arrays) {
std::cout << "\t" << std::to_string(item.id) << "\tdistance:" << std::to_string(item.distance);
std::cout << std::endl;
std::cout << "No." << index << " vector " << search_id << " top " << topk << " search result:" << std::endl;
for (size_t j = 0; j < topk; j++) {
size_t idx = i * nq + j;
std::cout << "\t" << topk_query_result.ids[idx] << "\t" << topk_query_result.distances[idx] << std::endl;
}
}
BLOCK_SPLITER
}
@ -166,11 +166,13 @@ class TimeRecorder {
void
CheckResult(const std::vector<std::pair<int64_t, milvus::RowRecord>>& search_record_array,
const std::vector<milvus::TopKQueryResult>& topk_query_result_array) {
const milvus::TopKQueryResult& topk_query_result) {
BLOCK_SPLITER
size_t nq = topk_query_result.row_num;
size_t result_k = topk_query_result.ids.size() / nq;
int64_t index = 0;
for (auto& result : topk_query_result_array) {
auto result_id = result.query_result_arrays[0].id;
for (size_t i = 0; i < nq; i++) {
auto result_id = topk_query_result.ids[i * result_k];
auto search_id = search_record_array[index++].first;
if (result_id != search_id) {
std::cout << "The top 1 result is wrong: " << result_id << " vs. " << search_id << std::endl;
@ -196,19 +198,18 @@ DoSearch(std::shared_ptr<milvus::Connection> conn,
}
auto start = std::chrono::high_resolution_clock::now();
std::vector<milvus::TopKQueryResult> topk_query_result_array;
milvus::TopKQueryResult topk_query_result;
{
TimeRecorder rc(phase_name);
milvus::Status stat =
conn->Search(TABLE_NAME, record_array, query_range_array, TOP_K, 32, topk_query_result_array);
milvus::Status stat = conn->Search(TABLE_NAME, record_array, query_range_array, TOP_K, 32, topk_query_result);
std::cout << "SearchVector function call status: " << stat.message() << std::endl;
}
auto finish = std::chrono::high_resolution_clock::now();
std::cout << "SEARCHVECTOR COST: "
<< std::chrono::duration_cast<std::chrono::duration<double>>(finish - start).count() << "s\n";
PrintSearchResult(search_record_array, topk_query_result_array);
CheckResult(search_record_array, topk_query_result_array);
PrintSearchResult(search_record_array, topk_query_result);
CheckResult(search_record_array, topk_query_result);
}
} // namespace

40
core/src/sdk/grpc/ClientProxy.cpp
View File

@ -188,23 +188,19 @@ ClientProxy::Insert(const std::string& table_name, const std::vector<RowRecord>&
for (auto& record : record_array) {
::milvus::grpc::RowRecord* grpc_record = insert_param.add_row_record_array();
for (size_t i = 0; i < record.data.size(); i++) {
grpc_record->add_vector_data(record.data[i]);
}
grpc_record->add_vector_data(record.data.begin(), record.data.end());
}
// Single thread
::milvus::grpc::VectorIds vector_ids;
if (!id_array.empty()) {
for (auto i = 0; i < id_array.size(); i++) {
insert_param.add_row_id_array(id_array[i]);
}
/* set user's ids */
insert_param.add_row_id_array(id_array.begin(), id_array.end());
client_ptr_->Insert(vector_ids, insert_param, status);
} else {
client_ptr_->Insert(vector_ids, insert_param, status);
for (size_t i = 0; i < vector_ids.vector_id_array_size(); i++) {
id_array.push_back(vector_ids.vector_id_array(i));
}
/* return Milvus generated ids back to user */
id_array.insert(id_array.end(), vector_ids.vector_id_array().begin(), vector_ids.vector_id_array().end());
}
#endif
} catch (std::exception& ex) {
@ -217,7 +213,7 @@ ClientProxy::Insert(const std::string& table_name, const std::vector<RowRecord>&
Status
ClientProxy::Search(const std::string& table_name, const std::vector<RowRecord>& query_record_array,
const std::vector<Range>& query_range_array, int64_t topk, int64_t nprobe,
std::vector<TopKQueryResult>& topk_query_result_array) {
TopKQueryResult& topk_query_result) {
try {
// step 1: convert vectors data
::milvus::grpc::SearchParam search_param;
@ -226,9 +222,7 @@ ClientProxy::Search(const std::string& table_name, const std::vector<RowRecord>&
search_param.set_nprobe(nprobe);
for (auto& record : query_record_array) {
::milvus::grpc::RowRecord* row_record = search_param.add_query_record_array();
for (auto& rec : record.data) {
row_record->add_vector_data(rec);
}
row_record->add_vector_data(record.data.begin(), record.data.end());
}
// step 2: convert range array
@ -239,21 +233,17 @@ ClientProxy::Search(const std::string& table_name, const std::vector<RowRecord>&
}
// step 3: search vectors
::milvus::grpc::TopKQueryResultList topk_query_result_list;
Status status = client_ptr_->Search(topk_query_result_list, search_param);
::milvus::grpc::TopKQueryResult result;
Status status = client_ptr_->Search(result, search_param);
// step 4: convert result array
for (uint64_t i = 0; i < topk_query_result_list.topk_query_result_size(); ++i) {
TopKQueryResult result;
for (uint64_t j = 0; j < topk_query_result_list.topk_query_result(i).query_result_arrays_size(); ++j) {
QueryResult query_result;
query_result.id = topk_query_result_list.topk_query_result(i).query_result_arrays(j).id();
query_result.distance = topk_query_result_list.topk_query_result(i).query_result_arrays(j).distance();
result.query_result_arrays.emplace_back(query_result);
}
topk_query_result.row_num = result.row_num();
topk_query_result.ids.resize(result.ids().size());
memcpy(topk_query_result.ids.data(), result.ids().data(), result.ids().size() * sizeof(int64_t));
topk_query_result.distances.resize(result.distances().size());
memcpy(topk_query_result.distances.data(), result.distances().data(),
result.distances().size() * sizeof(float));
topk_query_result_array.emplace_back(result);
}
return status;
} catch (std::exception& ex) {
return Status(StatusCode::UnknownError, "fail to search vectors: " + std::string(ex.what()));

2
core/src/sdk/grpc/ClientProxy.h
View File

@ -60,7 +60,7 @@ class ClientProxy : public Connection {
Status
Search(const std::string& table_name, const std::vector<RowRecord>& query_record_array,
const std::vector<Range>& query_range_array, int64_t topk, int64_t nprobe,
std::vector<TopKQueryResult>& topk_query_result_array) override;
TopKQueryResult& topk_query_result) override;
Status
DescribeTable(const std::string& table_name, TableSchema& table_schema) override;

10
core/src/sdk/grpc/GrpcClient.cpp
View File

@ -134,20 +134,20 @@ GrpcClient::Insert(::milvus::grpc::VectorIds& vector_ids, const ::milvus::grpc::
}
Status
GrpcClient::Search(::milvus::grpc::TopKQueryResultList& topk_query_result_list,
GrpcClient::Search(::milvus::grpc::TopKQueryResult& topk_query_result,
const ::milvus::grpc::SearchParam& search_param) {
::milvus::grpc::TopKQueryResult query_result;
ClientContext context;
::grpc::Status grpc_status = stub_->Search(&context, search_param, &topk_query_result_list);
::grpc::Status grpc_status = stub_->Search(&context, search_param, &topk_query_result);
if (!grpc_status.ok()) {
std::cerr << "SearchVector rpc failed!" << std::endl;
std::cerr << grpc_status.error_message() << std::endl;
return Status(StatusCode::RPCFailed, grpc_status.error_message());
}
if (topk_query_result_list.status().error_code() != grpc::SUCCESS) {
std::cerr << topk_query_result_list.status().reason() << std::endl;
return Status(StatusCode::ServerFailed, topk_query_result_list.status().reason());
if (topk_query_result.status().error_code() != grpc::SUCCESS) {
std::cerr << topk_query_result.status().reason() << std::endl;
return Status(StatusCode::ServerFailed, topk_query_result.status().reason());
}
return Status::OK();

2
core/src/sdk/grpc/GrpcClient.h
View File

@ -57,7 +57,7 @@ class GrpcClient {
Insert(grpc::VectorIds& vector_ids, const grpc::InsertParam& insert_param, Status& status);
Status
Search(::milvus::grpc::TopKQueryResultList& topk_query_result_list, const grpc::SearchParam& search_param);
Search(::milvus::grpc::TopKQueryResult& topk_query_result, const grpc::SearchParam& search_param);
Status
DescribeTable(grpc::TableSchema& grpc_schema, const std::string& table_name);

14
core/src/sdk/include/MilvusApi.h
View File

@ -78,19 +78,13 @@ struct RowRecord {
std::vector<float> data; ///< Vector raw data
};
/**
* @brief Query result
*/
struct QueryResult {
int64_t id; ///< Output result
double distance; ///< Vector similarity distance
};
/**
* @brief TopK query result
*/
struct TopKQueryResult {
std::vector<QueryResult> query_result_arrays; ///< TopK query result
int64_t row_num;
std::vector<int64_t> ids;
std::vector<float> distances;
};
/**
@ -261,7 +255,7 @@ class Connection {
virtual Status
Search(const std::string& table_name, const std::vector<RowRecord>& query_record_array,
const std::vector<Range>& query_range_array, int64_t topk, int64_t nprobe,
std::vector<TopKQueryResult>& topk_query_result_array) = 0;
TopKQueryResult& topk_query_result) = 0;
/**
* @brief Show table description

5
core/src/sdk/interface/ConnectionImpl.cpp
View File

@ -91,9 +91,8 @@ ConnectionImpl::Insert(const std::string& table_name, const std::vector<RowRecor
Status
ConnectionImpl::Search(const std::string& table_name, const std::vector<RowRecord>& query_record_array,
const std::vector<Range>& query_range_array, int64_t topk, int64_t nprobe,
std::vector<TopKQueryResult>& topk_query_result_array) {
return client_proxy_->Search(table_name, query_record_array, query_range_array, topk, nprobe,
topk_query_result_array);
TopKQueryResult& topk_query_result) {
return client_proxy_->Search(table_name, query_record_array, query_range_array, topk, nprobe, topk_query_result);
}
Status

2
core/src/sdk/interface/ConnectionImpl.h
View File

@ -62,7 +62,7 @@ class ConnectionImpl : public Connection {
Status
Search(const std::string& table_name, const std::vector<RowRecord>& query_record_array,
const std::vector<Range>& query_range_array, int64_t topk, int64_t nprobe,
std::vector<TopKQueryResult>& topk_query_result_array) override;
TopKQueryResult& topk_query_result) override;
Status
DescribeTable(const std::string& table_name, TableSchema& table_schema) override;

68
core/src/server/Config.cpp
View File

@ -25,6 +25,7 @@
#include "config/YamlConfigMgr.h"
#include "server/Config.h"
#include "utils/CommonUtil.h"
#include "utils/StringHelpFunctions.h"
#include "utils/ValidationUtil.h"
namespace milvus {
@ -341,6 +342,11 @@ Config::ResetDefaultConfig() {
return s;
}
s = SetResourceConfigSearchResources(CONFIG_RESOURCE_SEARCH_RESOURCES_DEFAULT);
if (!s.ok()) {
return s;
}
s = SetResourceConfigIndexBuildDevice(CONFIG_RESOURCE_INDEX_BUILD_DEVICE_DEFAULT);
if (!s.ok()) {
return s;
@ -788,6 +794,22 @@ Config::GetConfigStr(const std::string& parent_key, const std::string& child_key
return value;
}
std::string
Config::GetConfigSequenceStr(const std::string& parent_key, const std::string& child_key, const std::string& delim,
const std::string& default_value) {
std::string value;
if (!GetConfigValueInMem(parent_key, child_key, value).ok()) {
std::vector<std::string> sequence = GetConfigNode(parent_key).GetSequence(child_key);
if (sequence.empty()) {
value = default_value;
} else {
server::StringHelpFunctions::MergeStringWithDelimeter(sequence, delim, value);
}
SetConfigValueInMem(parent_key, child_key, value);
}
return value;
}
Status
Config::GetServerConfigAddress(std::string& value) {
value = GetConfigStr(CONFIG_SERVER, CONFIG_SERVER_ADDRESS, CONFIG_SERVER_ADDRESS_DEFAULT);
@ -1011,8 +1033,10 @@ Config::GetResourceConfigMode(std::string& value) {
Status
Config::GetResourceConfigSearchResources(std::vector<std::string>& value) {
ConfigNode resource_config = GetConfigNode(CONFIG_RESOURCE);
value = resource_config.GetSequence(CONFIG_RESOURCE_SEARCH_RESOURCES);
std::string str =
GetConfigSequenceStr(CONFIG_RESOURCE, CONFIG_RESOURCE_SEARCH_RESOURCES,
CONFIG_RESOURCE_SEARCH_RESOURCES_DELIMITER, CONFIG_RESOURCE_SEARCH_RESOURCES_DEFAULT);
server::StringHelpFunctions::SplitStringByDelimeter(str, CONFIG_RESOURCE_SEARCH_RESOURCES_DELIMITER, value);
return CheckResourceConfigSearchResources(value);
}
@ -1155,7 +1179,7 @@ Config::SetMetricConfigEnableMonitor(const std::string& value) {
return s;
}
SetConfigValueInMem(CONFIG_DB, CONFIG_METRIC_ENABLE_MONITOR, value);
SetConfigValueInMem(CONFIG_METRIC, CONFIG_METRIC_ENABLE_MONITOR, value);
return Status::OK();
}
@ -1166,7 +1190,7 @@ Config::SetMetricConfigCollector(const std::string& value) {
return s;
}
SetConfigValueInMem(CONFIG_DB, CONFIG_METRIC_COLLECTOR, value);
SetConfigValueInMem(CONFIG_METRIC, CONFIG_METRIC_COLLECTOR, value);
return Status::OK();
}
@ -1177,7 +1201,7 @@ Config::SetMetricConfigPrometheusPort(const std::string& value) {
return s;
}
SetConfigValueInMem(CONFIG_DB, CONFIG_METRIC_PROMETHEUS_PORT, value);
SetConfigValueInMem(CONFIG_METRIC, CONFIG_METRIC_PROMETHEUS_PORT, value);
return Status::OK();
}
@ -1189,7 +1213,7 @@ Config::SetCacheConfigCpuCacheCapacity(const std::string& value) {
return s;
}
SetConfigValueInMem(CONFIG_DB, CONFIG_CACHE_CPU_CACHE_CAPACITY, value);
SetConfigValueInMem(CONFIG_CACHE, CONFIG_CACHE_CPU_CACHE_CAPACITY, value);
return Status::OK();
}
@ -1200,7 +1224,7 @@ Config::SetCacheConfigCpuCacheThreshold(const std::string& value) {
return s;
}
SetConfigValueInMem(CONFIG_DB, CONFIG_CACHE_CPU_CACHE_THRESHOLD, value);
SetConfigValueInMem(CONFIG_CACHE, CONFIG_CACHE_CPU_CACHE_THRESHOLD, value);
return Status::OK();
}
@ -1211,7 +1235,7 @@ Config::SetCacheConfigGpuCacheCapacity(const std::string& value) {
return s;
}
SetConfigValueInMem(CONFIG_DB, CONFIG_CACHE_GPU_CACHE_CAPACITY, value);
SetConfigValueInMem(CONFIG_CACHE, CONFIG_CACHE_GPU_CACHE_CAPACITY, value);
return Status::OK();
}
@ -1222,7 +1246,7 @@ Config::SetCacheConfigGpuCacheThreshold(const std::string& value) {
return s;
}
SetConfigValueInMem(CONFIG_DB, CONFIG_CACHE_GPU_CACHE_THRESHOLD, value);
SetConfigValueInMem(CONFIG_CACHE, CONFIG_CACHE_GPU_CACHE_THRESHOLD, value);
return Status::OK();
}
@ -1233,7 +1257,7 @@ Config::SetCacheConfigCacheInsertData(const std::string& value) {
return s;
}
SetConfigValueInMem(CONFIG_DB, CONFIG_CACHE_CACHE_INSERT_DATA, value);
SetConfigValueInMem(CONFIG_CACHE, CONFIG_CACHE_CACHE_INSERT_DATA, value);
return Status::OK();
}
@ -1245,7 +1269,7 @@ Config::SetEngineConfigUseBlasThreshold(const std::string& value) {
return s;
}
SetConfigValueInMem(CONFIG_DB, CONFIG_ENGINE_USE_BLAS_THRESHOLD, value);
SetConfigValueInMem(CONFIG_ENGINE, CONFIG_ENGINE_USE_BLAS_THRESHOLD, value);
return Status::OK();
}
@ -1256,7 +1280,7 @@ Config::SetEngineConfigOmpThreadNum(const std::string& value) {
return s;
}
SetConfigValueInMem(CONFIG_DB, CONFIG_ENGINE_OMP_THREAD_NUM, value);
SetConfigValueInMem(CONFIG_ENGINE, CONFIG_ENGINE_OMP_THREAD_NUM, value);
return Status::OK();
}
@ -1267,7 +1291,7 @@ Config::SetEngineConfigGpuSearchThreshold(const std::string& value) {
return s;
}
SetConfigValueInMem(CONFIG_DB, CONFIG_ENGINE_GPU_SEARCH_THRESHOLD, value);
SetConfigValueInMem(CONFIG_ENGINE, CONFIG_ENGINE_GPU_SEARCH_THRESHOLD, value);
return Status::OK();
}
@ -1279,7 +1303,21 @@ Config::SetResourceConfigMode(const std::string& value) {
return s;
}
SetConfigValueInMem(CONFIG_DB, CONFIG_RESOURCE_MODE, value);
SetConfigValueInMem(CONFIG_RESOURCE, CONFIG_RESOURCE_MODE, value);
return Status::OK();
}
Status
Config::SetResourceConfigSearchResources(const std::string& value) {
std::vector<std::string> res_vec;
server::StringHelpFunctions::SplitStringByDelimeter(value, CONFIG_RESOURCE_SEARCH_RESOURCES_DELIMITER, res_vec);
Status s = CheckResourceConfigSearchResources(res_vec);
if (!s.ok()) {
return s;
}
SetConfigValueInMem(CONFIG_RESOURCE, CONFIG_RESOURCE_SEARCH_RESOURCES, value);
return Status::OK();
}
@ -1290,7 +1328,7 @@ Config::SetResourceConfigIndexBuildDevice(const std::string& value) {
return s;
}
SetConfigValueInMem(CONFIG_DB, CONFIG_RESOURCE_INDEX_BUILD_DEVICE, value);
SetConfigValueInMem(CONFIG_RESOURCE, CONFIG_RESOURCE_INDEX_BUILD_DEVICE, value);
return Status::OK();
}

7
core/src/server/Config.h
View File

@ -92,6 +92,8 @@ static const char* CONFIG_RESOURCE = "resource_config";
static const char* CONFIG_RESOURCE_MODE = "mode";
static const char* CONFIG_RESOURCE_MODE_DEFAULT = "simple";
static const char* CONFIG_RESOURCE_SEARCH_RESOURCES = "search_resources";
static const char* CONFIG_RESOURCE_SEARCH_RESOURCES_DELIMITER = ",";
static const char* CONFIG_RESOURCE_SEARCH_RESOURCES_DEFAULT = "cpu,gpu0";
static const char* CONFIG_RESOURCE_INDEX_BUILD_DEVICE = "index_build_device";
static const char* CONFIG_RESOURCE_INDEX_BUILD_DEVICE_DEFAULT = "gpu0";
@ -183,6 +185,9 @@ class Config {
std::string
GetConfigStr(const std::string& parent_key, const std::string& child_key, const std::string& default_value = "");
std::string
GetConfigSequenceStr(const std::string& parent_key, const std::string& child_key, const std::string& delim = ",",
const std::string& default_value = "");
public:
/* server config */
@ -304,6 +309,8 @@ class Config {
Status
SetResourceConfigMode(const std::string& value);
Status
SetResourceConfigSearchResources(const std::string& value);
Status
SetResourceConfigIndexBuildDevice(const std::string& value);
private:

4
core/src/server/grpc_impl/GrpcRequestHandler.cpp
View File

@ -75,7 +75,7 @@ GrpcRequestHandler::Insert(::grpc::ServerContext* context, const ::milvus::grpc:
::grpc::Status
GrpcRequestHandler::Search(::grpc::ServerContext* context, const ::milvus::grpc::SearchParam* request,
::milvus::grpc::TopKQueryResultList* response) {
::milvus::grpc::TopKQueryResult* response) {
std::vector<std::string> file_id_array;
BaseTaskPtr task_ptr = SearchTask::Create(request, file_id_array, response);
::milvus::grpc::Status grpc_status;
@ -87,7 +87,7 @@ GrpcRequestHandler::Search(::grpc::ServerContext* context, const ::milvus::grpc:
::grpc::Status
GrpcRequestHandler::SearchInFiles(::grpc::ServerContext* context, const ::milvus::grpc::SearchInFilesParam* request,
::milvus::grpc::TopKQueryResultList* response) {
::milvus::grpc::TopKQueryResult* response) {
std::vector<std::string> file_id_array;
for (int i = 0; i < request->file_id_array_size(); i++) {
file_id_array.push_back(request->file_id_array(i));

4
core/src/server/grpc_impl/GrpcRequestHandler.h
View File

@ -145,7 +145,7 @@ class GrpcRequestHandler final : public ::milvus::grpc::MilvusService::Service {
*/
::grpc::Status
Search(::grpc::ServerContext* context, const ::milvus::grpc::SearchParam* request,
::milvus::grpc::TopKQueryResultList* response) override;
::milvus::grpc::TopKQueryResult* response) override;
/**
* @brief Internal use query interface
@ -169,7 +169,7 @@ class GrpcRequestHandler final : public ::milvus::grpc::MilvusService::Service {
*/
::grpc::Status
SearchInFiles(::grpc::ServerContext* context, const ::milvus::grpc::SearchInFilesParam* request,
::milvus::grpc::TopKQueryResultList* response) override;
::milvus::grpc::TopKQueryResult* response) override;
/**
* @brief Get table schema

21
core/src/server/grpc_impl/GrpcRequestTask.cpp
View File

@ -20,6 +20,7 @@
#include <string.h>
#include <map>
#include <string>
#include <utility>
#include <vector>
//#include <gperftools/profiler.h>
@ -541,16 +542,16 @@ InsertTask::OnExecute() {
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
SearchTask::SearchTask(const ::milvus::grpc::SearchParam* search_vector_infos,
const std::vector<std::string>& file_id_array, ::milvus::grpc::TopKQueryResultList* response)
const std::vector<std::string>& file_id_array, ::milvus::grpc::TopKQueryResult* response)
: GrpcBaseTask(DQL_TASK_GROUP),
search_param_(search_vector_infos),
file_id_array_(file_id_array),
topk_result_list(response) {
topk_result_(response) {
}
BaseTaskPtr
SearchTask::Create(const ::milvus::grpc::SearchParam* search_vector_infos,
const std::vector<std::string>& file_id_array, ::milvus::grpc::TopKQueryResultList* response) {
const std::vector<std::string>& file_id_array, ::milvus::grpc::TopKQueryResult* response) {
if (search_vector_infos == nullptr) {
SERVER_LOG_ERROR << "grpc input is null!";
return nullptr;
@ -668,18 +669,10 @@ SearchTask::OnExecute() {
return Status::OK(); // empty table
}
size_t result_k = result_ids.size() / record_count;
// step 7: construct result array
for (size_t i = 0; i < record_count; i++) {
::milvus::grpc::TopKQueryResult* topk_query_result = topk_result_list->add_topk_query_result();
for (size_t j = 0; j < result_k; j++) {
::milvus::grpc::QueryResult* grpc_result = topk_query_result->add_query_result_arrays();
size_t idx = i * result_k + j;
grpc_result->set_id(result_ids[idx]);
grpc_result->set_distance(result_distances[idx]);
}
}
topk_result_->set_row_num(record_count);
topk_result_->add_ids(result_ids.begin(), result_ids.end());
topk_result_->add_distances(result_distances.begin(), result_distances.end());
// step 8: print time cost percent
rc.RecordSection("construct result and send");

6
core/src/server/grpc_impl/GrpcRequestTask.h
View File

@ -153,11 +153,11 @@ class SearchTask : public GrpcBaseTask {
public:
static BaseTaskPtr
Create(const ::milvus::grpc::SearchParam* search_param, const std::vector<std::string>& file_id_array,
::milvus::grpc::TopKQueryResultList* response);
::milvus::grpc::TopKQueryResult* response);
protected:
SearchTask(const ::milvus::grpc::SearchParam* search_param, const std::vector<std::string>& file_id_array,
::milvus::grpc::TopKQueryResultList* response);
::milvus::grpc::TopKQueryResult* response);
Status
OnExecute() override;
@ -165,7 +165,7 @@ class SearchTask : public GrpcBaseTask {
private:
const ::milvus::grpc::SearchParam* search_param_;
std::vector<std::string> file_id_array_;
::milvus::grpc::TopKQueryResultList* topk_result_list;
::milvus::grpc::TopKQueryResult* topk_result_;
};
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

2
core/src/server/grpc_impl/GrpcServer.cpp
View File

@ -93,7 +93,7 @@ GrpcServer::StartService() {
builder.SetCompressionAlgorithmSupportStatus(GRPC_COMPRESS_STREAM_GZIP, true);
builder.SetDefaultCompressionAlgorithm(GRPC_COMPRESS_STREAM_GZIP);
builder.SetDefaultCompressionLevel(GRPC_COMPRESS_LEVEL_HIGH);
builder.SetDefaultCompressionLevel(GRPC_COMPRESS_LEVEL_NONE);
GrpcRequestHandler service;

44
core/src/utils/StringHelpFunctions.cpp
View File

@ -39,39 +39,53 @@ StringHelpFunctions::TrimStringQuote(std::string& string, const std::string& qou
}
}
Status
void
StringHelpFunctions::SplitStringByDelimeter(const std::string& str, const std::string& delimeter,
std::vector<std::string>& result) {
if (str.empty()) {
return Status::OK();
return;
}
size_t last = 0;
size_t index = str.find_first_of(delimeter, last);
while (index != std::string::npos) {
result.emplace_back(str.substr(last, index - last));
last = index + 1;
index = str.find_first_of(delimeter, last);
size_t prev = 0, pos = 0;
while (true) {
pos = str.find_first_of(delimeter, prev);
if (pos == std::string::npos) {
result.emplace_back(str.substr(prev));
break;
} else {
result.emplace_back(str.substr(prev, pos - prev));
prev = pos + 1;
}
}
if (index - last > 0) {
std::string temp = str.substr(last);
result.emplace_back(temp);
}
void
StringHelpFunctions::MergeStringWithDelimeter(const std::vector<std::string>& strs, const std::string& delimeter,
std::string& result) {
if (strs.empty()) {
result = "";
return;
}
return Status::OK();
result = strs[0];
for (size_t i = 1; i < strs.size(); i++) {
result = result + delimeter + strs[i];
}
}
Status
StringHelpFunctions::SplitStringByQuote(const std::string& str, const std::string& delimeter, const std::string& quote,
std::vector<std::string>& result) {
if (quote.empty()) {
return SplitStringByDelimeter(str, delimeter, result);
SplitStringByDelimeter(str, delimeter, result);
return Status::OK();
}
size_t last = 0;
size_t index = str.find_first_of(quote, last);
if (index == std::string::npos) {
return SplitStringByDelimeter(str, delimeter, result);
SplitStringByDelimeter(str, delimeter, result);
return Status::OK();
}
std::string process_str = str;
@ -116,7 +130,7 @@ StringHelpFunctions::SplitStringByQuote(const std::string& str, const std::strin
}
if (!process_str.empty()) {
return SplitStringByDelimeter(process_str, delimeter, result);
SplitStringByDelimeter(process_str, delimeter, result);
}
return Status::OK();

5
core/src/utils/StringHelpFunctions.h
View File

@ -43,9 +43,12 @@ class StringHelpFunctions {
// ,b, | b |
// ,, | |
// a a
static Status
static void
SplitStringByDelimeter(const std::string& str, const std::string& delimeter, std::vector<std::string>& result);
static void
MergeStringWithDelimeter(const std::vector<std::string>& strs, const std::string& delimeter, std::string& result);
// assume the table has two columns, quote='\"', delimeter=','
// a,b a | b
// "aa,gg,yy",b aa,gg,yy | b

1
core/unittest/CMakeLists.txt
View File

@ -80,6 +80,7 @@ set(helper_files
${MILVUS_ENGINE_SRC}/utils/CommonUtil.cpp
${MILVUS_ENGINE_SRC}/utils/TimeRecorder.cpp
${MILVUS_ENGINE_SRC}/utils/Status.cpp
${MILVUS_ENGINE_SRC}/utils/StringHelpFunctions.cpp
${MILVUS_ENGINE_SRC}/utils/ValidationUtil.cpp
${MILVUS_ENGINE_SRC}/external/easyloggingpp/easylogging++.cc
)

12
core/unittest/db/CMakeLists.txt
View File

@ -17,8 +17,16 @@
# under the License.
#-------------------------------------------------------------------------------
aux_source_directory(${CMAKE_CURRENT_SOURCE_DIR} test_files)
set(test_files
${CMAKE_CURRENT_SOURCE_DIR}/test_db.cpp
${CMAKE_CURRENT_SOURCE_DIR}/test_db_mysql.cpp
${CMAKE_CURRENT_SOURCE_DIR}/test_engine.cpp
${CMAKE_CURRENT_SOURCE_DIR}/test_mem.cpp
${CMAKE_CURRENT_SOURCE_DIR}/test_meta.cpp
${CMAKE_CURRENT_SOURCE_DIR}/test_meta_mysql.cpp
${CMAKE_CURRENT_SOURCE_DIR}/test_misc.cpp
${CMAKE_CURRENT_SOURCE_DIR}/test_search.cpp
${CMAKE_CURRENT_SOURCE_DIR}/utils.cpp)
cuda_add_executable(test_db
${common_files}

4
core/unittest/db/utils.cpp
View File

@ -60,8 +60,8 @@ static const char
" port: 8080 # port prometheus used to fetch metrics\n"
"\n"
"cache_config:\n"
" cpu_mem_capacity: 16 # GB, CPU memory used for cache\n"
" cpu_mem_threshold: 0.85 # percentage of data kept when cache cleanup triggered\n"
" cpu_cache_capacity: 16 # GB, CPU memory used for cache\n"
" cpu_cache_threshold: 0.85 # percentage of data kept when cache cleanup triggered\n"
" cache_insert_data: false # whether load inserted data into cache\n"
"\n"
"engine_config:\n"

6
core/unittest/metrics/CMakeLists.txt
View File

@ -17,7 +17,11 @@
# under the License.
#-------------------------------------------------------------------------------
aux_source_directory(${CMAKE_CURRENT_SOURCE_DIR} test_files)
set(test_files
${CMAKE_CURRENT_SOURCE_DIR}/test_metricbase.cpp
${CMAKE_CURRENT_SOURCE_DIR}/test_metrics.cpp
${CMAKE_CURRENT_SOURCE_DIR}/test_prometheus.cpp
${CMAKE_CURRENT_SOURCE_DIR}/utils.cpp)
add_executable(test_metrics
${common_files}

12
core/unittest/scheduler/CMakeLists.txt
View File

@ -17,7 +17,17 @@
# under the License.
#-------------------------------------------------------------------------------
aux_source_directory(${CMAKE_CURRENT_SOURCE_DIR} test_files)
set(test_files
${CMAKE_CURRENT_SOURCE_DIR}/test_algorithm.cpp
${CMAKE_CURRENT_SOURCE_DIR}/test_event.cpp
${CMAKE_CURRENT_SOURCE_DIR}/test_node.cpp
${CMAKE_CURRENT_SOURCE_DIR}/test_normal.cpp
${CMAKE_CURRENT_SOURCE_DIR}/test_resource.cpp
${CMAKE_CURRENT_SOURCE_DIR}/test_resource_factory.cpp
${CMAKE_CURRENT_SOURCE_DIR}/test_resource_mgr.cpp
${CMAKE_CURRENT_SOURCE_DIR}/test_scheduler.cpp
${CMAKE_CURRENT_SOURCE_DIR}/test_task.cpp
${CMAKE_CURRENT_SOURCE_DIR}/test_tasktable.cpp)
cuda_add_executable(test_scheduler
${common_files}

0
core/unittest/scheduler/task_test.cpp → core/unittest/scheduler/test_task.cpp
View File

7
core/unittest/server/CMakeLists.txt
View File

@ -17,7 +17,12 @@
# under the License.
#-------------------------------------------------------------------------------
aux_source_directory(${CMAKE_CURRENT_SOURCE_DIR} test_files)
set(test_files
${CMAKE_CURRENT_SOURCE_DIR}/test_cache.cpp
${CMAKE_CURRENT_SOURCE_DIR}/test_config.cpp
${CMAKE_CURRENT_SOURCE_DIR}/test_rpc.cpp
${CMAKE_CURRENT_SOURCE_DIR}/test_util.cpp
${CMAKE_CURRENT_SOURCE_DIR}/utils.cpp)
include_directories("${CUDA_TOOLKIT_ROOT_DIR}/include")
link_directories("${CUDA_TOOLKIT_ROOT_DIR}/lib64")

215
core/unittest/server/test_config.cpp
View File

@ -19,10 +19,11 @@
#include <gtest/gtest-death-test.h>
#include "config/YamlConfigMgr.h"
#include "utils/CommonUtil.h"
#include "utils/ValidationUtil.h"
#include "server/Config.h"
#include "server/utils.h"
#include "utils/CommonUtil.h"
#include "utils/ValidationUtil.h"
#include "utils/StringHelpFunctions.h"
namespace {
@ -98,19 +99,191 @@ TEST_F(ConfigTest, CONFIG_TEST) {
ASSERT_TRUE(seqs.empty());
}
TEST_F(ConfigTest, SERVER_CONFIG_TEST) {
TEST_F(ConfigTest, SERVER_CONFIG_VALID_TEST) {
std::string config_path(CONFIG_PATH);
milvus::server::Config& config = milvus::server::Config::GetInstance();
milvus::Status s = config.LoadConfigFile(config_path + VALID_CONFIG_FILE);
ASSERT_TRUE(s.ok());
milvus::server::Config &config = milvus::server::Config::GetInstance();
milvus::Status s;
std::string str_val;
int32_t int32_val;
int64_t int64_val;
float float_val;
bool bool_val;
s = config.ValidateConfig();
/* server config */
std::string server_addr = "192.168.1.155";
s = config.SetServerConfigAddress(server_addr);
ASSERT_TRUE(s.ok());
config.PrintAll();
s = config.ResetDefaultConfig();
s = config.GetServerConfigAddress(str_val);
ASSERT_TRUE(s.ok());
ASSERT_TRUE(str_val == server_addr);
std::string server_port = "12345";
s = config.SetServerConfigPort(server_port);
ASSERT_TRUE(s.ok());
s = config.GetServerConfigPort(str_val);
ASSERT_TRUE(s.ok());
ASSERT_TRUE(str_val == server_port);
std::string server_mode = "cluster_readonly";
s = config.SetServerConfigDeployMode(server_mode);
ASSERT_TRUE(s.ok());
s = config.GetServerConfigDeployMode(str_val);
ASSERT_TRUE(s.ok());
ASSERT_TRUE(str_val == server_mode);
std::string server_time_zone = "UTC+6";
s = config.SetServerConfigTimeZone(server_time_zone);
ASSERT_TRUE(s.ok());
s = config.GetServerConfigTimeZone(str_val);
ASSERT_TRUE(s.ok());
ASSERT_TRUE(str_val == server_time_zone);
/* db config */
std::string db_primary_path = "/home/zilliz";
s = config.SetDBConfigPrimaryPath(db_primary_path);
ASSERT_TRUE(s.ok());
s = config.GetDBConfigPrimaryPath(str_val);
ASSERT_TRUE(s.ok());
ASSERT_TRUE(str_val == db_primary_path);
std::string db_secondary_path = "/home/zilliz";
s = config.SetDBConfigSecondaryPath(db_secondary_path);
ASSERT_TRUE(s.ok());
s = config.GetDBConfigSecondaryPath(str_val);
ASSERT_TRUE(s.ok());
ASSERT_TRUE(str_val == db_secondary_path);
std::string db_backend_url = "mysql://root:123456@127.0.0.1:19530/milvus";
s = config.SetDBConfigBackendUrl(db_backend_url);
ASSERT_TRUE(s.ok());
s = config.GetDBConfigBackendUrl(str_val);
ASSERT_TRUE(s.ok());
ASSERT_TRUE(str_val == db_backend_url);
int32_t db_archive_disk_threshold = 100;
s = config.SetDBConfigArchiveDiskThreshold(std::to_string(db_archive_disk_threshold));
ASSERT_TRUE(s.ok());
s = config.GetDBConfigArchiveDiskThreshold(int32_val);
ASSERT_TRUE(s.ok());
ASSERT_TRUE(int32_val == db_archive_disk_threshold);
int32_t db_archive_days_threshold = 365;
s = config.SetDBConfigArchiveDaysThreshold(std::to_string(db_archive_days_threshold));
ASSERT_TRUE(s.ok());
s = config.GetDBConfigArchiveDaysThreshold(int32_val);
ASSERT_TRUE(s.ok());
ASSERT_TRUE(int32_val == db_archive_days_threshold);
int32_t db_insert_buffer_size = 2;
s = config.SetDBConfigInsertBufferSize(std::to_string(db_insert_buffer_size));
ASSERT_TRUE(s.ok());
s = config.GetDBConfigInsertBufferSize(int32_val);
ASSERT_TRUE(s.ok());
ASSERT_TRUE(int32_val == db_insert_buffer_size);
/* metric config */
bool metric_enable_monitor = false;
s = config.SetMetricConfigEnableMonitor(std::to_string(metric_enable_monitor));
ASSERT_TRUE(s.ok());
s = config.GetMetricConfigEnableMonitor(bool_val);
ASSERT_TRUE(s.ok());
ASSERT_TRUE(bool_val == metric_enable_monitor);
std::string metric_collector = "prometheus";
s = config.SetMetricConfigCollector(metric_collector);
ASSERT_TRUE(s.ok());
s = config.GetMetricConfigCollector(str_val);
ASSERT_TRUE(str_val == metric_collector);
std::string metric_prometheus_port = "2222";
s = config.SetMetricConfigPrometheusPort(metric_prometheus_port);
ASSERT_TRUE(s.ok());
s = config.GetMetricConfigPrometheusPort(str_val);
ASSERT_TRUE(s.ok());
ASSERT_TRUE(str_val == metric_prometheus_port);
/* cache config */
int64_t cache_cpu_cache_capacity = 5;
s = config.SetCacheConfigCpuCacheCapacity(std::to_string(cache_cpu_cache_capacity));
ASSERT_TRUE(s.ok());
s = config.GetCacheConfigCpuCacheCapacity(int64_val);
ASSERT_TRUE(s.ok());
ASSERT_TRUE(int64_val == cache_cpu_cache_capacity);
float cache_cpu_cache_threshold = 0.1;
s = config.SetCacheConfigCpuCacheThreshold(std::to_string(cache_cpu_cache_threshold));
ASSERT_TRUE(s.ok());
s = config.GetCacheConfigCpuCacheThreshold(float_val);
ASSERT_TRUE(float_val == cache_cpu_cache_threshold);
int64_t cache_gpu_cache_capacity = 1;
s = config.SetCacheConfigGpuCacheCapacity(std::to_string(cache_gpu_cache_capacity));
ASSERT_TRUE(s.ok());
s = config.GetCacheConfigGpuCacheCapacity(int64_val);
ASSERT_TRUE(s.ok());
ASSERT_TRUE(int64_val == cache_gpu_cache_capacity);
float cache_gpu_cache_threshold = 0.2;
s = config.SetCacheConfigGpuCacheThreshold(std::to_string(cache_gpu_cache_threshold));
ASSERT_TRUE(s.ok());
s = config.GetCacheConfigGpuCacheThreshold(float_val);
ASSERT_TRUE(float_val == cache_gpu_cache_threshold);
bool cache_insert_data = true;
s = config.SetCacheConfigCacheInsertData(std::to_string(cache_insert_data));
ASSERT_TRUE(s.ok());
s = config.GetCacheConfigCacheInsertData(bool_val);
ASSERT_TRUE(bool_val == cache_insert_data);
/* engine config */
int32_t engine_use_blas_threshold = 50;
s = config.SetEngineConfigUseBlasThreshold(std::to_string(engine_use_blas_threshold));
ASSERT_TRUE(s.ok());
s = config.GetEngineConfigUseBlasThreshold(int32_val);
ASSERT_TRUE(s.ok());
ASSERT_TRUE(int32_val == engine_use_blas_threshold);
int32_t engine_omp_thread_num = 8;
s = config.SetEngineConfigOmpThreadNum(std::to_string(engine_omp_thread_num));
ASSERT_TRUE(s.ok());
s = config.GetEngineConfigOmpThreadNum(int32_val);
ASSERT_TRUE(s.ok());
ASSERT_TRUE(int32_val == engine_omp_thread_num);
int32_t engine_gpu_search_threshold = 800;
s = config.SetEngineConfigGpuSearchThreshold(std::to_string(engine_gpu_search_threshold));
ASSERT_TRUE(s.ok());
s = config.GetEngineConfigGpuSearchThreshold(int32_val);
ASSERT_TRUE(s.ok());
ASSERT_TRUE(int32_val == engine_gpu_search_threshold);
/* resource config */
std::string resource_mode = "simple";
s = config.SetResourceConfigMode(resource_mode);
ASSERT_TRUE(s.ok());
s = config.GetResourceConfigMode(str_val);
ASSERT_TRUE(s.ok());
ASSERT_TRUE(str_val == resource_mode);
std::vector<std::string> search_resources = {"cpu", "gpu0"};
std::vector<std::string> res_vec;
std::string res_str;
milvus::server::StringHelpFunctions::MergeStringWithDelimeter(search_resources,
milvus::server::CONFIG_RESOURCE_SEARCH_RESOURCES_DELIMITER, res_str);
s = config.SetResourceConfigSearchResources(res_str);
ASSERT_TRUE(s.ok());
s = config.GetResourceConfigSearchResources(res_vec);
ASSERT_TRUE(s.ok());
for (size_t i = 0; i < search_resources.size(); i++) {
ASSERT_TRUE(search_resources[i] == res_vec[i]);
}
int32_t resource_index_build_device = 0;
s = config.SetResourceConfigIndexBuildDevice("gpu" + std::to_string(resource_index_build_device));
ASSERT_TRUE(s.ok());
s = config.GetResourceConfigIndexBuildDevice(int32_val);
ASSERT_TRUE(s.ok());
ASSERT_TRUE(int32_val == resource_index_build_device);
}
TEST_F(ConfigTest, SERVER_CONFIG_INVALID_TEST) {
@ -226,9 +399,29 @@ TEST_F(ConfigTest, SERVER_CONFIG_INVALID_TEST) {
s = config.SetResourceConfigMode("default");
ASSERT_FALSE(s.ok());
s = config.SetResourceConfigSearchResources("gpu10");
ASSERT_FALSE(s.ok());
s = config.SetResourceConfigSearchResources("cpu");
ASSERT_FALSE(s.ok());
s = config.SetResourceConfigIndexBuildDevice("gup2");
ASSERT_FALSE(s.ok());
s = config.SetResourceConfigIndexBuildDevice("gpu16");
ASSERT_FALSE(s.ok());
}
TEST_F(ConfigTest, SERVER_CONFIG_TEST) {
std::string config_path(CONFIG_PATH);
milvus::server::Config& config = milvus::server::Config::GetInstance();
milvus::Status s = config.LoadConfigFile(config_path + VALID_CONFIG_FILE);
ASSERT_TRUE(s.ok());
s = config.ValidateConfig();
ASSERT_TRUE(s.ok());
config.PrintAll();
s = config.ResetDefaultConfig();
ASSERT_TRUE(s.ok());
}

23
core/unittest/server/test_rpc.cpp
View File

@ -202,9 +202,7 @@ TEST_F(RpcHandlerTest, INSERT_TEST) {
::milvus::grpc::VectorIds vector_ids;
for (auto& record : record_array) {
::milvus::grpc::RowRecord* grpc_record = request.add_row_record_array();
for (size_t i = 0; i < record.size(); i++) {
grpc_record->add_vector_data(record[i]);
}
grpc_record->add_vector_data(record.begin(), record.end());
}
handler->Insert(&context, &request, &vector_ids);
ASSERT_EQ(vector_ids.vector_id_array_size(), VECTOR_COUNT);
@ -213,7 +211,7 @@ TEST_F(RpcHandlerTest, INSERT_TEST) {
TEST_F(RpcHandlerTest, SEARCH_TEST) {
::grpc::ServerContext context;
::milvus::grpc::SearchParam request;
::milvus::grpc::TopKQueryResultList response;
::milvus::grpc::TopKQueryResult response;
//test null input
handler->Search(&context, nullptr, &response);
@ -241,22 +239,17 @@ TEST_F(RpcHandlerTest, SEARCH_TEST) {
::milvus::grpc::InsertParam insert_param;
for (auto& record : record_array) {
::milvus::grpc::RowRecord* grpc_record = insert_param.add_row_record_array();
for (size_t i = 0; i < record.size(); i++) {
grpc_record->add_vector_data(record[i]);
}
grpc_record->add_vector_data(record.begin(), record.end());
}
//insert vectors
insert_param.set_table_name(TABLE_NAME);
::milvus::grpc::VectorIds vector_ids;
handler->Insert(&context, &insert_param, &vector_ids);
sleep(7);
BuildVectors(0, 10, record_array);
for (auto& record : record_array) {
::milvus::grpc::RowRecord* row_record = request.add_query_record_array();
for (auto& rec : record) {
row_record->add_vector_data(rec);
}
row_record->add_vector_data(record.begin(), record.end());
}
handler->Search(&context, &request, &response);
@ -325,9 +318,7 @@ TEST_F(RpcHandlerTest, TABLES_TEST) {
for (auto& record : record_array) {
::milvus::grpc::RowRecord* grpc_record = request.add_row_record_array();
for (size_t i = 0; i < record.size(); i++) {
grpc_record->add_vector_data(record[i]);
}
grpc_record->add_vector_data(record.begin(), record.end());
}
//test vector_id size not equal to row record size
vector_ids.clear_vector_id_array();
@ -342,9 +333,7 @@ TEST_F(RpcHandlerTest, TABLES_TEST) {
vector_ids.clear_vector_id_array();
for (uint64_t i = 0; i < 10; ++i) {
::milvus::grpc::RowRecord* grpc_record = request.add_row_record_array();
for (size_t j = 0; j < 10; j++) {
grpc_record->add_vector_data(record_array[i][j]);
}
grpc_record->add_vector_data(record_array[i].begin(), record_array[i].end());
}
handler->Insert(&context, &request, &vector_ids);

5
core/unittest/server/test_util.cpp
View File

@ -117,12 +117,11 @@ TEST(UtilTest, STRINGFUNCTIONS_TEST) {
str = "a,b,c";
std::vector<std::string> result;
auto status = milvus::server::StringHelpFunctions::SplitStringByDelimeter(str, ",", result);
ASSERT_TRUE(status.ok());
milvus::server::StringHelpFunctions::SplitStringByDelimeter(str, ",", result);
ASSERT_EQ(result.size(), 3UL);
result.clear();
status = milvus::server::StringHelpFunctions::SplitStringByQuote(str, ",", "\"", result);
auto status = milvus::server::StringHelpFunctions::SplitStringByQuote(str, ",", "\"", result);
ASSERT_TRUE(status.ok());
ASSERT_EQ(result.size(), 3UL);

6
core/unittest/wrapper/CMakeLists.txt
View File

@ -17,7 +17,11 @@
# under the License.
#-------------------------------------------------------------------------------
aux_source_directory(${CMAKE_CURRENT_SOURCE_DIR} test_files)
set(test_files
${CMAKE_CURRENT_SOURCE_DIR}/test_hybrid_index.cpp
${CMAKE_CURRENT_SOURCE_DIR}/test_knowhere.cpp
${CMAKE_CURRENT_SOURCE_DIR}/test_wrapper.cpp
${CMAKE_CURRENT_SOURCE_DIR}/utils.cpp)
set(wrapper_files
${MILVUS_ENGINE_SRC}/wrapper/DataTransfer.cpp

4
core/unittest/wrapper/utils.cpp
View File

@ -50,8 +50,8 @@ static const char
" port: 8080 # port prometheus used to fetch metrics\n"
"\n"
"cache_config:\n"
" cpu_mem_capacity: 16 # GB, CPU memory used for cache\n"
" cpu_mem_threshold: 0.85 # percentage of data kept when cache cleanup triggered\n"
" cpu_cache_capacity: 16 # GB, CPU memory used for cache\n"
" cpu_cache_threshold: 0.85 # percentage of data kept when cache cleanup triggered\n"
" cache_insert_data: false # whether load inserted data into cache\n"
"\n"
"engine_config:\n"

4
tests/milvus-java-test/pom.xml
View File

@ -99,7 +99,7 @@
<dependency>
<groupId>io.milvus</groupId>
<artifactId>milvus-sdk-java</artifactId>
<version>0.2.0-SNAPSHOT</version>
<version>0.2.2</version>
</dependency>
<!-- <dependency>-->
@ -134,4 +134,4 @@
</dependencies>
</project>
</project>

3
tests/milvus_python_test/test_connect.py
View File

@ -273,7 +273,8 @@ class TestConnect:
milvus = Milvus()
uri_value = "tcp://%s:%s" % (args["ip"], args["port"])
with pytest.raises(Exception) as e:
milvus.connect(host=args["ip"], port=19540, uri=uri_value, timeout=1)
res = milvus.connect(host=args["ip"], port=39540, uri=uri_value, timeout=1)
logger.getLogger().info(res)
assert not milvus.connected()
def _test_add_vector_and_disconnect_concurrently(self):

3
tests/milvus_python_test/test_mix.py
View File

@ -26,6 +26,7 @@ index_params = {'index_type': IndexType.IVFLAT, 'nlist': 16384}
class TestMixBase:
# TODO: enable
@pytest.mark.level(2)
def test_search_during_createIndex(self, args):
loops = 100000
table = gen_unique_str()
@ -180,4 +181,4 @@ def check_result(result, id):
if len(result) >= 5:
return id in [result[0].id, result[1].id, result[2].id, result[3].id, result[4].id]
else:
return id in (i.id for i in result)
return id in (i.id for i in result)

2
tests/milvus_python_test/test_ping.py
View File

@ -1,7 +1,7 @@
import logging
import pytest
__version__ = '0.5.2'
__version__ = '0.5.3'
class TestPing: