Merge remote-tracking branch 'upstream/master'

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,21 @@
+**What type of PR is this?**
+
+api-change / bug / design / documentation / feature
+
+
+**What this PR does / why we need it:**
+
+
+**Which issue(s) this PR fixes:**
+
+Fixes #
+
+
+**Special notes for your reviewer:**
+
+
+**Does this PR introduce a user-facing change?:**
+
+
+**Additional documentation (e.g. design docs, usage docs, etc.):**
+

CHANGELOG.md

@@ -18,6 +18,7 @@ Please mark all change in change log and use the issue from GitHub
 - \#766 - If partition tag is similar, wrong partition is searched
 - \#771 - Add server build commit info interface
 - \#759 - Put C++ sdk out of milvus/core
+- \#813 - Add push mode for prometheus monitor
 - \#815 - Support MinIO storage
 - \#910 - Change Milvus c++ standard to c++17
 
@@ -29,6 +30,8 @@ Please mark all change in change log and use the issue from GitHub
 - \#848 - Add ready-to-use config files to the Milvus repo for enhanced user experince
 - \#860 - Remove redundant checks in CacheMgr's constructor
 - \#908 - Move "primary_path" and "secondary_path" to storage config
+- \#931 - Remove "collector" from config
+- \#966 - Update NOTICE.md
 
 ## Task
 

CODEOWNERS

@@ -0,0 +1,10 @@
+# Each line is a component followed by one or more owners. 
+
+* @JinHai-CN
+/core   @JinHai-CN @XuPeng-SH
+/ci     @ZhifengZhang-CN
+/docker @ZhifengZhang-CN
+/docs   @jielinxu @yamasite
+/sdk    @fishpenguin
+/shards @XuPeng-SH
+/tests  @del-zhenwu

CONTRIBUTING.md

@@ -26,7 +26,7 @@ Contributions to Milvus fall into the following categories.
 
 ### Contributing code
 
-If you have improvements to Milvus, send us your pull requests! For those just getting started, see [GitHub workflow](#github-workflow).
+If you have improvements to Milvus, send us your pull requests! For those just getting started, see [GitHub workflow](#github-workflow). Make sure to refer to the related issue in your pull request's comment and update CHANGELOG.md.
 
 The Milvus team members will review your pull requests, and once it is accepted, the status of the projects to which it is associated will be changed to **Reviewer approved**. This means we are working on submitting your pull request to the internal repository. After the change has been submitted internally, your pull request will be merged automatically on GitHub.
 

INSTALL.md

@@ -13,6 +13,7 @@ If you encounter any problems/issues compiling Milvus from source, please refer
 
   If your operating system is not Ubuntu 18.04 or higher, we recommend you to pull a [docker image of Ubuntu 18.04](https://docs.docker.com/install/linux/docker-ce/ubuntu/) as your compilation environment.
   
+- GCC 7.0 or higher to support C++17
 - CMake 3.12 or higher
 
 ##### For GPU-enabled version, you will also need:

NOTICE.md

@@ -23,4 +23,7 @@
 | grpc          | [Apache 2.0](https://github.com/grpc/grpc/blob/master/LICENSE)                                                   |
 | EASYLOGGINGPP | [MIT](https://github.com/zuhd-org/easyloggingpp/blob/master/LICENSE)                                                          |
 | Json          | [MIT](https://github.com/nlohmann/json/blob/develop/LICENSE.MIT)                                                          |
+| opentracing-cpp | [Apache License 2.0](https://github.com/opentracing/opentracing-cpp/blob/master/LICENSE)                                     |
+| libfiu        | [BOLA](https://github.com/albertito/libfiu/blob/master/LICENSE)                                                          |
+| aws-sdk-cpp   | [Apache License 2.0](https://github.com/cydrain/aws-sdk-cpp/blob/master/LICENSE)                                              |
 

README.md

@@ -8,6 +8,7 @@
 ![Release_date](https://img.shields.io/badge/release%20date-December-yellowgreen)
 [![codecov](https://codecov.io/gh/milvus-io/milvus/branch/master/graph/badge.svg)](https://codecov.io/gh/milvus-io/milvus)
 [![All Contributors](https://img.shields.io/badge/all_contributors-6-orange.svg?style=flat-square)](#contributors-)
+[![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/3563/badge)](https://bestpractices.coreinfrastructure.org/projects/3563)
 
 [中文版](README_CN.md) | [日本語版](README_JP.md)
 
@@ -23,7 +24,7 @@ Keep up-to-date with newest releases and latest updates by reading Milvus [relea
 
 ## Get started
 
-See the [Milvus install guide](https://www.milvus.io/docs/guides/get_started/install_milvus/install_milvus.md) for using Docker containers. To install Milvus from source code, see [build from source](install.md).
+See the [Milvus install guide](https://www.milvus.io/docs/guides/get_started/install_milvus/install_milvus.md) for using Docker containers. To install Milvus from source code, see [build from source](INSTALL.md).
 
 To edit Milvus settings, read [Milvus configuration](https://www.milvus.io/docs/v0.6.0/reference/milvus_config.md).
 

README_CN.md

@@ -23,7 +23,7 @@ Milvus 提供稳定的 [Python](https://github.com/milvus-io/pymilvus)、[Java](
 
 ## 开始使用 Milvus
 
-请参阅 [Milvus 安装指南](https://www.milvus.io/cn/docs/guides/get_started/install_milvus/install_milvus.md) 使用 Docker 容器安装 Milvus。若要基于源码编译，请访问 [源码安装](install.md)。
+请参阅 [Milvus 安装指南](https://www.milvus.io/cn/docs/guides/get_started/install_milvus/install_milvus.md) 使用 Docker 容器安装 Milvus。若要基于源码编译，请访问 [源码安装](INSTALL.md)。
 
 若要更改 Milvus 设置，请参阅 [Milvus 配置](https://www.milvus.io/cn/docs/reference/milvus_config.md)。
 

README_JP.md

@@ -23,7 +23,7 @@ Milvus [リリースノート](https://www.milvus.io/docs/v0.6.0/releases/v0.6.0
 
 ## はじめに
 
-DockerでMilvusをインストールすることは簡単です。[Milvusインストール案内](https://www.milvus.io/docs/guides/get_started/install_milvus/install_milvus.md) を参考してください。ソースからMilvusを構築するために、[ソースから構築する](install.md)を参考してください。
+DockerでMilvusをインストールすることは簡単です。[Milvusインストール案内](https://www.milvus.io/docs/guides/get_started/install_milvus/install_milvus.md) を参考してください。ソースからMilvusを構築するために、[ソースから構築する](INSTALL.md)を参考してください。
 
 Milvusをコンフィグするために、[Milvusコンフィグ](https://www.milvus.io/docs/reference/milvus_config.md)を読んでください。
 

RELEASE.md

@@ -0,0 +1,17 @@
+# Milvus Release Methodology and Criterias
+
+## Release methodology
+
+Milvus releases are packages that have been approved for general public release, with varying degrees of caveat regarding their perceived quality or potential for change. 
+They are stable releases intended for everyday usage by developers and non-developers.
+
+Project versioning follows the specification of [Semantic Versioning 2.0.0](https://semver.org/).
+
+## Release criteria
+
+- Milvus core test code coverage must be at least 90%.
+- Reported bugs should not have any critical issues.
+- All bugs, new features, enhancements must be tested.
+- All documents need to be reviewed with no broken link.
+- Pressure testing, stability testing, accuracy testing and performance testing results should be evaluated.
+

SECURITY.md

@@ -0,0 +1,15 @@
+# Security Policy
+
+## Supported versions
+
+The following versions of Milvus are currently being supported with security updates.
+
+| Version | Supported          |
+| ------- | ------------------ |
+| 0.6.0   | ✔️                |
+| <= 0.5.3  | :x:              |
+
+## Reporting a vulnerability
+
+To report a security vulnerability, please reach out to the Milvus team via <hai.jin@zilliz.com>.
+

SUPPORT.md

@@ -0,0 +1,14 @@
+# Support for deploying and using Milvus
+
+We use GitHub for tracking bugs and feature requests. If you need any support for using Milvus, please refer to the following resources below.
+
+## Documentation
+- [User Documentation](https://www.milvus.io/docs/guides/get_started/install_milvus/install_milvus.md)
+- [Troubleshooting Guide](https://www.milvus.io/docs/v0.6.0/guides/troubleshoot.md)
+- [FAQ](https://www.milvus.io/docs/v0.6.0/faq/operational_faq.md)
+
+## Real-time chat
+[Slack](https://join.slack.com/t/milvusio/shared_invite/enQtNzY1OTQ0NDI3NjMzLWNmYmM1NmNjOTQ5MGI5NDhhYmRhMGU5M2NhNzhhMDMzY2MzNDdlYjM5ODQ5MmE3ODFlYzU3YjJkNmVlNDQ2ZTk): The #general channel is the place where people offer support.
+
+## Other
+[Bootcamp](https://github.com/milvus-io/bootcamp): It provides more scenario-based applications and demos of Milvus.

ci/jenkins/step/build.groovy

@@ -4,9 +4,9 @@ timeout(time: 75, unit: 'MINUTES') {
             def checkResult = sh(script: "./check_ccache.sh -l ${params.JFROG_ARTFACTORY_URL}/ccache", returnStatus: true)
 
             if ("${BINARY_VERSION}" == "gpu") {
-                sh "/bin/bash --login -c \". ./before-install.sh && ./build.sh -t ${params.BUILD_TYPE} -o ${env.MILVUS_INSTALL_PREFIX} -l -g -u -c\""
+                sh "/bin/bash --login -c \". ./before-install.sh && ./build.sh -t ${params.BUILD_TYPE} -o ${env.MILVUS_INSTALL_PREFIX} -l -g -u -c -i\""
             } else {
-                sh "/bin/bash --login -c \". ./before-install.sh && ./build.sh -t ${params.BUILD_TYPE} -o ${env.MILVUS_INSTALL_PREFIX} -l -u -c\""
+                sh "/bin/bash --login -c \". ./before-install.sh && ./build.sh -t ${params.BUILD_TYPE} -o ${env.MILVUS_INSTALL_PREFIX} -l -u -c -i\""
             }
             sh "./update_ccache.sh -l ${params.JFROG_ARTFACTORY_URL}/ccache -u ${USERNAME} -p ${PASSWORD}"
         }

ci/scripts/build.sh

@@ -21,9 +21,10 @@ BUILD_COVERAGE="OFF"
 RUN_CPPLINT="OFF"
 GPU_VERSION="OFF"
 WITH_MKL="OFF"
+FIU_ENABLE="OFF"
 CUDA_COMPILER=/usr/local/cuda/bin/nvcc
 
-while getopts "o:t:b:f:pgulcmh" arg
+while getopts "o:t:b:f:pgulcmih" arg
 do
         case $arg in
              o)
@@ -57,6 +58,9 @@ do
              m)
                 WITH_MKL="ON"
                 ;;
+             i)
+                FIU_ENABLE="ON"
+                ;;
              h) # help
                 echo "
 
@@ -71,10 +75,11 @@ parameter:
 -l: run cpplint, clang-format and clang-tidy(default: OFF)
 -c: code coverage(default: OFF)
 -m: build with MKL(default: OFF)
+-i: build FIU_ENABLE(default: OFF)
 -h: help
 
 usage:
-./build.sh -o \${INSTALL_PREFIX} -t \${BUILD_TYPE} -b \${CORE_BUILD_DIR} -f \${FAISS_ROOT} [-p] [-g] [-u] [-l] [-c] [-m] [-h]
+./build.sh -o \${INSTALL_PREFIX} -t \${BUILD_TYPE} -b \${CORE_BUILD_DIR} -f \${FAISS_ROOT} [-p] [-g] [-u] [-l] [-c] [-m] [-i] [-h]
                 "
                 exit 0
                 ;;
@@ -105,6 +110,7 @@ CMAKE_CMD="cmake \
 -DFAISS_WITH_MKL=${WITH_MKL} \
 -DArrow_SOURCE=AUTO \
 -DFAISS_SOURCE=AUTO \
+-DMILVUS_WITH_FIU=${FIU_ENABLE} \
 ${MILVUS_CORE_DIR}"
 echo ${CMAKE_CMD}
 ${CMAKE_CMD}

core/conf/demo/server_config.yaml

@@ -20,7 +20,7 @@ version: 0.1
 #----------------------+------------------------------------------------------------+------------+-----------------+
 # Server Config        | Description                                                | Type       | Default         |
 #----------------------+------------------------------------------------------------+------------+-----------------+
-# address              | IP address that Milvus server monitors.                    | String     | 0.0.0.0         |
+# address              | IP address that Milvus server monitors.                    | IP         | 0.0.0.0         |
 #----------------------+------------------------------------------------------------+------------+-----------------+
 # port                 | Port that Milvus server monitors. Port range (1024, 65535) | Integer    | 19530           |
 #----------------------+------------------------------------------------------------+------------+-----------------+
@@ -68,7 +68,7 @@ db_config:
 #----------------------+------------------------------------------------------------+------------+-----------------+
 # minio_enable         | Enable MinIO storage or not.                               | Boolean    | false           |
 #----------------------+------------------------------------------------------------+------------+-----------------+
-# minio_address        | MinIO storage service IP address.                          | String     | 127.0.0.1       |
+# minio_address        | MinIO storage service IP address.                          | IP         | 127.0.0.1       |
 #----------------------+------------------------------------------------------------+------------+-----------------+
 # minio_port           | MinIO storage service port. Port range (1024, 65535)       | Integer    | 9000            |
 #----------------------+------------------------------------------------------------+------------+-----------------+
@@ -93,20 +93,19 @@ storage_config:
 #----------------------+------------------------------------------------------------+------------+-----------------+
 # enable_monitor       | Enable monitoring function or not.                         | Boolean    | false           |
 #----------------------+------------------------------------------------------------+------------+-----------------+
-# collector            | Connected monitoring system to collect metrics.            | String     | Prometheus      |
+# address              | Pushgateway address                                        | IP         | 127.0.0.1       +
 #----------------------+------------------------------------------------------------+------------+-----------------+
-# port                 | Port to visit Prometheus, port range (1024, 65535)         | Integer    | 8080            |
+# port                 | Pushgateway port, port range (1024, 65535)                 | Integer    | 9091            |
 #----------------------+------------------------------------------------------------+------------+-----------------+
 metric_config:
   enable_monitor: false
-  collector: prometheus
-  prometheus_config:
-    port: 8080
+  address: 127.0.0.1
+  port: 9091
 
 #----------------------+------------------------------------------------------------+------------+-----------------+
 # Cache Config         | Description                                                | Type       | Default         |
 #----------------------+------------------------------------------------------------+------------+-----------------+
-# cpu_cache_capacity   | The size of CPU memory used for caching data for faster    | Integet    | 4 (GB)          |
+# cpu_cache_capacity   | The size of CPU memory used for caching data for faster    | Integer    | 4 (GB)          |
 #                      | query. The sum of 'cpu_cache_capacity' and                 |            |                 |
 #                      | 'insert_buffer_size' must be less than system memory size. |            |                 |
 #----------------------+------------------------------------------------------------+------------+-----------------+

core/conf/server_cpu_config.template

@@ -20,7 +20,7 @@ version: 0.1
 #----------------------+------------------------------------------------------------+------------+-----------------+
 # Server Config        | Description                                                | Type       | Default         |
 #----------------------+------------------------------------------------------------+------------+-----------------+
-# address              | IP address that Milvus server monitors.                    | String     | 0.0.0.0         |
+# address              | IP address that Milvus server monitors.                    | IP         | 0.0.0.0         |
 #----------------------+------------------------------------------------------------+------------+-----------------+
 # port                 | Port that Milvus server monitors. Port range (1024, 65535) | Integer    | 19530           |
 #----------------------+------------------------------------------------------------+------------+-----------------+
@@ -68,7 +68,7 @@ db_config:
 #----------------------+------------------------------------------------------------+------------+-----------------+
 # minio_enable         | Enable MinIO storage or not.                               | Boolean    | false           |
 #----------------------+------------------------------------------------------------+------------+-----------------+
-# minio_address        | MinIO storage service IP address.                          | String     | 127.0.0.1       |
+# minio_address        | MinIO storage service IP address.                          | IP         | 127.0.0.1       |
 #----------------------+------------------------------------------------------------+------------+-----------------+
 # minio_port           | MinIO storage service port. Port range (1024, 65535)       | Integer    | 9000            |
 #----------------------+------------------------------------------------------------+------------+-----------------+
@@ -93,20 +93,19 @@ storage_config:
 #----------------------+------------------------------------------------------------+------------+-----------------+
 # enable_monitor       | Enable monitoring function or not.                         | Boolean    | false           |
 #----------------------+------------------------------------------------------------+------------+-----------------+
-# collector            | Connected monitoring system to collect metrics.            | String     | Prometheus      |
+# address              | Pushgateway address                                        | IP         | 127.0.0.1       +
 #----------------------+------------------------------------------------------------+------------+-----------------+
-# port                 | Port to visit Prometheus, port range (1024, 65535)         | Integer    | 8080            |
+# port                 | Pushgateway port, port range (1024, 65535)                 | Integer    | 9091            |
 #----------------------+------------------------------------------------------------+------------+-----------------+
 metric_config:
   enable_monitor: false
-  collector: prometheus
-  prometheus_config:
-    port: 8080
+  address: 127.0.0.1
+  port: 9091
 
 #----------------------+------------------------------------------------------------+------------+-----------------+
 # Cache Config         | Description                                                | Type       | Default         |
 #----------------------+------------------------------------------------------------+------------+-----------------+
-# cpu_cache_capacity   | The size of CPU memory used for caching data for faster    | Integet    | 4 (GB)          |
+# cpu_cache_capacity   | The size of CPU memory used for caching data for faster    | Integer    | 4 (GB)          |
 #                      | query. The sum of 'cpu_cache_capacity' and                 |            |                 |
 #                      | 'insert_buffer_size' must be less than system memory size. |            |                 |
 #----------------------+------------------------------------------------------------+------------+-----------------+

core/conf/server_gpu_config.template

@@ -20,7 +20,7 @@ version: 0.1
 #----------------------+------------------------------------------------------------+------------+-----------------+
 # Server Config        | Description                                                | Type       | Default         |
 #----------------------+------------------------------------------------------------+------------+-----------------+
-# address              | IP address that Milvus server monitors.                    | String     | 0.0.0.0         |
+# address              | IP address that Milvus server monitors.                    | IP         | 0.0.0.0         |
 #----------------------+------------------------------------------------------------+------------+-----------------+
 # port                 | Port that Milvus server monitors. Port range (1024, 65535) | Integer    | 19530           |
 #----------------------+------------------------------------------------------------+------------+-----------------+
@@ -68,7 +68,7 @@ db_config:
 #----------------------+------------------------------------------------------------+------------+-----------------+
 # minio_enable         | Enable MinIO storage or not.                               | Boolean    | false           |
 #----------------------+------------------------------------------------------------+------------+-----------------+
-# minio_address        | MinIO storage service IP address.                          | String     | 127.0.0.1       |
+# minio_address        | MinIO storage service IP address.                          | IP         | 127.0.0.1       |
 #----------------------+------------------------------------------------------------+------------+-----------------+
 # minio_port           | MinIO storage service port. Port range (1024, 65535)       | Integer    | 9000            |
 #----------------------+------------------------------------------------------------+------------+-----------------+
@@ -93,20 +93,19 @@ storage_config:
 #----------------------+------------------------------------------------------------+------------+-----------------+
 # enable_monitor       | Enable monitoring function or not.                         | Boolean    | false           |
 #----------------------+------------------------------------------------------------+------------+-----------------+
-# collector            | Connected monitoring system to collect metrics.            | String     | Prometheus      |
+# address              | Pushgateway address                                        | IP         | 127.0.0.1       +
 #----------------------+------------------------------------------------------------+------------+-----------------+
-# port                 | Port to visit Prometheus, port range (1024, 65535)         | Integer    | 8080            |
+# port                 | Pushgateway port, port range (1024, 65535)                 | Integer    | 9091            |
 #----------------------+------------------------------------------------------------+------------+-----------------+
 metric_config:
   enable_monitor: false
-  collector: prometheus
-  prometheus_config:
-    port: 8080
+  address: 127.0.0.1
+  port: 9091
 
 #----------------------+------------------------------------------------------------+------------+-----------------+
 # Cache Config         | Description                                                | Type       | Default         |
 #----------------------+------------------------------------------------------------+------------+-----------------+
-# cpu_cache_capacity   | The size of CPU memory used for caching data for faster    | Integet    | 4 (GB)          |
+# cpu_cache_capacity   | The size of CPU memory used for caching data for faster    | Integer    | 4 (GB)          |
 #                      | query. The sum of 'cpu_cache_capacity' and                 |            |                 |
 #                      | 'insert_buffer_size' must be less than system memory size. |            |                 |
 #----------------------+------------------------------------------------------------+------------+-----------------+

core/src/CMakeLists.txt

@@ -137,6 +137,7 @@ set(prometheus_lib
         prometheus-cpp-push
         prometheus-cpp-pull
         prometheus-cpp-core
+        curl
         )
 
 set(boost_lib

core/src/db/DBImpl.cpp

@@ -631,6 +631,7 @@ DBImpl::StartMetricTask() {
     server::Metrics::GetInstance().CPUCoreUsagePercentSet();
     server::Metrics::GetInstance().GPUTemperature();
     server::Metrics::GetInstance().CPUTemperature();
+    server::Metrics::GetInstance().PushToGateway();
 
     // ENGINE_LOG_TRACE << "Metric task finished";
 }

core/src/db/Utils.cpp

@@ -126,7 +126,7 @@ DeleteTablePath(const DBMetaOptions& options, const std::string& table_id, bool
     if (minio_enable) {
         std::string table_path = options.path_ + TABLES_FOLDER + table_id;
 
-        auto storage_inst = milvus::storage::S3ClientWrapper::GetInstance();
+        auto& storage_inst = milvus::storage::S3ClientWrapper::GetInstance();
         Status stat = storage_inst.DeleteObjects(table_path);
         if (!stat.ok()) {
             return stat;

core/src/metrics/MetricBase.h

@@ -18,7 +18,7 @@
 #pragma once
 
 #include "SystemInfo.h"
-#include "utils/Error.h"
+#include "utils/Status.h"
 
 #include <string>
 
@@ -32,8 +32,9 @@ class MetricsBase {
         return instance;
     }
 
-    virtual ErrorCode
+    virtual Status
     Init() {
+        return Status::OK();
     }
 
     virtual void
@@ -203,6 +204,10 @@ class MetricsBase {
     virtual void
     CPUTemperature() {
     }
+
+    virtual void
+    PushToGateway() {
+    }
 };
 
 }  // namespace server

core/src/metrics/Metrics.cpp

@@ -34,17 +34,8 @@ Metrics::GetInstance() {
 
 MetricsBase&
 Metrics::CreateMetricsCollector() {
-    Config& config = Config::GetInstance();
-    std::string collector_type_str;
-
-    config.GetMetricConfigCollector(collector_type_str);
-
 #ifdef MILVUS_WITH_PROMETHEUS
-    if (collector_type_str == "prometheus") {
-        return PrometheusMetrics::GetInstance();
-    } else {
-        return MetricsBase::GetInstance();
-    }
+    return PrometheusMetrics::GetInstance();
 #else
     return MetricsBase::GetInstance();
 #endif

core/src/metrics/prometheus/PrometheusMetrics.cpp

@@ -27,39 +27,39 @@
 namespace milvus {
 namespace server {
 
-ErrorCode
+Status
 PrometheusMetrics::Init() {
     try {
         Config& config = Config::GetInstance();
-        Status s = config.GetMetricConfigEnableMonitor(startup_);
-        if (!s.ok()) {
-            return s.code();
-        }
+        CONFIG_CHECK(config.GetMetricConfigEnableMonitor(startup_));
         if (!startup_) {
-            return SERVER_SUCCESS;
+            return Status::OK();
         }
 
         // Following should be read from config file.
-        std::string bind_address;
-        s = config.GetMetricConfigPrometheusPort(bind_address);
-        if (!s.ok()) {
-            return s.code();
-        }
+        std::string push_port, push_address;
+        CONFIG_CHECK(config.GetMetricConfigPort(push_port));
+        CONFIG_CHECK(config.GetMetricConfigAddress(push_address));
 
         const std::string uri = std::string("/metrics");
-        const std::size_t num_threads = 2;
+        // const std::size_t num_threads = 2;
+
+        auto labels = prometheus::Gateway::GetInstanceLabel("pushgateway");
+
+        // Init pushgateway
+        gateway_ = std::make_shared<prometheus::Gateway>(push_address, push_port, "milvus_metrics", labels);
 
         // Init Exposer
-        exposer_ptr_ = std::make_shared<prometheus::Exposer>(bind_address, uri, num_threads);
+        // exposer_ptr_ = std::make_shared<prometheus::Exposer>(bind_address, uri, num_threads);
 
-        // Exposer Registry
-        exposer_ptr_->RegisterCollectable(registry_);
+        // Pushgateway Registry
+        gateway_->RegisterCollectable(registry_);
     } catch (std::exception& ex) {
         SERVER_LOG_ERROR << "Failed to connect prometheus server: " << std::string(ex.what());
-        return SERVER_UNEXPECTED_ERROR;
+        return Status(SERVER_UNEXPECTED_ERROR, ex.what());
     }
 
-    return SERVER_SUCCESS;
+    return Status::OK();
 }
 
 void

core/src/metrics/prometheus/PrometheusMetrics.h

@@ -18,6 +18,7 @@
 #pragma once
 
 #include <prometheus/exposer.h>
+#include <prometheus/gateway.h>
 #include <prometheus/registry.h>
 #include <iostream>
 #include <memory>
@@ -25,7 +26,8 @@
 #include <vector>
 
 #include "metrics/MetricBase.h"
-#include "utils/Error.h"
+#include "utils/Log.h"
+#include "utils/Status.h"
 
 #define METRICS_NOW_TIME std::chrono::system_clock::now()
 //#define server::Metrics::GetInstance() server::GetInstance()
@@ -42,11 +44,11 @@ class PrometheusMetrics : public MetricsBase {
         return instance;
     }
 
-    ErrorCode
-    Init();
+    Status
+    Init() override;
 
  private:
-    std::shared_ptr<prometheus::Exposer> exposer_ptr_;
+    std::shared_ptr<prometheus::Gateway> gateway_;
     std::shared_ptr<prometheus::Registry> registry_ = std::make_shared<prometheus::Registry>();
     bool startup_ = false;
 
@@ -293,9 +295,18 @@ class PrometheusMetrics : public MetricsBase {
     void
     CPUTemperature() override;
 
-    std::shared_ptr<prometheus::Exposer>&
-    exposer_ptr() {
-        return exposer_ptr_;
+    void
+    PushToGateway() override {
+        if (startup_) {
+            if (gateway_->Push() != 200) {
+                ENGINE_LOG_WARNING << "Metrics pushgateway failed";
+            }
+        }
+    }
+
+    std::shared_ptr<prometheus::Gateway>&
+    gateway() {
+        return gateway_;
     }
 
     //    prometheus::Exposer& exposer() { return exposer_;}

core/src/server/Config.cpp

@@ -131,11 +131,11 @@ Config::ValidateConfig() {
     bool metric_enable_monitor;
     CONFIG_CHECK(GetMetricConfigEnableMonitor(metric_enable_monitor));
 
-    std::string metric_collector;
-    CONFIG_CHECK(GetMetricConfigCollector(metric_collector));
+    std::string metric_address;
+    CONFIG_CHECK(GetMetricConfigAddress(metric_address));
 
-    std::string metric_prometheus_port;
-    CONFIG_CHECK(GetMetricConfigPrometheusPort(metric_prometheus_port));
+    std::string metric_port;
+    CONFIG_CHECK(GetMetricConfigPort(metric_port));
 
     /* cache config */
     int64_t cache_cpu_cache_capacity;
@@ -213,8 +213,8 @@ Config::ResetDefaultConfig() {
 
     /* metric config */
     CONFIG_CHECK(SetMetricConfigEnableMonitor(CONFIG_METRIC_ENABLE_MONITOR_DEFAULT));
-    CONFIG_CHECK(SetMetricConfigCollector(CONFIG_METRIC_COLLECTOR_DEFAULT));
-    CONFIG_CHECK(SetMetricConfigPrometheusPort(CONFIG_METRIC_PROMETHEUS_PORT_DEFAULT));
+    CONFIG_CHECK(SetMetricConfigAddress(CONFIG_METRIC_ADDRESS_DEFAULT));
+    CONFIG_CHECK(SetMetricConfigPort(CONFIG_METRIC_PORT_DEFAULT));
 
     /* cache config */
     CONFIG_CHECK(SetCacheConfigCpuCacheCapacity(CONFIG_CACHE_CPU_CACHE_CAPACITY_DEFAULT));
@@ -547,26 +547,24 @@ Config::CheckMetricConfigEnableMonitor(const std::string& value) {
 }
 
 Status
-Config::CheckMetricConfigCollector(const std::string& value) {
-    if (value != "prometheus") {
-        std::string msg =
-            "Invalid metric collector: " + value + ". Possible reason: metric_config.collector is invalid.";
-        return Status(SERVER_INVALID_ARGUMENT, msg);
+Config::CheckMetricConfigAddress(const std::string& value) {
+    if (!ValidationUtil::ValidateIpAddress(value).ok()) {
+        std::string msg = "Invalid metric ip: " + value + ". Possible reason: metric_config.ip is invalid.";
+        return Status(SERVER_INVALID_ARGUMENT, "Invalid metric config ip: " + value);
     }
     return Status::OK();
 }
 
 Status
-Config::CheckMetricConfigPrometheusPort(const std::string& value) {
+Config::CheckMetricConfigPort(const std::string& value) {
     if (!ValidationUtil::ValidateStringIsNumber(value).ok()) {
-        std::string msg = "Invalid prometheus port: " + value +
-                          ". Possible reason: metric_config.prometheus_config.port is not a number.";
+        std::string msg = "Invalid metric port: " + value + ". Possible reason: metric_config.port is not a number.";
         return Status(SERVER_INVALID_ARGUMENT, msg);
     } else {
         int32_t port = std::stoi(value);
         if (!(port > 1024 && port < 65535)) {
-            std::string msg = "Invalid prometheus port: " + value +
-                              ". Possible reason: metric_config.prometheus_config.port is not in range (1024, 65535).";
+            std::string msg = "Invalid metric port: " + value +
+                              ". Possible reason: metric_config.port is not in range (1024, 65535).";
             return Status(SERVER_INVALID_ARGUMENT, msg);
         }
     }
@@ -934,13 +932,13 @@ Config::GetDBConfigPreloadTable(std::string& value) {
 /* storage config */
 Status
 Config::GetStorageConfigPrimaryPath(std::string& value) {
-    value = GetConfigStr(CONFIG_DB, CONFIG_STORAGE_PRIMARY_PATH, CONFIG_STORAGE_PRIMARY_PATH_DEFAULT);
+    value = GetConfigStr(CONFIG_STORAGE, CONFIG_STORAGE_PRIMARY_PATH, CONFIG_STORAGE_PRIMARY_PATH_DEFAULT);
     return CheckStorageConfigPrimaryPath(value);
 }
 
 Status
 Config::GetStorageConfigSecondaryPath(std::string& value) {
-    value = GetConfigStr(CONFIG_DB, CONFIG_STORAGE_SECONDARY_PATH, CONFIG_STORAGE_SECONDARY_PATH_DEFAULT);
+    value = GetConfigStr(CONFIG_STORAGE, CONFIG_STORAGE_SECONDARY_PATH, CONFIG_STORAGE_SECONDARY_PATH_DEFAULT);
     return CheckStorageConfigSecondaryPath(value);
 }
 
@@ -994,15 +992,15 @@ Config::GetMetricConfigEnableMonitor(bool& value) {
 }
 
 Status
-Config::GetMetricConfigCollector(std::string& value) {
-    value = GetConfigStr(CONFIG_METRIC, CONFIG_METRIC_COLLECTOR, CONFIG_METRIC_COLLECTOR_DEFAULT);
+Config::GetMetricConfigAddress(std::string& value) {
+    value = GetConfigStr(CONFIG_METRIC, CONFIG_METRIC_ADDRESS, CONFIG_METRIC_ADDRESS_DEFAULT);
     return Status::OK();
 }
 
 Status
-Config::GetMetricConfigPrometheusPort(std::string& value) {
-    value = GetConfigStr(CONFIG_METRIC, CONFIG_METRIC_PROMETHEUS_PORT, CONFIG_METRIC_PROMETHEUS_PORT_DEFAULT);
-    return CheckMetricConfigPrometheusPort(value);
+Config::GetMetricConfigPort(std::string& value) {
+    value = GetConfigStr(CONFIG_METRIC, CONFIG_METRIC_PORT, CONFIG_METRIC_PORT_DEFAULT);
+    return CheckMetricConfigPort(value);
 }
 
 /* cache config */
@@ -1214,19 +1212,19 @@ Config::SetDBConfigInsertBufferSize(const std::string& value) {
 Status
 Config::SetStorageConfigPrimaryPath(const std::string& value) {
     CONFIG_CHECK(CheckStorageConfigPrimaryPath(value));
-    return SetConfigValueInMem(CONFIG_DB, CONFIG_STORAGE_PRIMARY_PATH, value);
+    return SetConfigValueInMem(CONFIG_STORAGE, CONFIG_STORAGE_PRIMARY_PATH, value);
 }
 
 Status
 Config::SetStorageConfigSecondaryPath(const std::string& value) {
     CONFIG_CHECK(CheckStorageConfigSecondaryPath(value));
-    return SetConfigValueInMem(CONFIG_DB, CONFIG_STORAGE_SECONDARY_PATH, value);
+    return SetConfigValueInMem(CONFIG_STORAGE, CONFIG_STORAGE_SECONDARY_PATH, value);
 }
 
 Status
 Config::SetStorageConfigMinioEnable(const std::string& value) {
     CONFIG_CHECK(CheckStorageConfigMinioEnable(value));
-    return SetConfigValueInMem(CONFIG_METRIC, CONFIG_METRIC_ENABLE_MONITOR, value);
+    return SetConfigValueInMem(CONFIG_STORAGE, CONFIG_STORAGE_MINIO_ENABLE, value);
 }
 
 Status
@@ -1267,15 +1265,15 @@ Config::SetMetricConfigEnableMonitor(const std::string& value) {
 }
 
 Status
-Config::SetMetricConfigCollector(const std::string& value) {
-    CONFIG_CHECK(CheckMetricConfigCollector(value));
-    return SetConfigValueInMem(CONFIG_METRIC, CONFIG_METRIC_COLLECTOR, value);
+Config::SetMetricConfigAddress(const std::string& value) {
+    CONFIG_CHECK(CheckMetricConfigAddress(value));
+    return SetConfigValueInMem(CONFIG_METRIC, CONFIG_METRIC_ADDRESS, value);
 }
 
 Status
-Config::SetMetricConfigPrometheusPort(const std::string& value) {
-    CONFIG_CHECK(CheckMetricConfigPrometheusPort(value));
-    return SetConfigValueInMem(CONFIG_METRIC, CONFIG_METRIC_PROMETHEUS_PORT, value);
+Config::SetMetricConfigPort(const std::string& value) {
+    CONFIG_CHECK(CheckMetricConfigPort(value));
+    return SetConfigValueInMem(CONFIG_METRIC, CONFIG_METRIC_PORT, value);
 }
 
 /* cache config */

core/src/server/Config.h

@@ -95,11 +95,10 @@ static const char* CONFIG_CACHE_CACHE_INSERT_DATA_DEFAULT = "false";
 static const char* CONFIG_METRIC = "metric_config";
 static const char* CONFIG_METRIC_ENABLE_MONITOR = "enable_monitor";
 static const char* CONFIG_METRIC_ENABLE_MONITOR_DEFAULT = "false";
-static const char* CONFIG_METRIC_COLLECTOR = "collector";
-static const char* CONFIG_METRIC_COLLECTOR_DEFAULT = "prometheus";
-static const char* CONFIG_METRIC_PROMETHEUS = "prometheus_config";
-static const char* CONFIG_METRIC_PROMETHEUS_PORT = "port";
-static const char* CONFIG_METRIC_PROMETHEUS_PORT_DEFAULT = "8080";
+static const char* CONFIG_METRIC_ADDRESS = "address";
+static const char* CONFIG_METRIC_ADDRESS_DEFAULT = "127.0.0.1";
+static const char* CONFIG_METRIC_PORT = "port";
+static const char* CONFIG_METRIC_PORT_DEFAULT = "9091";
 
 /* engine config */
 static const char* CONFIG_ENGINE = "engine_config";
@@ -210,9 +209,9 @@ class Config {
     Status
     CheckMetricConfigEnableMonitor(const std::string& value);
     Status
-    CheckMetricConfigCollector(const std::string& value);
+    CheckMetricConfigAddress(const std::string& value);
     Status
-    CheckMetricConfigPrometheusPort(const std::string& value);
+    CheckMetricConfigPort(const std::string& value);
 
     /* cache config */
     Status
@@ -298,9 +297,9 @@ class Config {
     Status
     GetMetricConfigEnableMonitor(bool& value);
     Status
-    GetMetricConfigCollector(std::string& value);
+    GetMetricConfigAddress(std::string& value);
     Status
-    GetMetricConfigPrometheusPort(std::string& value);
+    GetMetricConfigPort(std::string& value);
 
     /* cache config */
     Status
@@ -380,9 +379,9 @@ class Config {
     Status
     SetMetricConfigEnableMonitor(const std::string& value);
     Status
-    SetMetricConfigCollector(const std::string& value);
+    SetMetricConfigAddress(const std::string& value);
     Status
-    SetMetricConfigPrometheusPort(const std::string& value);
+    SetMetricConfigPort(const std::string& value);
 
     /* cache config */
     Status

core/src/server/DBWrapper.cpp

@@ -165,7 +165,8 @@ DBWrapper::StartService() {
         db_ = engine::DBFactory::Build(opt);
     } catch (std::exception& ex) {
         std::cerr << "Error: failed to open database: " << ex.what()
-                  << ". Possible reason: the meta system does not work." << std::endl;
+                  << ". Possible reason: Meta Tables schema is damaged "
+                  << "or created by in-compatible Milvus version." << std::endl;
         kill(0, SIGUSR1);
     }
 

core/src/server/grpc_impl/GrpcServer.cpp

@@ -55,6 +55,7 @@ class NoReusePortOption : public ::grpc::ServerBuilderOption {
     void
     UpdateArguments(::grpc::ChannelArguments* args) override {
         args->SetInt(GRPC_ARG_ALLOW_REUSEPORT, 0);
+        args->SetInt(GRPC_ARG_MAX_CONCURRENT_STREAMS, 20);
     }
 
     void

core/src/storage/s3/S3ClientMock.h

@@ -0,0 +1,111 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <memory>
+#include <utility>
+
+#include <aws/core/Aws.h>
+#include <aws/core/auth/AWSCredentialsProvider.h>
+#include <aws/core/client/ClientConfiguration.h>
+#include <aws/core/utils/Outcome.h>
+#include <aws/core/utils/StringUtils.h>
+#include <aws/s3/S3Client.h>
+#include <aws/s3/model/CreateBucketRequest.h>
+#include <aws/s3/model/DeleteBucketRequest.h>
+#include <aws/s3/model/DeleteObjectRequest.h>
+#include <aws/s3/model/GetObjectRequest.h>
+#include <aws/s3/model/PutObjectRequest.h>
+
+namespace milvus {
+namespace storage {
+
+/*
+ * This is a class that represents a S3 Client which is used to mimic the put/get operations of a actual s3 client.
+ * During a put object, the body of the request is stored as well as the metadata of the request. This data is then
+ * populated into a get object result when a get operation is called.
+ */
+class S3ClientMock : public Aws::S3::S3Client {
+ public:
+    explicit S3ClientMock(Aws::Client::ClientConfiguration clientConfiguration = Aws::Client::ClientConfiguration())
+        : S3Client(Aws::Auth::AWSCredentials("", ""), clientConfiguration) {
+    }
+
+    Aws::S3::Model::CreateBucketOutcome
+    CreateBucket(const Aws::S3::Model::CreateBucketRequest& request) const override {
+        Aws::S3::Model::CreateBucketResult result;
+        return Aws::S3::Model::CreateBucketOutcome(std::move(result));
+    }
+
+    Aws::S3::Model::DeleteBucketOutcome
+    DeleteBucket(const Aws::S3::Model::DeleteBucketRequest& request) const override {
+        Aws::NoResult result;
+        return Aws::S3::Model::DeleteBucketOutcome(std::move(result));
+    }
+
+    Aws::S3::Model::PutObjectOutcome
+    PutObject(const Aws::S3::Model::PutObjectRequest& request) const override {
+        Aws::String key = request.GetKey();
+        std::shared_ptr<Aws::IOStream> body = request.GetBody();
+        aws_map_[key] = body;
+
+        Aws::S3::Model::PutObjectResult result;
+        return Aws::S3::Model::PutObjectOutcome(std::move(result));
+    }
+
+    Aws::S3::Model::GetObjectOutcome
+    GetObject(const Aws::S3::Model::GetObjectRequest& request) const override {
+        auto factory = request.GetResponseStreamFactory();
+        Aws::Utils::Stream::ResponseStream resp_stream(factory);
+
+        try {
+            std::shared_ptr<Aws::IOStream> body = aws_map_.at(request.GetKey());
+            Aws::String body_str((Aws::IStreamBufIterator(*body)), Aws::IStreamBufIterator());
+
+            resp_stream.GetUnderlyingStream().write(body_str.c_str(), body_str.length());
+            resp_stream.GetUnderlyingStream().flush();
+            Aws::AmazonWebServiceResult<Aws::Utils::Stream::ResponseStream> awsStream(
+                std::move(resp_stream), Aws::Http::HeaderValueCollection());
+
+            Aws::S3::Model::GetObjectResult result(std::move(awsStream));
+            return Aws::S3::Model::GetObjectOutcome(std::move(result));
+        } catch (...) {
+            return Aws::S3::Model::GetObjectOutcome();
+        }
+    }
+
+    Aws::S3::Model::ListObjectsOutcome
+    ListObjects(const Aws::S3::Model::ListObjectsRequest& request) const override {
+        /* TODO: add object key list into ListObjectsOutcome */
+
+        Aws::S3::Model::ListObjectsResult result;
+        return Aws::S3::Model::ListObjectsOutcome(std::move(result));
+    }
+
+    Aws::S3::Model::DeleteObjectOutcome
+    DeleteObject(const Aws::S3::Model::DeleteObjectRequest& request) const override {
+        Aws::String key = request.GetKey();
+        aws_map_.erase(key);
+        Aws::S3::Model::DeleteObjectResult result;
+        Aws::S3::Model::DeleteObjectOutcome(std::move(result));
+        return result;
+    }
+
+    mutable Aws::Map<Aws::String, std::shared_ptr<Aws::IOStream>> aws_map_;
+};
+
+}  // namespace storage
+}  // namespace milvus

core/src/storage/s3/S3ClientWrapper.cpp

@@ -22,12 +22,14 @@
 #include <aws/s3/model/GetObjectRequest.h>
 #include <aws/s3/model/ListObjectsRequest.h>
 #include <aws/s3/model/PutObjectRequest.h>
+#include <fiu-local.h>
 #include <fstream>
 #include <iostream>
 #include <memory>
 #include <utility>
 
 #include "server/Config.h"
+#include "storage/s3/S3ClientMock.h"
 #include "storage/s3/S3ClientWrapper.h"
 #include "utils/Error.h"
 #include "utils/Log.h"
@@ -40,6 +42,7 @@ S3ClientWrapper::StartService() {
     server::Config& config = server::Config::GetInstance();
     bool minio_enable = false;
     CONFIG_CHECK(config.GetStorageConfigMinioEnable(minio_enable));
+    fiu_do_on("S3ClientWrapper.StartService.minio_disable", minio_enable = false);
     if (!minio_enable) {
         STORAGE_LOG_INFO << "MinIO not enabled!";
         return Status::OK();
@@ -52,29 +55,30 @@ S3ClientWrapper::StartService() {
     CONFIG_CHECK(config.GetStorageConfigMinioBucket(minio_bucket_));
 
     Aws::InitAPI(options_);
-    Aws::Client::ClientConfiguration cfg;
 
+    Aws::Client::ClientConfiguration cfg;
     cfg.endpointOverride = minio_address_ + ":" + minio_port_;
     cfg.scheme = Aws::Http::Scheme::HTTP;
     cfg.verifySSL = false;
-    client_ptr_ = new Aws::S3::S3Client(Aws::Auth::AWSCredentials(minio_access_key_, minio_secret_key_), cfg,
-                                        Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Always, false);
-    if (client_ptr_ == nullptr) {
-        std::string str = "Cannot connect S3 server.";
-        return milvus::Status(SERVER_UNEXPECTED_ERROR, str);
+    client_ptr_ =
+        std::make_shared<Aws::S3::S3Client>(Aws::Auth::AWSCredentials(minio_access_key_, minio_secret_key_), cfg,
+                                            Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Always, false);
+
+    bool mock_enable = false;
+    fiu_do_on("S3ClientWrapper.StartService.mock_enable", mock_enable = true);
+    if (mock_enable) {
+        client_ptr_ = std::make_shared<S3ClientMock>();
     }
 
     return CreateBucket();
 }
 
-Status
+void
 S3ClientWrapper::StopService() {
     if (client_ptr_ != nullptr) {
-        delete client_ptr_;
         client_ptr_ = nullptr;
     }
     Aws::ShutdownAPI(options_);
-    return Status::OK();
 }
 
 Status
@@ -84,6 +88,7 @@ S3ClientWrapper::CreateBucket() {
 
     auto outcome = client_ptr_->CreateBucket(request);
 
+    fiu_do_on("S3ClientWrapper.CreateBucket.outcome.fail", outcome = Aws::S3::Model::CreateBucketOutcome());
     if (!outcome.IsSuccess()) {
         auto err = outcome.GetError();
         if (err.GetErrorType() != Aws::S3::S3Errors::BUCKET_ALREADY_OWNED_BY_YOU) {
@@ -103,6 +108,7 @@ S3ClientWrapper::DeleteBucket() {
 
     auto outcome = client_ptr_->DeleteBucket(request);
 
+    fiu_do_on("S3ClientWrapper.DeleteBucket.outcome.fail", outcome = Aws::S3::Model::DeleteBucketOutcome());
     if (!outcome.IsSuccess()) {
         auto err = outcome.GetError();
         STORAGE_LOG_ERROR << "ERROR: DeleteBucket: " << err.GetExceptionName() << ": " << err.GetMessage();
@@ -131,6 +137,7 @@ S3ClientWrapper::PutObjectFile(const std::string& object_name, const std::string
 
     auto outcome = client_ptr_->PutObject(request);
 
+    fiu_do_on("S3ClientWrapper.PutObjectFile.outcome.fail", outcome = Aws::S3::Model::PutObjectOutcome());
     if (!outcome.IsSuccess()) {
         auto err = outcome.GetError();
         STORAGE_LOG_ERROR << "ERROR: PutObject: " << err.GetExceptionName() << ": " << err.GetMessage();
@@ -152,6 +159,7 @@ S3ClientWrapper::PutObjectStr(const std::string& object_name, const std::string&
 
     auto outcome = client_ptr_->PutObject(request);
 
+    fiu_do_on("S3ClientWrapper.PutObjectStr.outcome.fail", outcome = Aws::S3::Model::PutObjectOutcome());
     if (!outcome.IsSuccess()) {
         auto err = outcome.GetError();
         STORAGE_LOG_ERROR << "ERROR: PutObject: " << err.GetExceptionName() << ": " << err.GetMessage();
@@ -169,6 +177,7 @@ S3ClientWrapper::GetObjectFile(const std::string& object_name, const std::string
 
     auto outcome = client_ptr_->GetObject(request);
 
+    fiu_do_on("S3ClientWrapper.GetObjectFile.outcome.fail", outcome = Aws::S3::Model::GetObjectOutcome());
     if (!outcome.IsSuccess()) {
         auto err = outcome.GetError();
         STORAGE_LOG_ERROR << "ERROR: GetObject: " << err.GetExceptionName() << ": " << err.GetMessage();
@@ -191,6 +200,7 @@ S3ClientWrapper::GetObjectStr(const std::string& object_name, std::string& conte
 
     auto outcome = client_ptr_->GetObject(request);
 
+    fiu_do_on("S3ClientWrapper.GetObjectStr.outcome.fail", outcome = Aws::S3::Model::GetObjectOutcome());
     if (!outcome.IsSuccess()) {
         auto err = outcome.GetError();
         STORAGE_LOG_ERROR << "ERROR: GetObject: " << err.GetExceptionName() << ": " << err.GetMessage();
@@ -217,6 +227,7 @@ S3ClientWrapper::ListObjects(std::vector<std::string>& object_list, const std::s
 
     auto outcome = client_ptr_->ListObjects(request);
 
+    fiu_do_on("S3ClientWrapper.ListObjects.outcome.fail", outcome = Aws::S3::Model::ListObjectsOutcome());
     if (!outcome.IsSuccess()) {
         auto err = outcome.GetError();
         STORAGE_LOG_ERROR << "ERROR: ListObjects: " << err.GetExceptionName() << ": " << err.GetMessage();
@@ -244,6 +255,7 @@ S3ClientWrapper::DeleteObject(const std::string& object_name) {
 
     auto outcome = client_ptr_->DeleteObject(request);
 
+    fiu_do_on("S3ClientWrapper.DeleteObject.outcome.fail", outcome = Aws::S3::Model::DeleteObjectOutcome());
     if (!outcome.IsSuccess()) {
         auto err = outcome.GetError();
         STORAGE_LOG_ERROR << "ERROR: DeleteObject: " << err.GetExceptionName() << ": " << err.GetMessage();

core/src/storage/s3/S3ClientWrapper.h

@@ -19,6 +19,7 @@
 
 #include <aws/core/Aws.h>
 #include <aws/s3/S3Client.h>
+#include <memory>
 #include <string>
 #include <vector>
 #include "storage/IStorage.h"
@@ -28,9 +29,6 @@ namespace storage {
 
 class S3ClientWrapper : public IStorage {
  public:
-    S3ClientWrapper() = default;
-    ~S3ClientWrapper() = default;
-
     static S3ClientWrapper&
     GetInstance() {
         static S3ClientWrapper wrapper;
@@ -39,7 +37,7 @@ class S3ClientWrapper : public IStorage {
 
     Status
     StartService();
-    Status
+    void
     StopService();
 
     Status
@@ -62,7 +60,7 @@ class S3ClientWrapper : public IStorage {
     DeleteObjects(const std::string& marker) override;
 
  private:
-    Aws::S3::S3Client* client_ptr_ = nullptr;
+    std::shared_ptr<Aws::S3::S3Client> client_ptr_;
     Aws::SDKOptions options_;
 
     std::string minio_address_;

core/unittest/db/utils.cpp

@@ -61,13 +61,12 @@ static const char* CONFIG_STR =
     "\n"
     "metric_config:\n"
     "  enable_monitor: false             # enable monitoring or not\n"
-    "  collector: prometheus             # prometheus\n"
-    "  prometheus_config:\n"
-    "    port: 8080                      # port prometheus used to fetch metrics\n"
+    "  address: 127.0.0.1\n"
+    "  port: 9091                        # port prometheus used to fetch metrics\n"
     "\n"
     "cache_config:\n"
-    "  cpu_mem_capacity: 16              # GB, CPU memory used for cache\n"
-    "  cpu_mem_threshold: 0.85           # percentage of data kept when cache cleanup triggered\n"
+    "  cpu_cache_capacity: 4             # GB, CPU memory used for cache\n"
+    "  cpu_cache_threshold: 0.85         # percentage of data kept when cache cleanup triggered\n"
     "  cache_insert_data: false          # whether load inserted data into cache\n"
     "\n"
     "engine_config:\n"

core/unittest/metrics/test_metricbase.cpp

@@ -62,5 +62,6 @@ TEST(MetricbaseTest, METRICBASE_TEST) {
     instance.ConnectionGaugeIncrement();
     instance.ConnectionGaugeDecrement();
     instance.KeepingAliveCounterIncrement();
+    instance.PushToGateway();
     instance.OctetsSet();
 }

core/unittest/metrics/test_metrics.cpp

@@ -30,17 +30,9 @@
 #include "db/meta/SqliteMetaImpl.h"
 
 TEST_F(MetricTest, METRIC_TEST) {
-    milvus::server::Config::GetInstance().SetMetricConfigCollector("zabbix");
-    milvus::server::Metrics::GetInstance();
-    milvus::server::Config::GetInstance().SetMetricConfigCollector("prometheus");
-    milvus::server::Metrics::GetInstance();
-
     milvus::server::SystemInfo::GetInstance().Init();
-//    server::Metrics::GetInstance().Init();
-//    server::Metrics::GetInstance().exposer_ptr()->RegisterCollectable(server::Metrics::GetInstance().registry_ptr());
     milvus::server::Metrics::GetInstance().Init();
 
-//    server::PrometheusMetrics::GetInstance().exposer_ptr()->RegisterCollectable(server::PrometheusMetrics::GetInstance().registry_ptr());
     milvus::cache::CpuCacheMgr::GetInstance()->SetCapacity(1UL * 1024 * 1024 * 1024);
     std::cout << milvus::cache::CpuCacheMgr::GetInstance()->CacheCapacity() << std::endl;
 

core/unittest/metrics/test_prometheus.cpp

@@ -67,6 +67,7 @@ TEST(PrometheusTest, PROMETHEUS_TEST) {
     instance.ConnectionGaugeIncrement();
     instance.ConnectionGaugeDecrement();
     instance.KeepingAliveCounterIncrement();
+    instance.PushToGateway();
     instance.OctetsSet();
 
     instance.CPUCoreUsagePercentSet();

core/unittest/server/test_config.cpp

@@ -177,7 +177,7 @@ TEST_F(ConfigTest, SERVER_CONFIG_VALID_TEST) {
     ASSERT_TRUE(config.GetStorageConfigSecondaryPath(str_val).ok());
     ASSERT_TRUE(str_val == storage_secondary_path);
 
-    bool storage_minio_enable = false;
+    bool storage_minio_enable = true;
     ASSERT_TRUE(config.SetStorageConfigMinioEnable(std::to_string(storage_minio_enable)).ok());
     ASSERT_TRUE(config.GetStorageConfigMinioEnable(bool_val).ok());
     ASSERT_TRUE(bool_val == storage_minio_enable);
@@ -213,15 +213,15 @@ TEST_F(ConfigTest, SERVER_CONFIG_VALID_TEST) {
     ASSERT_TRUE(config.GetMetricConfigEnableMonitor(bool_val).ok());
     ASSERT_TRUE(bool_val == metric_enable_monitor);
 
-    std::string metric_collector = "prometheus";
-    ASSERT_TRUE(config.SetMetricConfigCollector(metric_collector).ok());
-    ASSERT_TRUE(config.GetMetricConfigCollector(str_val).ok());
-    ASSERT_TRUE(str_val == metric_collector);
+    std::string metric_address = "192.168.0.2";
+    ASSERT_TRUE(config.SetMetricConfigAddress(metric_address).ok());
+    ASSERT_TRUE(config.GetMetricConfigAddress(str_val).ok());
+    ASSERT_TRUE(str_val == metric_address);
 
-    std::string metric_prometheus_port = "2222";
-    ASSERT_TRUE(config.SetMetricConfigPrometheusPort(metric_prometheus_port).ok());
-    ASSERT_TRUE(config.GetMetricConfigPrometheusPort(str_val).ok());
-    ASSERT_TRUE(str_val == metric_prometheus_port);
+    std::string metric_port = "2222";
+    ASSERT_TRUE(config.SetMetricConfigPort(metric_port).ok());
+    ASSERT_TRUE(config.GetMetricConfigPort(str_val).ok());
+    ASSERT_TRUE(str_val == metric_port);
 
     /* cache config */
     int64_t cache_cpu_cache_capacity = 1;
@@ -298,12 +298,14 @@ TEST_F(ConfigTest, SERVER_CONFIG_VALID_TEST) {
 #endif
 }
 
-std::string gen_get_command(const std::string& parent_node, const std::string& child_node) {
+std::string
+gen_get_command(const std::string& parent_node, const std::string& child_node) {
     std::string cmd = "get_config " + parent_node + ms::CONFIG_NODE_DELIMITER + child_node;
     return cmd;
 }
 
-std::string gen_set_command(const std::string& parent_node, const std::string& child_node, const std::string& value) {
+std::string
+gen_set_command(const std::string& parent_node, const std::string& child_node, const std::string& value) {
     std::string cmd = "set_config " + parent_node + ms::CONFIG_NODE_DELIMITER + child_node + " " + value;
     return cmd;
 }
@@ -517,9 +519,9 @@ TEST_F(ConfigTest, SERVER_CONFIG_INVALID_TEST) {
     /* metric config */
     ASSERT_FALSE(config.SetMetricConfigEnableMonitor("Y").ok());
 
-    ASSERT_FALSE(config.SetMetricConfigCollector("zilliz").ok());
+    ASSERT_FALSE(config.SetMetricConfigAddress("127.0.0").ok());
 
-    ASSERT_FALSE(config.SetMetricConfigPrometheusPort("0xff").ok());
+    ASSERT_FALSE(config.SetMetricConfigPort("0xff").ok());
 
     /* cache config */
     ASSERT_FALSE(config.SetCacheConfigCpuCacheCapacity("a").ok());

core/unittest/server/utils.cpp

@@ -48,12 +48,11 @@ static const char* VALID_CONFIG_STR =
     "\n"
     "metric_config:\n"
     "  enable_monitor: false             # enable monitoring or not\n"
-    "  collector: prometheus             # prometheus\n"
-    "  prometheus_config:\n"
-    "    port: 8080                      # port prometheus uses to fetch metrics\n"
+    "  address: 127.0.0.1\n"
+    "  port: 8080                        # port prometheus uses to fetch metrics\n"
     "\n"
     "cache_config:\n"
-    "  cpu_cache_capacity: 4            # GB, CPU memory used for cache\n"
+    "  cpu_cache_capacity: 4             # GB, CPU memory used for cache\n"
     "  cpu_cache_threshold: 0.85         \n"
     "  cache_insert_data: false          # whether to load inserted data into cache\n"
     "\n"

core/unittest/storage/CMakeLists.txt

@@ -41,4 +41,4 @@ target_link_libraries(test_storage
         ${unittest_libs}
         )
 
-install(TARGETS test_storage DESTINATION bin)
+install(TARGETS test_storage DESTINATION unittest)

core/unittest/storage/test_s3_client.cpp

@@ -19,27 +19,31 @@
 #include <gtest/gtest.h>
 #include <fstream>
 #include <memory>
+#include <fiu-local.h>
+#include <fiu-control.h>
 
 #include "easyloggingpp/easylogging++.h"
 #include "server/Config.h"
-#include "storage/IStorage.h"
 #include "storage/s3/S3ClientWrapper.h"
+#include "storage/s3/S3IOReader.h"
+#include "storage/s3/S3IOWriter.h"
+#include "storage/IStorage.h"
 #include "storage/utils.h"
 
 INITIALIZE_EASYLOGGINGPP
 
 TEST_F(StorageTest, S3_CLIENT_TEST) {
+    fiu_init(0);
+
     const std::string filename = "/tmp/test_file_in";
+    const std::string filename_dummy = "/tmp/test_file_dummy";
     const std::string filename_out = "/tmp/test_file_out";
-    const std::string object_name = "/tmp/test_obj";
+    const std::string objname = "/tmp/test_obj";
+    const std::string objname_dummy = "/tmp/test_obj_dummy";
     const std::string content = "abcdefghijklmnopqrstuvwxyz";
 
-    std::string config_path(CONFIG_PATH);
-    config_path += CONFIG_FILE;
-    milvus::server::Config& config = milvus::server::Config::GetInstance();
-    ASSERT_TRUE(config.LoadConfigFile(config_path).ok());
-
-    auto storage_inst = milvus::storage::S3ClientWrapper::GetInstance();
+    auto& storage_inst = milvus::storage::S3ClientWrapper::GetInstance();
+    fiu_enable("S3ClientWrapper.StartService.mock_enable", 1, NULL, 0);
     ASSERT_TRUE(storage_inst.StartService().ok());
 
     ///////////////////////////////////////////////////////////////////////////
@@ -64,18 +68,137 @@ TEST_F(StorageTest, S3_CLIENT_TEST) {
     ///////////////////////////////////////////////////////////////////////////
     /* check PutObjectStr() and GetObjectStr() */
     {
-        ASSERT_TRUE(storage_inst.PutObjectStr(object_name, content).ok());
+        ASSERT_TRUE(storage_inst.PutObjectStr(objname, content).ok());
 
         std::string content_out;
-        ASSERT_TRUE(storage_inst.GetObjectStr(object_name, content_out).ok());
+        ASSERT_TRUE(storage_inst.GetObjectStr(objname, content_out).ok());
         ASSERT_TRUE(content_out == content);
     }
 
     ///////////////////////////////////////////////////////////////////////////
+    ASSERT_TRUE(storage_inst.DeleteObject(filename).ok());
+    ASSERT_TRUE(storage_inst.DeleteObject(objname).ok());
+
     ASSERT_TRUE(storage_inst.DeleteObjects("/tmp").ok());
 
     ASSERT_TRUE(storage_inst.DeleteBucket().ok());
 
-    ASSERT_TRUE(storage_inst.StopService().ok());
+    storage_inst.StopService();
 }
 
+TEST_F(StorageTest, S3_RW_TEST) {
+    fiu_init(0);
+
+    const std::string index_name = "/tmp/test_index";
+    const std::string content = "abcdefg";
+
+    auto& storage_inst = milvus::storage::S3ClientWrapper::GetInstance();
+    fiu_enable("S3ClientWrapper.StartService.mock_enable", 1, NULL, 0);
+    ASSERT_TRUE(storage_inst.StartService().ok());
+
+    {
+        milvus::storage::S3IOWriter writer(index_name);
+        size_t len = content.length();
+        writer.write(&len, sizeof(len));
+        writer.write((void*)(content.data()), len);
+        ASSERT_TRUE(len + sizeof(len) == writer.length());
+    }
+
+    {
+        milvus::storage::S3IOReader reader(index_name);
+        size_t length = reader.length();
+        size_t rp = 0;
+        reader.seekg(rp);
+        std::string content_out;
+        while (rp < length) {
+            size_t len;
+            reader.read(&len, sizeof(len));
+            rp += sizeof(len);
+            reader.seekg(rp);
+
+            auto data = new char[len];
+            reader.read(data, len);
+            rp += len;
+            reader.seekg(rp);
+
+            content_out += std::string(data, len);
+
+            delete[] data;
+        }
+
+        ASSERT_TRUE(content == content_out);
+    }
+
+    storage_inst.StopService();
+}
+
+TEST_F(StorageTest, S3_FAIL_TEST) {
+    fiu_init(0);
+
+    const std::string filename = "/tmp/test_file_in";
+    const std::string filename_dummy = "/tmp/test_file_dummy";
+    const std::string filename_out = "/tmp/test_file_out";
+    const std::string objname = "/tmp/test_obj";
+    const std::string objname_dummy = "/tmp/test_obj_dummy";
+    const std::string content = "abcdefghijklmnopqrstuvwxyz";
+
+    auto& storage_inst = milvus::storage::S3ClientWrapper::GetInstance();
+
+    fiu_enable("S3ClientWrapper.StartService.minio_disable", 1, NULL, 0);
+    ASSERT_TRUE(storage_inst.StartService().ok());
+    fiu_disable("S3ClientWrapper.StartService.minio_disable");
+
+    fiu_enable("S3ClientWrapper.StartService.mock_enable", 1, NULL, 0);
+    ASSERT_TRUE(storage_inst.StartService().ok());
+    fiu_disable("S3ClientWrapper.StartService.mock_enable");
+
+    fiu_enable("S3ClientWrapper.CreateBucket.outcome.fail", 1, NULL, 0);
+    ASSERT_FALSE(storage_inst.CreateBucket().ok());
+    fiu_disable("S3ClientWrapper.CreateBucket.outcome.fail");
+
+    ///////////////////////////////////////////////////////////////////////////
+    /* check PutObjectFile() and GetObjectFile() */
+    {
+        fiu_enable("S3ClientWrapper.PutObjectFile.outcome.fail", 1, NULL, 0);
+        ASSERT_FALSE(storage_inst.PutObjectFile(filename, filename).ok());
+        fiu_disable("S3ClientWrapper.PutObjectFile.outcome.fail");
+
+        fiu_enable("S3ClientWrapper.GetObjectFile.outcome.fail", 1, NULL, 0);
+        ASSERT_FALSE(storage_inst.GetObjectFile(filename, filename_out).ok());
+        fiu_disable("S3ClientWrapper.GetObjectFile.outcome.fail");
+
+        ASSERT_FALSE(storage_inst.PutObjectFile(filename_dummy, filename_dummy).ok());
+        ASSERT_FALSE(storage_inst.GetObjectFile(filename_dummy, filename_out).ok());
+    }
+
+    ///////////////////////////////////////////////////////////////////////////
+    /* check PutObjectStr() and GetObjectStr() */
+    {
+        fiu_enable("S3ClientWrapper.PutObjectStr.outcome.fail", 1, NULL, 0);
+        ASSERT_FALSE(storage_inst.PutObjectStr(objname, content).ok());
+        fiu_disable("S3ClientWrapper.PutObjectStr.outcome.fail");
+
+        std::string content_out;
+        fiu_enable("S3ClientWrapper.GetObjectStr.outcome.fail", 1, NULL, 0);
+        ASSERT_FALSE(storage_inst.GetObjectStr(objname, content_out).ok());
+        fiu_disable("S3ClientWrapper.GetObjectStr.outcome.fail");
+
+        ASSERT_FALSE(storage_inst.GetObjectStr(objname_dummy, content_out).ok());
+    }
+
+    ///////////////////////////////////////////////////////////////////////////
+    fiu_enable("S3ClientWrapper.DeleteObject.outcome.fail", 1, NULL, 0);
+    ASSERT_FALSE(storage_inst.DeleteObject(filename).ok());
+    fiu_disable("S3ClientWrapper.DeleteObject.outcome.fail");
+
+    fiu_enable("S3ClientWrapper.ListObjects.outcome.fail", 1, NULL, 0);
+    ASSERT_FALSE(storage_inst.DeleteObjects("/tmp").ok());
+    fiu_disable("S3ClientWrapper.ListObjects.outcome.fail");
+    ASSERT_TRUE(storage_inst.DeleteObjects("/tmp").ok());
+
+    fiu_enable("S3ClientWrapper.DeleteBucket.outcome.fail", 1, NULL, 0);
+    ASSERT_FALSE(storage_inst.DeleteBucket().ok());
+    fiu_disable("S3ClientWrapper.DeleteBucket.outcome.fail");
+
+    storage_inst.StopService();
+}

core/unittest/storage/utils.cpp

@@ -18,6 +18,7 @@
 #include <fstream>
 #include <string>
 
+#include "server/Config.h"
 #include "storage/utils.h"
 #include "utils/CommonUtil.h"
 
@@ -50,7 +51,11 @@ void
 StorageTest::SetUp() {
     std::string config_path(CONFIG_PATH);
     milvus::server::CommonUtil::CreateDirectory(config_path);
-    WriteToFile(config_path + CONFIG_FILE, CONFIG_STR);
+    config_path += CONFIG_FILE;
+    WriteToFile(config_path, CONFIG_STR);
+
+    milvus::server::Config& config = milvus::server::Config::GetInstance();
+    ASSERT_TRUE(config.LoadConfigFile(config_path).ok());
 }
 
 void

core/unittest/wrapper/utils.cpp

@@ -45,13 +45,12 @@ static const char* CONFIG_STR =
     "\n"
     "metric_config:\n"
     "  enable_monitor: false             # enable monitoring or not\n"
-    "  collector: prometheus             # prometheus\n"
-    "  prometheus_config:\n"
-    "    port: 8080                      # port prometheus used to fetch metrics\n"
+    "  address: 127.0.0.1\n"
+    "  port: 8080                        # port prometheus used to fetch metrics\n"
     "\n"
     "cache_config:\n"
-    "  cpu_mem_capacity: 16              # GB, CPU memory used for cache\n"
-    "  cpu_mem_threshold: 0.85           # percentage of data kept when cache cleanup triggered\n"
+    "  cpu_cache_capacity: 4             # GB, CPU memory used for cache\n"
+    "  cpu_cache_threshold: 0.85         # percentage of data kept when cache cleanup triggered\n"
     "  cache_insert_data: false          # whether load inserted data into cache\n"
     "\n"
     "engine_config:\n"

docker-compose.yml

@@ -40,7 +40,7 @@ services:
       - milvus
     command: &ubuntu-command >
       /bin/bash -c "
-        /milvus/ci/scripts/build.sh -t Release -o ${MILVUS_INSTALL_PREFIX} -l -u -c
+        /milvus/ci/scripts/build.sh -t Release -o ${MILVUS_INSTALL_PREFIX} -l -u -c -i
         /milvus/ci/scripts/coverage.sh -o ${MILVUS_INSTALL_PREFIX} -u root -p 123456 -t mysql"
 
   centos-core:
@@ -60,7 +60,7 @@ services:
       - milvus
     command: &centos-command >
       /bin/bash --login -c "
-        /milvus/ci/scripts/build.sh -t Release -o ${MILVUS_INSTALL_PREFIX} -l -u -c
+        /milvus/ci/scripts/build.sh -t Release -o ${MILVUS_INSTALL_PREFIX} -l -u -c -i
         /milvus/ci/scripts/coverage.sh -o ${MILVUS_INSTALL_PREFIX} -u root -p 123456 -t mysql"
 
 networks:

docs/README.md

@@ -16,10 +16,10 @@ The following is a list of existing test reports:
 
 - [IVF_SQ8](test_report/milvus_ivfsq8_test_report_detailed_version.md)
 - [IVF_SQ8H](test_report/milvus_ivfsq8h_test_report_detailed_version.md)
-- [IVFLAT](test-report/ivfflat_test_report_en.md)
+- [IVFLAT](test_report/ivfflat_test_report_en.md)
 
 To read the CN version of these reports:
 
 - [IVF_SQ8_cn](test_report/milvus_ivfsq8_test_report_detailed_version_cn.md)
 - [IVF_SQ8H_cn](test_report/milvus_ivfsq8h_test_report_detailed_version_cn.md)
-- [IVFLAT_cn](test-report/ivfflat_test_report_cn.md)
+- [IVFLAT_cn](test_report/ivfflat_test_report_cn.md)

tests/milvus-java-test/pom.xml

@@ -99,7 +99,7 @@
         <dependency>
             <groupId>io.milvus</groupId>
             <artifactId>milvus-sdk-java</artifactId>
-            <version>0.3.0</version>
+            <version>0.4.1-SNAPSHOT</version>
         </dependency>
 
 <!--        <dependency>-->
@@ -134,4 +134,4 @@
 
     </dependencies>
 
-</project>
+</project>

tests/milvus-java-test/src/main/java/com/MainClass.java

@@ -15,7 +15,7 @@ import java.util.List;
 
 public class MainClass {
     private static String host = "127.0.0.1";
-    private static String port = "19530";
+    private static String port = "19532";
     private int index_file_size = 50;
     public int dimension = 128;
 

tests/milvus-java-test/src/main/java/com/Partition.java

@@ -0,0 +1,205 @@
+package com;
+
+import io.milvus.client.MilvusClient;
+import io.milvus.client.Response;
+import io.milvus.client.ShowPartitionsResponse;
+import org.apache.commons.lang3.RandomStringUtils;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Random;
+
+public class Partition {
+    int dimension = 128;
+
+    public List<List<Float>> gen_vectors(Integer nb) {
+        List<List<Float>> xb = new LinkedList<>();
+        Random random = new Random();
+        for (int i = 0; i < nb; ++i) {
+            LinkedList<Float> vector = new LinkedList<>();
+            for (int j = 0; j < dimension; j++) {
+                vector.add(random.nextFloat());
+            }
+            xb.add(vector);
+        }
+        return xb;
+    }
+
+    // ----------------------------- create partition cases in ---------------------------------
+
+    // create partition
+    @Test(dataProvider = "Table", dataProviderClass = MainClass.class)
+    public void test_create_partition(MilvusClient client, String tableName) {
+        String partitionName = RandomStringUtils.randomAlphabetic(10);
+        String tag = RandomStringUtils.randomAlphabetic(10);
+        io.milvus.client.Partition partition = new io.milvus.client.Partition.Builder(tableName, partitionName, tag).build();
+        Response createpResponse = client.createPartition(partition);
+        assert (createpResponse.ok());
+        // show partitions
+        List<io.milvus.client.Partition> partitions = client.showPartitions(tableName).getPartitionList();
+        System.out.println(partitions);
+        List<String> partitionNames = new ArrayList<>();
+        for (int i=0; i<partitions.size(); ++i) {
+            partitionNames.add(partitions.get(i).getPartitionName());
+        }
+        Assert.assertTrue(partitionNames.contains(partitionName));
+    }
+
+    // create partition, partition existed
+    @Test(dataProvider = "Table", dataProviderClass = MainClass.class)
+    public void test_create_partition_name_existed(MilvusClient client, String tableName) {
+        String partitionName = RandomStringUtils.randomAlphabetic(10);
+        String tag = RandomStringUtils.randomAlphabetic(10);
+        io.milvus.client.Partition partition = new io.milvus.client.Partition.Builder(tableName, partitionName, tag).build();
+        Response createpResponse = client.createPartition(partition);
+        assert (createpResponse.ok());
+        String newTag = RandomStringUtils.randomAlphabetic(10);
+        io.milvus.client.Partition newPartition = new io.milvus.client.Partition.Builder(tableName, partitionName, newTag).build();
+        Response newCreatepResponse = client.createPartition(newPartition);
+        assert (!newCreatepResponse.ok());
+    }
+
+    // create partition, tag name existed
+    @Test(dataProvider = "Table", dataProviderClass = MainClass.class)
+    public void test_create_partition_tag_name_existed(MilvusClient client, String tableName) {
+        String partitionName = RandomStringUtils.randomAlphabetic(10);
+        String tag = RandomStringUtils.randomAlphabetic(10);
+        io.milvus.client.Partition partition = new io.milvus.client.Partition.Builder(tableName, partitionName, tag).build();
+        Response createpResponse = client.createPartition(partition);
+        assert (createpResponse.ok());
+        io.milvus.client.Partition newPartition = new io.milvus.client.Partition.Builder(tableName, partitionName, tag).build();
+        Response newCreatepResponse = client.createPartition(newPartition);
+        assert (!newCreatepResponse.ok());
+    }
+
+    // ----------------------------- drop partition cases in ---------------------------------
+
+    // drop a partition created before, drop by partition name
+    @Test(dataProvider = "Table", dataProviderClass = MainClass.class)
+    public void test_drop_partition(MilvusClient client, String tableName) {
+        String partitionName = RandomStringUtils.randomAlphabetic(10);
+        String tag = RandomStringUtils.randomAlphabetic(10);
+        io.milvus.client.Partition partition = new io.milvus.client.Partition.Builder(tableName, partitionName, tag).build();
+        Response createpResponse = client.createPartition(partition);
+        assert (createpResponse.ok());
+        Response response = client.dropPartition(partitionName);
+        assert (response.ok());
+        // show partitions
+        int length = client.showPartitions(tableName).getPartitionList().size();
+        Assert.assertEquals(length, 0);
+    }
+
+    // drop a partition repeat created before, drop by partition name
+    @Test(dataProvider = "Table", dataProviderClass = MainClass.class)
+    public void test_drop_partition_repeat(MilvusClient client, String tableName) throws InterruptedException {
+        String partitionName = RandomStringUtils.randomAlphabetic(10);
+        String tag = RandomStringUtils.randomAlphabetic(10);
+        io.milvus.client.Partition partition = new io.milvus.client.Partition.Builder(tableName, partitionName, tag).build();
+        Response createpResponse = client.createPartition(partition);
+        assert (createpResponse.ok());
+        Response response = client.dropPartition(partitionName);
+        assert (response.ok());
+        Thread.currentThread().sleep(2000);
+        Response newResponse = client.dropPartition(partitionName);
+        assert (!newResponse.ok());
+    }
+
+    // drop a partition created before, drop by tag
+    @Test(dataProvider = "Table", dataProviderClass = MainClass.class)
+    public void test_drop_partition_with_tag(MilvusClient client, String tableName) {
+        String partitionName = RandomStringUtils.randomAlphabetic(10);
+        String tag = RandomStringUtils.randomAlphabetic(10);
+        io.milvus.client.Partition partition = new io.milvus.client.Partition.Builder(tableName, partitionName, tag).build();
+        Response createpResponse = client.createPartition(partition);
+        assert (createpResponse.ok());
+        Response response = client.dropPartition(tableName, tag);
+        assert (response.ok());
+        // show partitions
+        int length = client.showPartitions(tableName).getPartitionList().size();
+        Assert.assertEquals(length, 0);
+    }
+
+    // drop a partition not created before
+    @Test(dataProvider = "Table", dataProviderClass = MainClass.class)
+    public void test_drop_partition_not_existed(MilvusClient client, String tableName) {
+        String partitionName = RandomStringUtils.randomAlphabetic(10);
+        String tag = RandomStringUtils.randomAlphabetic(10);
+        io.milvus.client.Partition partition = new io.milvus.client.Partition.Builder(tableName, partitionName, tag).build();
+        Response createpResponse = client.createPartition(partition);
+        assert(createpResponse.ok());
+        String newPartitionName = RandomStringUtils.randomAlphabetic(10);
+        Response response = client.dropPartition(newPartitionName);
+        assert(!response.ok());
+    }
+
+    // drop a partition not created before
+    @Test(dataProvider = "Table", dataProviderClass = MainClass.class)
+    public void test_drop_partition_tag_not_existed(MilvusClient client, String tableName) {
+        String partitionName = RandomStringUtils.randomAlphabetic(10);
+        String tag = RandomStringUtils.randomAlphabetic(10);
+        io.milvus.client.Partition partition = new io.milvus.client.Partition.Builder(tableName, partitionName, tag).build();
+        Response createpResponse = client.createPartition(partition);
+        assert(createpResponse.ok());
+        String newTag = RandomStringUtils.randomAlphabetic(10);
+        Response response = client.dropPartition(tableName, newTag);
+        assert(!response.ok());
+    }
+
+    // ----------------------------- show partitions cases in ---------------------------------
+
+    // create partition, then show partitions
+    @Test(dataProvider = "Table", dataProviderClass = MainClass.class)
+    public void test_show_partitions(MilvusClient client, String tableName) {
+        String partitionName = RandomStringUtils.randomAlphabetic(10);
+        String tag = RandomStringUtils.randomAlphabetic(10);
+        io.milvus.client.Partition partition = new io.milvus.client.Partition.Builder(tableName, partitionName, tag).build();
+        Response createpResponse = client.createPartition(partition);
+        assert (createpResponse.ok());
+        ShowPartitionsResponse response = client.showPartitions(tableName);
+        assert (response.getResponse().ok());
+        List<String> partitionNames = new ArrayList<>();
+        for (int i=0; i<response.getPartitionList().size(); ++i) {
+            partitionNames.add(response.getPartitionList().get(i).getPartitionName());
+            if (response.getPartitionList().get(i).getPartitionName() == partitionName) {
+                Assert.assertTrue(response.getPartitionList().get(i).getTableName() == tableName);
+                Assert.assertTrue(response.getPartitionList().get(i).getTag() == tag);
+            }
+        }
+        Assert.assertTrue(partitionNames.contains(partitionName));
+    }
+
+    // create multi partition, then show partitions
+    @Test(dataProvider = "Table", dataProviderClass = MainClass.class)
+    public void test_show_partitions_multi(MilvusClient client, String tableName) {
+        String partitionName = RandomStringUtils.randomAlphabetic(10);
+        String tag = RandomStringUtils.randomAlphabetic(10);
+        io.milvus.client.Partition partition = new io.milvus.client.Partition.Builder(tableName, partitionName, tag).build();
+        Response createpResponse = client.createPartition(partition);
+        assert (createpResponse.ok());
+        String newPartitionName = RandomStringUtils.randomAlphabetic(10);
+        String tagNew = RandomStringUtils.randomAlphabetic(10);
+        io.milvus.client.Partition newPartition = new io.milvus.client.Partition.Builder(tableName, newPartitionName, tagNew).build();
+        Response newCreatepResponse = client.createPartition(newPartition);
+        assert (newCreatepResponse.ok());
+        ShowPartitionsResponse response = client.showPartitions(tableName);
+        assert (response.getResponse().ok());
+        List<String> partitionNames = response.getPartitionNameList();
+//        for (int i=0; i<response.getPartitionList().size(); ++i) {
+//            partitionNames.add(response.getPartitionList().get(i).getPartitionName());
+//            if (response.getPartitionList().get(i).getPartitionName() == newPartitionName) {
+//                Assert.assertTrue(response.getPartitionList().get(i).getTableName() == tableName);
+//                Assert.assertTrue(response.getPartitionList().get(i).getTag() == tagNew);
+//            }
+//        }
+        System.out.println(partitionNames);
+        Assert.assertTrue(partitionNames.contains(partitionName));
+        Assert.assertTrue(partitionNames.contains(newPartitionName));
+        List<String> tagNames = response.getPartitionTagList();
+        Assert.assertTrue(tagNames.contains(tag));
+        Assert.assertTrue(tagNames.contains(tagNew));
+    }
+
+}

tests/milvus-java-test/src/main/java/com/TestAddVectors.java

@@ -1,8 +1,7 @@
 package com;
 
-import io.milvus.client.InsertParam;
-import io.milvus.client.InsertResponse;
-import io.milvus.client.MilvusClient;
+import io.milvus.client.*;
+import org.apache.commons.lang3.RandomStringUtils;
 import org.testng.Assert;
 import org.testng.annotations.Test;
 
@@ -12,6 +11,7 @@ import java.util.stream.Stream;
 
 public class TestAddVectors {
     int dimension = 128;
+    String tag = "tag";
 
     public List<List<Float>> gen_vectors(Integer nb) {
         List<List<Float>> xb = new LinkedList<>();
@@ -28,7 +28,7 @@ public class TestAddVectors {
 
     @Test(dataProvider = "Table", dataProviderClass = MainClass.class)
     public void test_add_vectors_table_not_existed(MilvusClient client, String tableName) throws InterruptedException {
-        int nb = 10000;
+        int nb = 1000;
         List<List<Float>> vectors = gen_vectors(nb);
         String tableNameNew = tableName + "_";
         InsertParam insertParam = new InsertParam.Builder(tableNameNew, vectors).build();
@@ -47,7 +47,7 @@ public class TestAddVectors {
 
     @Test(dataProvider = "Table", dataProviderClass = MainClass.class)
     public void test_add_vectors(MilvusClient client, String tableName) throws InterruptedException {
-        int nb = 10000;
+        int nb = 1000;
         List<List<Float>> vectors = gen_vectors(nb);
         InsertParam insertParam = new InsertParam.Builder(tableName, vectors).build();
         InsertResponse res = client.insert(insertParam);
@@ -79,7 +79,7 @@ public class TestAddVectors {
 
     @Test(dataProvider = "Table", dataProviderClass = MainClass.class)
     public void test_add_vectors_with_ids(MilvusClient client, String tableName) throws InterruptedException {
-        int nb = 10000;
+        int nb = 1000;
         List<List<Float>> vectors = gen_vectors(nb);
         // Add vectors with ids
         List<Long> vectorIds;
@@ -111,7 +111,7 @@ public class TestAddVectors {
 
     @Test(dataProvider = "Table", dataProviderClass = MainClass.class)
     public void test_add_vectors_with_invalid_dimension(MilvusClient client, String tableName) {
-        int nb = 10000;
+        int nb = 1000;
         List<List<Float>> vectors = gen_vectors(nb);
         vectors.get(0).add((float) 0);
         InsertParam insertParam = new InsertParam.Builder(tableName, vectors).build();
@@ -121,7 +121,7 @@ public class TestAddVectors {
 
     @Test(dataProvider = "Table", dataProviderClass = MainClass.class)
     public void test_add_vectors_with_invalid_vectors(MilvusClient client, String tableName) {
-        int nb = 10000;
+        int nb = 1000;
         List<List<Float>> vectors = gen_vectors(nb);
         vectors.set(0, new ArrayList<>());
         InsertParam insertParam = new InsertParam.Builder(tableName, vectors).build();
@@ -147,4 +147,53 @@ public class TestAddVectors {
         Assert.assertEquals(client.getTableRowCount(tableName).getTableRowCount(), nb * loops);
     }
 
+    // ----------------------------- partition cases in Insert ---------------------------------
+    // Add vectors into table with given tag
+    @Test(dataProvider = "Table", dataProviderClass = MainClass.class)
+    public void test_add_vectors_partition(MilvusClient client, String tableName) throws InterruptedException {
+        int nb = 1000;
+        List<List<Float>> vectors = gen_vectors(nb);
+        String partitionName = RandomStringUtils.randomAlphabetic(10);
+        io.milvus.client.Partition partition = new io.milvus.client.Partition.Builder(tableName, partitionName, tag).build();
+        Response createpResponse = client.createPartition(partition);
+        assert(createpResponse.ok());
+        InsertParam insertParam = new InsertParam.Builder(tableName, vectors).withPartitionTag(tag).build();
+        InsertResponse res = client.insert(insertParam);
+        assert(res.getResponse().ok());
+        Thread.currentThread().sleep(1000);
+        // Assert table row count
+        Assert.assertEquals(client.getTableRowCount(tableName).getTableRowCount(), nb);
+    }
+
+    // Add vectors into table, which tag not existed
+    @Test(dataProvider = "Table", dataProviderClass = MainClass.class)
+    public void test_add_vectors_partition_tag_not_existed(MilvusClient client, String tableName) {
+        int nb = 1000;
+        String newTag = RandomStringUtils.randomAlphabetic(10);
+        List<List<Float>> vectors = gen_vectors(nb);
+        String partitionName = RandomStringUtils.randomAlphabetic(10);
+        io.milvus.client.Partition partition = new io.milvus.client.Partition.Builder(tableName, partitionName, tag).build();
+        Response createpResponse = client.createPartition(partition);
+        assert(createpResponse.ok());
+        InsertParam insertParam = new InsertParam.Builder(tableName, vectors).withPartitionTag(newTag).build();
+        InsertResponse res = client.insert(insertParam);
+        assert(!res.getResponse().ok());
+    }
+
+    // Create table, add vectors into table
+    @Test(dataProvider = "Table", dataProviderClass = MainClass.class)
+    public void test_add_vectors_partition_A(MilvusClient client, String tableName) throws InterruptedException {
+        int nb = 1000;
+        List<List<Float>> vectors = gen_vectors(nb);
+        String partitionName = RandomStringUtils.randomAlphabetic(10);
+        io.milvus.client.Partition partition = new io.milvus.client.Partition.Builder(tableName, partitionName, tag).build();
+        Response createpResponse = client.createPartition(partition);
+        assert(createpResponse.ok());
+        InsertParam insertParam = new InsertParam.Builder(tableName, vectors).build();
+        InsertResponse res = client.insert(insertParam);
+        assert(res.getResponse().ok());
+        Thread.currentThread().sleep(1000);
+        // Assert table row count
+        Assert.assertEquals(client.getTableRowCount(tableName).getTableRowCount(), nb);
+    }
 }

tests/milvus-java-test/src/main/java/com/TestConnect.java

@@ -22,12 +22,13 @@ public class TestConnect {
     @Test(dataProvider = "DefaultConnectArgs", dataProviderClass = MainClass.class)
     public void test_connect_repeat(String host, String port) {
         MilvusGrpcClient client = new MilvusGrpcClient();
-        ConnectParam connectParam = new ConnectParam.Builder()
-                .withHost(host)
-                .withPort(port)
-                .build();
+
         Response res = null;
         try {
+            ConnectParam connectParam = new ConnectParam.Builder()
+                    .withHost(host)
+                    .withPort(port)
+                    .build();
             res = client.connect(connectParam);
             res = client.connect(connectParam);
         } catch (ConnectFailedException e) {
@@ -40,14 +41,14 @@ public class TestConnect {
     @Test(dataProvider="InvalidConnectArgs")
     public void test_connect_invalid_connect_args(String ip, String port) {
         MilvusClient client = new MilvusGrpcClient();
-        ConnectParam connectParam = new ConnectParam.Builder()
-                .withHost(ip)
-                .withPort(port)
-                .build();
         Response res = null;
         try {
+            ConnectParam connectParam = new ConnectParam.Builder()
+                    .withHost(ip)
+                    .withPort(port)
+                    .build();
             res = client.connect(connectParam);
-        } catch (ConnectFailedException e) {
+        } catch (Exception e) {
             e.printStackTrace();
         }
         Assert.assertEquals(res, null);

tests/milvus-java-test/src/main/java/com/TestIndex.java

@@ -134,7 +134,7 @@ public class TestIndex {
 
     @Test(dataProvider = "Table", dataProviderClass = MainClass.class)
     public void test_create_index_IVFSQ8H(MilvusClient client, String tableName) throws InterruptedException {
-        IndexType indexType = IndexType.IVF_SQ8H;
+        IndexType indexType = IndexType.IVF_SQ8_H;
         List<List<Float>> vectors = gen_vectors(nb);
         InsertParam insertParam = new InsertParam.Builder(tableName, vectors).build();
         client.insert(insertParam);

tests/milvus-java-test/src/main/java/com/TestMix.java

@@ -120,6 +120,31 @@ public class TestMix {
         Assert.assertEquals(getTableRowCountResponse.getTableRowCount(), thread_num * nb);
     }
 
+    @Test(dataProvider = "Table", dataProviderClass = MainClass.class)
+    public void test_add_vectors_partition_threads(MilvusClient client, String tableName) throws InterruptedException {
+        int thread_num = 10;
+        String tag = RandomStringUtils.randomAlphabetic(10);
+        String partitionName = RandomStringUtils.randomAlphabetic(10);
+        io.milvus.client.Partition partition = new io.milvus.client.Partition.Builder(tableName, partitionName, tag).build();
+        client.createPartition(partition);
+        List<List<Float>> vectors = gen_vectors(nb,false);
+        InsertParam insertParam = new InsertParam.Builder(tableName, vectors).withPartitionTag(tag).build();
+        ForkJoinPool executor = new ForkJoinPool();
+        for (int i = 0; i < thread_num; i++) {
+            executor.execute(
+                    () -> {
+                        InsertResponse res_insert = client.insert(insertParam);
+                        assert (res_insert.getResponse().ok());
+                    });
+        }
+        executor.awaitQuiescence(100, TimeUnit.SECONDS);
+        executor.shutdown();
+
+        Thread.sleep(2000);
+        GetTableRowCountResponse getTableRowCountResponse = client.getTableRowCount(tableName);
+        Assert.assertEquals(getTableRowCountResponse.getTableRowCount(), thread_num * nb);
+    }
+
     @Test(dataProvider = "Table", dataProviderClass = MainClass.class)
     public void test_add_index_vectors_threads(MilvusClient client, String tableName) throws InterruptedException {
         int thread_num = 50;

tests/milvus-java-test/src/main/java/com/TestPS.java

@@ -0,0 +1,115 @@
+package com;
+
+import io.milvus.client.*;
+import org.apache.commons.cli.*;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ForkJoinPool;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public class TestPS {
+    private static int dimension = 128;
+    private static String host = "192.168.1.101";
+    private static String port = "19530";
+
+    public static void setHost(String host) {
+        TestPS.host = host;
+    }
+
+    public static void setPort(String port) {
+        TestPS.port = port;
+    }
+
+    public static List<Float> normalize(List<Float> w2v){
+        float squareSum = w2v.stream().map(x -> x * x).reduce((float) 0, Float::sum);
+        final float norm = (float) Math.sqrt(squareSum);
+        w2v = w2v.stream().map(x -> x / norm).collect(Collectors.toList());
+        return w2v;
+    }
+
+    public static List<List<Float>> gen_vectors(int nb, boolean norm) {
+        List<List<Float>> xb = new ArrayList<>();
+        Random random = new Random();
+        for (int i = 0; i < nb; ++i) {
+            List<Float> vector = new ArrayList<>();
+            for (int j = 0; j < dimension; j++) {
+                vector.add(random.nextFloat());
+            }
+            if (norm == true) {
+                vector = normalize(vector);
+            }
+            xb.add(vector);
+        }
+        return xb;
+    }
+
+    public static void main(String[] args) throws ConnectFailedException {
+        int nq = 5;
+        int nb = 1;
+        int nprobe = 32;
+        int top_k = 10;
+        int loops = 100000;
+//        int index_file_size = 1024;
+        String tableName = "sift_1b_2048_128_l2";
+
+        CommandLineParser parser = new DefaultParser();
+        Options options = new Options();
+        options.addOption("h", "host", true, "milvus-server hostname/ip");
+        options.addOption("p", "port", true, "milvus-server port");
+        try {
+            CommandLine cmd = parser.parse(options, args);
+            String host = cmd.getOptionValue("host");
+            if (host != null) {
+                setHost(host);
+            }
+            String port = cmd.getOptionValue("port");
+            if (port != null) {
+                setPort(port);
+            }
+            System.out.println("Host: "+host+", Port: "+port);
+        }
+        catch(ParseException exp) {
+            System.err.println("Parsing failed.  Reason: " + exp.getMessage() );
+        }
+
+        List<List<Float>> vectors = gen_vectors(nb, true);
+        List<List<Float>> queryVectors = gen_vectors(nq, true);
+        MilvusClient client = new MilvusGrpcClient();
+        ConnectParam connectParam = new ConnectParam.Builder()
+                .withHost(host)
+                .withPort(port)
+                .build();
+        client.connect(connectParam);
+//        String tableName = RandomStringUtils.randomAlphabetic(10);
+//        TableSchema tableSchema = new TableSchema.Builder(tableName, dimension)
+//                .withIndexFileSize(index_file_size)
+//                .withMetricType(MetricType.IP)
+//                .build();
+//        Response res = client.createTable(tableSchema);
+        List<Long> vectorIds;
+        vectorIds = Stream.iterate(0L, n -> n)
+                .limit(nb)
+                .collect(Collectors.toList());
+        InsertParam insertParam = new InsertParam.Builder(tableName, vectors).withVectorIds(vectorIds).build();
+        ForkJoinPool executor_search = new ForkJoinPool();
+        for (int i = 0; i < loops; i++) {
+            executor_search.execute(
+                    () -> {
+                        InsertResponse res_insert = client.insert(insertParam);
+                        assert (res_insert.getResponse().ok());
+                        System.out.println("In insert");
+                        SearchParam searchParam = new SearchParam.Builder(tableName, queryVectors).withNProbe(nprobe).withTopK(top_k).build();
+                        SearchResponse res_search = client.search(searchParam);
+                        assert (res_search.getResponse().ok());
+                    });
+        }
+        executor_search.awaitQuiescence(300, TimeUnit.SECONDS);
+        executor_search.shutdown();
+        GetTableRowCountResponse getTableRowCountResponse = client.getTableRowCount(tableName);
+        System.out.println(getTableRowCountResponse.getTableRowCount());
+    }
+}

tests/milvus-java-test/src/main/java/com/TestSearchVectors.java

@@ -1,10 +1,12 @@
 package com;
 
 import io.milvus.client.*;
+import org.apache.commons.lang3.RandomStringUtils;
 import org.testng.Assert;
 import org.testng.annotations.Test;
 
 import java.util.*;
+import java.util.concurrent.TimeUnit;
 import java.util.stream.Collectors;
 import java.util.stream.Stream;
 
@@ -53,7 +55,7 @@ public class TestSearchVectors {
     }
 
     @Test(dataProvider = "Table", dataProviderClass = MainClass.class)
-    public void test_search_table_not_existed(MilvusClient client, String tableName) throws InterruptedException {
+    public void test_search_table_not_existed(MilvusClient client, String tableName)  {
         String tableNameNew = tableName + "_";
         int nq = 5;
         int nb = 100;
@@ -65,7 +67,7 @@ public class TestSearchVectors {
     }
 
     @Test(dataProvider = "Table", dataProviderClass = MainClass.class)
-    public void test_search_index_IVFLAT(MilvusClient client, String tableName) throws InterruptedException {
+    public void test_search_index_IVFLAT(MilvusClient client, String tableName)  {
         IndexType indexType = IndexType.IVFLAT;
         int nq = 5;
         List<List<Float>> vectors = gen_vectors(nb, false);
@@ -84,7 +86,7 @@ public class TestSearchVectors {
     }
 
     @Test(dataProvider = "Table", dataProviderClass = MainClass.class)
-    public void test_search_ids_IVFLAT(MilvusClient client, String tableName) throws InterruptedException {
+    public void test_search_ids_IVFLAT(MilvusClient client, String tableName)  {
         IndexType indexType = IndexType.IVFLAT;
         int nq = 5;
         List<List<Float>> vectors = gen_vectors(nb, true);
@@ -121,7 +123,7 @@ public class TestSearchVectors {
     }
 
     @Test(dataProvider = "Table", dataProviderClass = MainClass.class)
-    public void test_search_distance_IVFLAT(MilvusClient client, String tableName) throws InterruptedException {
+    public void test_search_distance_IVFLAT(MilvusClient client, String tableName)  {
         IndexType indexType = IndexType.IVFLAT;
         int nq = 5;
         List<List<Float>> vectors = gen_vectors(nb, true);
@@ -144,7 +146,120 @@ public class TestSearchVectors {
     }
 
     @Test(dataProvider = "Table", dataProviderClass = MainClass.class)
-    public void test_search_index_IVFSQ8(MilvusClient client, String tableName) throws InterruptedException {
+    public void test_search_distance_partition(MilvusClient client, String tableName) {
+        IndexType indexType = IndexType.IVFLAT;
+        int nq = 5;
+        String tag = RandomStringUtils.randomAlphabetic(10);
+        List<List<Float>> vectors = gen_vectors(nb, true);
+        List<List<Float>> queryVectors = vectors.subList(0,nq);
+        String partitionName = RandomStringUtils.randomAlphabetic(10);
+        io.milvus.client.Partition partition = new io.milvus.client.Partition.Builder(tableName, partitionName, tag).build();
+        client.createPartition(partition);
+        InsertParam insertParam = new InsertParam.Builder(tableName, vectors).withPartitionTag(tag).build();
+        client.insert(insertParam);
+        Index index = new Index.Builder().withIndexType(indexType)
+                .withNList(n_list)
+                .build();
+        CreateIndexParam createIndexParam = new CreateIndexParam.Builder(tableName).withIndex(index).build();
+        client.createIndex(createIndexParam);
+        SearchParam searchParam = new SearchParam.Builder(tableName, queryVectors).withNProbe(n_probe).withTopK(top_k).build();
+        List<List<SearchResponse.QueryResult>> res_search = client.search(searchParam).getQueryResultsList();
+        double distance = res_search.get(0).get(0).getDistance();
+        if (tableName.startsWith("L2")) {
+            Assert.assertEquals(distance, 0.0, epsilon);
+        }else if (tableName.startsWith("IP")) {
+            Assert.assertEquals(distance, 1.0, epsilon);
+        }
+    }
+
+    @Test(dataProvider = "Table", dataProviderClass = MainClass.class)
+    public void test_search_distance_partition_not_exited(MilvusClient client, String tableName) {
+        IndexType indexType = IndexType.IVFLAT;
+        int nq = 5;
+        String tag = RandomStringUtils.randomAlphabetic(10);
+        List<List<Float>> vectors = gen_vectors(nb, true);
+        List<List<Float>> queryVectors = vectors.subList(0,nq);
+        String partitionName = RandomStringUtils.randomAlphabetic(10);
+        io.milvus.client.Partition partition = new io.milvus.client.Partition.Builder(tableName, partitionName, tag).build();
+        client.createPartition(partition);
+        InsertParam insertParam = new InsertParam.Builder(tableName, vectors).withPartitionTag(tag).build();
+        client.insert(insertParam);
+        Index index = new Index.Builder().withIndexType(indexType)
+                .withNList(n_list)
+                .build();
+        CreateIndexParam createIndexParam = new CreateIndexParam.Builder(tableName).withIndex(index).build();
+        client.createIndex(createIndexParam);
+        String tagNew = RandomStringUtils.randomAlphabetic(10);
+        List<String> queryTags = new ArrayList<>();
+        queryTags.add(tagNew);
+        SearchParam searchParam = new SearchParam.Builder(tableName, queryVectors).withNProbe(n_probe).withTopK(top_k).withPartitionTags(queryTags).build();
+        SearchResponse res_search = client.search(searchParam);
+        assert (res_search.getResponse().ok());
+        Assert.assertEquals(res_search.getQueryResultsList().size(), 0);
+    }
+
+    @Test(dataProvider = "Table", dataProviderClass = MainClass.class)
+    public void test_search_distance_partition_empty(MilvusClient client, String tableName) {
+        IndexType indexType = IndexType.IVFLAT;
+        int nq = 5;
+        String tag = RandomStringUtils.randomAlphabetic(10);
+        List<List<Float>> vectors = gen_vectors(nb, true);
+        List<List<Float>> queryVectors = vectors.subList(0,nq);
+        String partitionName = RandomStringUtils.randomAlphabetic(10);
+        io.milvus.client.Partition partition = new io.milvus.client.Partition.Builder(tableName, partitionName, tag).build();
+        client.createPartition(partition);
+        InsertParam insertParam = new InsertParam.Builder(tableName, vectors).withPartitionTag(tag).build();
+        client.insert(insertParam);
+        Index index = new Index.Builder().withIndexType(indexType)
+                .withNList(n_list)
+                .build();
+        CreateIndexParam createIndexParam = new CreateIndexParam.Builder(tableName).withIndex(index).build();
+        client.createIndex(createIndexParam);
+        String tagNew = "";
+        List<String> queryTags = new ArrayList<>();
+        queryTags.add(tagNew);
+        SearchParam searchParam = new SearchParam.Builder(tableName, queryVectors).withNProbe(n_probe).withTopK(top_k).withPartitionTags(queryTags).build();
+        SearchResponse res_search = client.search(searchParam);
+        assert (!res_search.getResponse().ok());
+    }
+
+    @Test(dataProvider = "Table", dataProviderClass = MainClass.class)
+    public void test_search_distance_partition_A(MilvusClient client, String tableName) throws InterruptedException {
+//        IndexType indexType = IndexType.IVFLAT;
+        String tag = RandomStringUtils.randomAlphabetic(10);
+        String tagNew = RandomStringUtils.randomAlphabetic(10);
+        List<List<Float>> vectors = gen_vectors(nb, true);
+        List<List<Float>> vectorsNew = gen_vectors(nb, true);
+        String partitionName = RandomStringUtils.randomAlphabetic(10);
+        String partitionNameNew = RandomStringUtils.randomAlphabetic(10);
+        io.milvus.client.Partition partition = new io.milvus.client.Partition.Builder(tableName, partitionName, tag).build();
+        io.milvus.client.Partition partitionNew = new io.milvus.client.Partition.Builder(tableName, partitionNameNew, tagNew).build();
+        client.createPartition(partition);
+        client.createPartition(partitionNew);
+        InsertParam insertParam = new InsertParam.Builder(tableName, vectors).withPartitionTag(tag).build();
+        InsertResponse res = client.insert(insertParam);
+        System.out.println(res.getVectorIds().size());
+        InsertParam insertParamNew = new InsertParam.Builder(tableName, vectorsNew).withPartitionTag(tagNew).build();
+        InsertResponse resNew = client.insert(insertParamNew);
+        TimeUnit.SECONDS.sleep(2);
+        System.out.println(resNew.getVectorIds().size());
+        List<String> queryTags = new ArrayList<>();
+        queryTags.add(tag);
+        List<List<Float>> queryVectors;
+        queryVectors = vectors.subList(0,2);
+        queryVectors.add(vectorsNew.get(0));
+        System.out.println(queryVectors.size());
+        SearchParam searchParam = new SearchParam.Builder(tableName, queryVectors).withNProbe(n_probe).withTopK(top_k).withPartitionTags(queryTags).build();
+        List<List<Long>> res_search = client.search(searchParam).getResultIdsList();
+        System.out.println(res_search.get(0));
+        System.out.println(res.getVectorIds());
+//        System.out.println(res_search.get(2));
+        Assert.assertTrue(res.getVectorIds().containsAll(res_search.get(0)));
+        Assert.assertTrue(resNew.getVectorIds().contains(res_search.get(2)));
+    }
+
+    @Test(dataProvider = "Table", dataProviderClass = MainClass.class)
+    public void test_search_index_IVFSQ8(MilvusClient client, String tableName)  {
         IndexType indexType = IndexType.IVF_SQ8;
         int nq = 5;
         List<List<Float>> vectors = gen_vectors(nb, false);
@@ -178,7 +293,7 @@ public class TestSearchVectors {
     }
 
     @Test(dataProvider = "Table", dataProviderClass = MainClass.class)
-    public void test_search_distance_IVFSQ8(MilvusClient client, String tableName) throws InterruptedException {
+    public void test_search_distance_IVFSQ8(MilvusClient client, String tableName)  {
         IndexType indexType = IndexType.IVF_SQ8;
         int nq = 5;
         int nb = 1000;
@@ -192,7 +307,7 @@ public class TestSearchVectors {
         CreateIndexParam createIndexParam = new CreateIndexParam.Builder(tableName).withIndex(index).build();
         client.createIndex(createIndexParam);
         SearchParam searchParam = new SearchParam.Builder(tableName, queryVectors).withNProbe(n_probe).withTopK(top_k).build();
-        List<List<Double>> res_search = client.search(searchParam).getResultDistancesList();
+        List<List<Float>> res_search = client.search(searchParam).getResultDistancesList();
         for (int i = 0; i < nq; i++) {
             double distance = res_search.get(i).get(0);
             System.out.println(distance);
@@ -205,7 +320,7 @@ public class TestSearchVectors {
     }
 
     @Test(dataProvider = "Table", dataProviderClass = MainClass.class)
-    public void test_search_index_FLAT(MilvusClient client, String tableName) throws InterruptedException {
+    public void test_search_index_FLAT(MilvusClient client, String tableName)  {
         IndexType indexType = IndexType.FLAT;
         int nq = 5;
         List<List<Float>> vectors = gen_vectors(nb, false);
@@ -274,7 +389,7 @@ public class TestSearchVectors {
     }
 
     @Test(dataProvider = "Table", dataProviderClass = MainClass.class)
-    public void test_search_distance_FLAT(MilvusClient client, String tableName) throws InterruptedException {
+    public void test_search_distance_FLAT(MilvusClient client, String tableName)  {
         IndexType indexType = IndexType.FLAT;
         int nq = 5;
         List<List<Float>> vectors = gen_vectors(nb, true);
@@ -297,7 +412,7 @@ public class TestSearchVectors {
     }
 
     @Test(dataProvider = "Table", dataProviderClass = MainClass.class)
-    public void test_search_invalid_n_probe(MilvusClient client, String tableName) throws InterruptedException {
+    public void test_search_invalid_n_probe(MilvusClient client, String tableName)  {
         IndexType indexType = IndexType.IVF_SQ8;
         int nq = 5;
         int n_probe_new = 0;
@@ -316,7 +431,7 @@ public class TestSearchVectors {
     }
 
     @Test(dataProvider = "Table", dataProviderClass = MainClass.class)
-    public void test_search_invalid_top_k(MilvusClient client, String tableName) throws InterruptedException {
+    public void test_search_invalid_top_k(MilvusClient client, String tableName)  {
         IndexType indexType = IndexType.IVF_SQ8;
         int nq = 5;
         int top_k_new = 0;
@@ -353,7 +468,7 @@ public class TestSearchVectors {
 //    }
 
     @Test(dataProvider = "Table", dataProviderClass = MainClass.class)
-    public void test_search_index_range(MilvusClient client, String tableName) throws InterruptedException {
+    public void test_search_index_range(MilvusClient client, String tableName)  {
         IndexType indexType = IndexType.IVF_SQ8;
         int nq = 5;
         List<List<Float>> vectors = gen_vectors(nb, false);

tests/milvus_ann_acc/.gitignore

@@ -1,2 +0,0 @@
-__pycache__/
-logs/

tests/milvus_ann_acc/README.md

@@ -1,21 +0,0 @@
-# Requirements
-
-- python 3.6+
-- pip install -r requirements.txt
-
-# How to use this Test Project
-
-This project is used to test search accuracy based on the given datasets (https://github.com/erikbern/ann-benchmarks#data-sets)
-
-1. start your milvus server
-2. update your test configuration in test.py
-3. run command
-
-```shell
-python test.py
-```
-
-# Contribution getting started
-
-- Follow PEP-8 for naming and black for formatting.
-

tests/milvus_ann_acc/ci/jenkinsfile/acc_test.groovy

@@ -1,16 +0,0 @@
-timeout(time: 7200, unit: 'MINUTES') {
-    try {
-        dir ("milvu_ann_acc") {
-            print "Git clone url: ${TEST_URL}:${TEST_BRANCH}"
-            checkout([$class: 'GitSCM', branches: [[name: "${TEST_BRANCH}"]], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: "${params.GIT_USER}", url: "${TEST_URL}", name: 'origin', refspec: "+refs/heads/${TEST_BRANCH}:refs/remotes/origin/${TEST_BRANCH}"]]])
-            print "Install requirements"
-            sh 'python3 -m pip install -r requirements.txt -i http://pypi.douban.com/simple --trusted-host pypi.douban.com'
-            // sleep(120000)
-            sh "python3 main.py --suite=${params.SUITE} --host=acc-test-${env.JOB_NAME}-${env.BUILD_NUMBER}-engine.milvus.svc.cluster.local --port=19530"
-        }
-    } catch (exc) {
-        echo 'Milvus Ann Accuracy Test Failed !'
-        throw exc
-    }
-}
-

tests/milvus_ann_acc/ci/jenkinsfile/cleanup.groovy

@@ -1,13 +0,0 @@
-try {
-    def result = sh script: "helm status ${env.JOB_NAME}-${env.BUILD_NUMBER}", returnStatus: true
-    if (!result) {
-        sh "helm del --purge ${env.JOB_NAME}-${env.BUILD_NUMBER}"
-    }
-} catch (exc) {
-    def result = sh script: "helm status ${env.JOB_NAME}-${env.BUILD_NUMBER}", returnStatus: true
-    if (!result) {
-        sh "helm del --purge ${env.JOB_NAME}-${env.BUILD_NUMBER}"
-    }
-    throw exc
-}
-

tests/milvus_ann_acc/ci/jenkinsfile/deploy_default_server.groovy

@@ -1,22 +0,0 @@
-timeout(time: 30, unit: 'MINUTES') {
-    try {
-        dir ("milvus") {
-            sh 'helm init --client-only --skip-refresh --stable-repo-url https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts'
-            sh 'helm repo update'
-            checkout([$class: 'GitSCM', branches: [[name: "${HELM_BRANCH}"]], userRemoteConfigs: [[url: "${HELM_URL}", name: 'origin', refspec: "+refs/heads/${HELM_BRANCH}:refs/remotes/origin/${HELM_BRANCH}"]]])
-            dir ("milvus") {
-                sh "helm install --wait --timeout 300 --set engine.image.tag=${IMAGE_TAG} --set expose.type=clusterIP --name acc-test-${env.JOB_NAME}-${env.BUILD_NUMBER} -f ci/db_backend/sqlite_${params.IMAGE_TYPE}_values.yaml -f ci/filebeat/values.yaml --namespace milvus --version ${HELM_BRANCH} ."
-            }
-        }
-        // dir ("milvus") {
-        //     checkout([$class: 'GitSCM', branches: [[name: "${env.SERVER_BRANCH}"]], userRemoteConfigs: [[url: "${env.SERVER_URL}", name: 'origin', refspec: "+refs/heads/${env.SERVER_BRANCH}:refs/remotes/origin/${env.SERVER_BRANCH}"]]])
-        //     dir ("milvus") {
-        //         load "ci/jenkins/step/deploySingle2Dev.groovy"
-        //     }
-        // }
-    } catch (exc) {
-        echo 'Deploy Milvus Server Failed !'
-        throw exc
-    }
-}
-

tests/milvus_ann_acc/client.py

@@ -1,146 +0,0 @@
-import pdb
-import random
-import logging
-import json
-import time, datetime
-from multiprocessing import Process
-import numpy
-import sklearn.preprocessing
-from milvus import Milvus, IndexType, MetricType
-
-logger = logging.getLogger("milvus_acc.client")
-
-SERVER_HOST_DEFAULT = "127.0.0.1"
-SERVER_PORT_DEFAULT = 19530
-
-
-def time_wrapper(func):
-    """
-    This decorator prints the execution time for the decorated function.
-    """
-    def wrapper(*args, **kwargs):
-        start = time.time()
-        result = func(*args, **kwargs)
-        end = time.time()
-        logger.info("Milvus {} run in {}s".format(func.__name__, round(end - start, 2)))
-        return result
-    return wrapper
-
-
-class MilvusClient(object):
-    def __init__(self, table_name=None, host=None, port=None):
-        self._milvus = Milvus()
-        self._table_name = table_name
-        try:
-            if not host:
-                self._milvus.connect(
-                    host = SERVER_HOST_DEFAULT,
-                    port = SERVER_PORT_DEFAULT)
-            else:
-                self._milvus.connect(
-                    host = host,
-                    port = port)
-        except Exception as e:
-            raise e
-
-    def __str__(self):
-        return 'Milvus table %s' % self._table_name
-
-    def check_status(self, status):
-        if not status.OK():
-            logger.error(status.message)
-            raise Exception("Status not ok")
-
-    def create_table(self, table_name, dimension, index_file_size, metric_type):
-        if not self._table_name:
-            self._table_name = table_name
-        if metric_type == "l2":
-            metric_type = MetricType.L2
-        elif metric_type == "ip":
-            metric_type = MetricType.IP
-        else:
-            logger.error("Not supported metric_type: %s" % metric_type)
-        self._metric_type = metric_type
-        create_param = {'table_name': table_name,
-                 'dimension': dimension,
-                 'index_file_size': index_file_size, 
-                 "metric_type": metric_type}
-        status = self._milvus.create_table(create_param)
-        self.check_status(status)
-
-    @time_wrapper
-    def insert(self, X, ids):
-        if self._metric_type == MetricType.IP:
-            logger.info("Set normalize for metric_type: Inner Product")
-            X = sklearn.preprocessing.normalize(X, axis=1, norm='l2')
-        X = X.astype(numpy.float32)
-        status, result = self._milvus.add_vectors(self._table_name, X.tolist(), ids=ids)
-        self.check_status(status)
-        return status, result
-
-    @time_wrapper
-    def create_index(self, index_type, nlist):
-        if index_type == "flat":
-            index_type = IndexType.FLAT
-        elif index_type == "ivf_flat":
-            index_type = IndexType.IVFLAT
-        elif index_type == "ivf_sq8":
-            index_type = IndexType.IVF_SQ8
-        elif index_type == "ivf_sq8h":
-            index_type = IndexType.IVF_SQ8H
-        elif index_type == "nsg":
-            index_type = IndexType.NSG
-        elif index_type == "ivf_pq":
-            index_type = IndexType.IVF_PQ
-        index_params = {
-            "index_type": index_type,
-            "nlist": nlist,
-        }
-        logger.info("Building index start, table_name: %s, index_params: %s" % (self._table_name, json.dumps(index_params)))
-        status = self._milvus.create_index(self._table_name, index=index_params, timeout=6*3600)
-        self.check_status(status)
-
-    def describe_index(self):
-        return self._milvus.describe_index(self._table_name)
-
-    def drop_index(self):
-        logger.info("Drop index: %s" % self._table_name)
-        return self._milvus.drop_index(self._table_name)
-
-    @time_wrapper
-    def query(self, X, top_k, nprobe):
-        if self._metric_type == MetricType.IP:
-            logger.info("Set normalize for metric_type: Inner Product")
-            X = sklearn.preprocessing.normalize(X, axis=1, norm='l2')
-        X = X.astype(numpy.float32)
-        status, results = self._milvus.search_vectors(self._table_name, top_k, nprobe, X.tolist())
-        self.check_status(status)
-        ids = []
-        for result in results:
-            tmp_ids = []
-            for item in result:
-                tmp_ids.append(item.id)
-            ids.append(tmp_ids)
-        return ids
-
-    def count(self):
-        return self._milvus.get_table_row_count(self._table_name)[1]
-
-    def delete(self, table_name):
-        logger.info("Start delete table: %s" % table_name)
-        return self._milvus.delete_table(table_name)
-
-    def describe(self):
-        return self._milvus.describe_table(self._table_name)
-
-    def exists_table(self, table_name):
-        return self._milvus.has_table(table_name)
-
-    def get_server_version(self):
-        status, res = self._milvus.server_version()
-        self.check_status(status)
-        return res
-
-    @time_wrapper
-    def preload_table(self):
-        return self._milvus.preload_table(self._table_name)

tests/milvus_ann_acc/config.yaml

@@ -1,17 +0,0 @@
-datasets: 
-  sift-128-euclidean: 
-    cpu_cache_size: 16
-    gpu_cache_size: 5
-    index_file_size: [1024]
-  nytimes-16-angular:
-    cpu_cache_size: 16
-    gpu_cache_size: 5
-    index_file_size: [1024]
-
-index:
-  index_types: ['flat', 'ivf_flat', 'ivf_sq8']
-  nlists: [8092, 16384]
-  
-search:
-  nprobes: [1, 8, 32]
-  top_ks: [10]

tests/milvus_ann_acc/main.py

@@ -1,57 +0,0 @@
-import os
-import sys
-import argparse
-from yaml import load, dump
-import logging
-from logging import handlers
-from client import MilvusClient
-import runner
-
-LOG_FOLDER = "logs"
-logger = logging.getLogger("milvus_acc")
-formatter = logging.Formatter('[%(asctime)s] [%(levelname)-4s] [%(pathname)s:%(lineno)d] %(message)s')
-if not os.path.exists(LOG_FOLDER):
-    os.system('mkdir -p %s' % LOG_FOLDER)
-fileTimeHandler = handlers.TimedRotatingFileHandler(os.path.join(LOG_FOLDER, 'acc'), "D", 1, 10)
-fileTimeHandler.suffix = "%Y%m%d.log"
-fileTimeHandler.setFormatter(formatter)
-logging.basicConfig(level=logging.DEBUG)
-fileTimeHandler.setFormatter(formatter)
-logger.addHandler(fileTimeHandler)
-
-
-def main():
-    parser = argparse.ArgumentParser(
-        formatter_class=argparse.ArgumentDefaultsHelpFormatter)
-    parser.add_argument(
-        "--host",
-        default="127.0.0.1",
-        help="server host")
-    parser.add_argument(
-        "--port",
-        default=19530,
-        help="server port")   
-    parser.add_argument(
-        '--suite',
-        metavar='FILE',
-        help='load config definitions from suite_czr'
-             '.yaml',
-        default='suite_czr.yaml')
-    args = parser.parse_args()
-    if args.suite:
-        with open(args.suite, "r") as f:
-            suite = load(f)
-            hdf5_path = suite["hdf5_path"]
-            dataset_configs = suite["datasets"]
-            if not hdf5_path or not dataset_configs:
-                logger.warning("No datasets given")
-                sys.exit()
-            f.close()
-    for dataset_config in dataset_configs:
-        logger.debug(dataset_config)
-        milvus_instance = MilvusClient(host=args.host, port=args.port)
-        runner.run(milvus_instance, dataset_config, hdf5_path)
-
-
-if __name__ == "__main__":
-    main()

tests/milvus_ann_acc/requirements.txt

@@ -1,9 +0,0 @@
-numpy==1.16.3
-pymilvus-test>=0.2.0
-scikit-learn==0.19.1
-h5py==2.7.1
-influxdb==5.2.2
-pyyaml>=5.1
-tableprint==0.8.0
-ansicolors==1.1.8
-scipy==1.3.1

tests/milvus_ann_acc/runner.py

@@ -1,162 +0,0 @@
-import os
-import pdb
-import time
-import random
-import sys
-import logging
-import h5py
-import numpy
-from influxdb import InfluxDBClient
-
-INSERT_INTERVAL = 100000
-# s
-DELETE_INTERVAL_TIME = 5
-INFLUXDB_HOST = "192.168.1.194"
-INFLUXDB_PORT = 8086
-INFLUXDB_USER = "admin"
-INFLUXDB_PASSWD = "admin"
-INFLUXDB_NAME = "test_result"
-influxdb_client = InfluxDBClient(host=INFLUXDB_HOST, port=INFLUXDB_PORT, username=INFLUXDB_USER, password=INFLUXDB_PASSWD, database=INFLUXDB_NAME)
-
-logger = logging.getLogger("milvus_acc.runner")
-
-
-def parse_dataset_name(dataset_name):
-    data_type = dataset_name.split("-")[0]
-    dimension = int(dataset_name.split("-")[1])
-    metric = dataset_name.split("-")[-1]
-    # metric = dataset.attrs['distance']
-    # dimension = len(dataset["train"][0])
-    if metric == "euclidean":
-        metric_type = "l2"
-    elif metric  == "angular":
-        metric_type = "ip"
-    return ("ann"+data_type, dimension, metric_type)
-
-
-def get_dataset(hdf5_path, dataset_name):
-    file_path = os.path.join(hdf5_path, '%s.hdf5' % dataset_name)
-    if not os.path.exists(file_path):
-        raise Exception("%s not existed" % file_path)
-    dataset = h5py.File(file_path)
-    return dataset
-
-
-def get_table_name(hdf5_path, dataset_name, index_file_size):
-    data_type, dimension, metric_type = parse_dataset_name(dataset_name)
-    dataset = get_dataset(hdf5_path, dataset_name)
-    table_size = len(dataset["train"])
-    table_size = str(table_size // 1000000)+"m"
-    table_name = data_type+'_'+table_size+'_'+str(index_file_size)+'_'+str(dimension)+'_'+metric_type
-    return table_name
-
-
-def recall_calc(result_ids, true_ids, top_k, recall_k):
-    sum_intersect_num = 0
-    recall = 0.0
-    for index, result_item in enumerate(result_ids):
-        # if len(set(true_ids[index][:top_k])) != len(set(result_item)):
-        #     logger.warning("Error happened: query result length is wrong")
-        #     continue
-        tmp = set(true_ids[index][:recall_k]).intersection(set(result_item))
-        sum_intersect_num = sum_intersect_num + len(tmp)
-    recall = round(sum_intersect_num / (len(result_ids) * recall_k), 4)
-    return recall
-
-
-def run(milvus, config, hdf5_path, force=True):
-    server_version = milvus.get_server_version()
-    logger.info(server_version)
-    
-    for dataset_name, config_value in config.items():
-        dataset = get_dataset(hdf5_path, dataset_name)
-        index_file_sizes = config_value["index_file_sizes"]
-        index_types = config_value["index_types"]
-        nlists = config_value["nlists"]
-        search_param = config_value["search_param"]
-        top_ks = search_param["top_ks"]
-        nprobes = search_param["nprobes"]
-        nqs = search_param["nqs"]
-
-        for index_file_size in index_file_sizes:
-            table_name = get_table_name(hdf5_path, dataset_name, index_file_size)  
-            if milvus.exists_table(table_name):
-                if force is True:
-                    logger.info("Re-create table: %s" % table_name)
-                    milvus.delete(table_name)
-                    time.sleep(DELETE_INTERVAL_TIME)
-                else:
-                    logger.warning("Table name: %s existed" % table_name)
-                    continue
-            data_type, dimension, metric_type = parse_dataset_name(dataset_name)
-            milvus.create_table(table_name, dimension, index_file_size, metric_type)
-            logger.info(milvus.describe())
-            insert_vectors = numpy.array(dataset["train"])
-            # milvus.insert(insert_vectors)
-
-            loops = len(insert_vectors) // INSERT_INTERVAL + 1
-            for i in range(loops):
-                start = i*INSERT_INTERVAL
-                end = min((i+1)*INSERT_INTERVAL, len(insert_vectors))
-                tmp_vectors = insert_vectors[start:end]
-                if start < end:
-                    milvus.insert(tmp_vectors, ids=[i for i in range(start, end)])
-            time.sleep(20)
-            row_count = milvus.count()
-            logger.info("Table: %s, row count: %s" % (table_name, row_count))
-            if milvus.count() != len(insert_vectors):
-                logger.error("Table row count is not equal to insert vectors")
-                return
-            for index_type in index_types:
-                for nlist in nlists:
-                    milvus.create_index(index_type, nlist)
-                    logger.info(milvus.describe_index())
-                    logger.info("Start preload table: %s, index_type: %s, nlist: %s" % (table_name, index_type, nlist))
-                    milvus.preload_table()
-                    true_ids = numpy.array(dataset["neighbors"])
-                    for nprobe in nprobes:
-                        for nq in nqs:
-                            query_vectors = numpy.array(dataset["test"][:nq])
-                            for top_k in top_ks:
-                                rec1 = 0.0
-                                rec10 = 0.0
-                                rec100 = 0.0
-                                result_ids = milvus.query(query_vectors, top_k, nprobe)
-                                logger.info("Query result: %s" % len(result_ids))
-                                rec1 = recall_calc(result_ids, true_ids, top_k, 1)
-                                if top_k == 10:
-                                    rec10 = recall_calc(result_ids, true_ids, top_k, 10)
-                                if top_k == 100:
-                                    rec10 = recall_calc(result_ids, true_ids, top_k, 10)
-                                    rec100 = recall_calc(result_ids, true_ids, top_k, 100)
-                                avg_radio = recall_calc(result_ids, true_ids, top_k, top_k)
-                                logger.debug("Recall_1: %s" % rec1)
-                                logger.debug("Recall_10: %s" % rec10)
-                                logger.debug("Recall_100: %s" % rec100)
-                                logger.debug("Accuracy: %s" % avg_radio)
-                                acc_record = [{
-                                    "measurement": "accuracy",
-                                    "tags": {
-                                        "server_version": server_version,
-                                        "dataset": dataset_name,
-                                        "index_file_size": index_file_size,
-                                        "index_type": index_type,
-                                        "nlist": nlist,
-                                        "search_nprobe": nprobe,
-                                        "top_k": top_k,
-                                        "nq": len(query_vectors)
-                                    },
-                                    # "time": time.ctime(),
-                                    "time": time.strftime("%Y-%m-%dT%H:%M:%SZ"),
-                                    "fields": {
-                                        "recall1": rec1,
-                                        "recall10": rec10,
-                                        "recall100": rec100,
-                                        "avg_radio": avg_radio
-                                    }
-                                }]
-                                logger.info(acc_record)
-                                try:
-                                    res = influxdb_client.write_points(acc_record)
-                                except Exception as e:
-                                    logger.error("Insert infuxdb failed: %s" % str(e))

tests/milvus_ann_acc/suite.yaml

@@ -1,29 +0,0 @@
-datasets:
-  - sift-128-euclidean: 
-      index_file_sizes: [1024]
-      index_types: ['ivf_flat', 'ivf_sq8', 'ivf_sq8h']
-      # index_types: ['ivf_sq8']
-      nlists: [16384]
-      search_param:
-        nprobes: [1, 8, 16, 32, 64, 128, 256, 512]
-        top_ks: [10]
-        nqs: [10000]
-  - glove-25-angular:
-      index_file_sizes: [1024]
-      index_types: ['ivf_flat', 'ivf_sq8', 'ivf_sq8h']
-      # index_types: ['ivf_sq8']
-      nlists: [16384]
-      search_param:
-        nprobes: [1, 8, 16, 32, 64, 128, 256, 512]
-        top_ks: [10]
-        nqs: [10000]
-  - glove-200-angular:
-      index_file_sizes: [1024]
-      index_types: ['ivf_flat', 'ivf_sq8', 'ivf_sq8h']
-      # index_types: ['ivf_sq8']
-      nlists: [16384]
-      search_param:
-        nprobes: [1, 8, 16, 32, 64, 128, 256, 512]
-        top_ks: [10]
-        nqs: [10000]
-hdf5_path: /test/milvus/ann_hdf5/

tests/milvus_ann_acc/suite.yaml.bak

@@ -1,11 +0,0 @@
-datasets:
-  - glove-200-angular:
-      index_file_sizes: [1024]
-      index_types: ['ivf_sq8']
-      # index_types: ['ivf_sq8']
-      nlists: [16384]
-      search_param:
-        nprobes: [256, 400, 256]
-        top_ks: [100]
-        nqs: [10000]
-hdf5_path: /test/milvus/ann_hdf5/

tests/milvus_ann_acc/suite_cpu.yaml

@@ -1,19 +0,0 @@
-datasets:
-  - sift-128-euclidean: 
-      index_file_sizes: [1024]
-      index_types: ['ivf_flat', 'ivf_sq8']
-      nlists: [16384]
-      search_param:
-        nprobes: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]
-        top_ks: [10]
-        nqs: [10000]
-  - glove-200-angular:
-      index_file_sizes: [1024]
-      index_types: ['ivf_flat', 'ivf_sq8']
-      # index_types: ['ivf_sq8']
-      nlists: [16384]
-      search_param:
-        nprobes: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]
-        top_ks: [10]
-        nqs: [10000]
-hdf5_path: /test/milvus/ann_hdf5/

tests/milvus_ann_acc/suite_czr.yaml

@@ -1,20 +0,0 @@
-datasets:
-  - sift-128-euclidean:
-      index_file_sizes: [1024]
-      index_types: ['ivf_sq8', 'ivf_sq8h']
-      # index_types: ['ivf_sq8']
-      nlists: [16384]
-      search_param:
-        nprobes: [16, 128, 1024]
-        top_ks: [1, 10, 100]
-        nqs: [10, 100, 1000]
-  - glove-200-angular:
-      index_file_sizes: [1024]
-      index_types: ['ivf_sq8', 'ivf_sq8h']
-      # index_types: ['ivf_sq8']
-      nlists: [16384]
-      search_param:
-        nprobes: [16, 128, 1024]
-        top_ks: [1, 10, 100]
-        nqs: [10, 100, 1000]
-hdf5_path: /test/milvus/ann_hdf5/

tests/milvus_ann_acc/suite_debug.yaml

@@ -1,19 +0,0 @@
-datasets:
-  - sift-128-euclidean: 
-      index_file_sizes: [1024]
-      index_types: ['ivf_flat', 'ivf_sq8']
-      nlists: [16384]
-      search_param:
-        nprobes: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]
-        top_ks: [10]
-        nqs: [10000]
-  - glove-200-angular:
-      index_file_sizes: [1024]
-      index_types: ['ivf_flat', 'ivf_sq8']
-      # index_types: ['ivf_sq8']
-      nlists: [16384]
-      search_param:
-        nprobes: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]
-        top_ks: [10]
-        nqs: [10000]
-hdf5_path: /test/milvus/ann_hdf5/

tests/milvus_ann_acc/suite_gpu.yaml

@@ -1,19 +0,0 @@
-datasets:
-  - sift-128-euclidean: 
-      index_file_sizes: [1024]
-      index_types: ['ivf_flat', 'ivf_sq8', 'ivf_sq8h', 'ivf_pq']
-      nlists: [16384]
-      search_param:
-        nprobes: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]
-        top_ks: [10]
-        nqs: [10000]
-  - glove-200-angular:
-      index_file_sizes: [1024]
-      index_types: ['ivf_flat', 'ivf_sq8']
-      # index_types: ['ivf_sq8']
-      nlists: [16384]
-      search_param:
-        nprobes: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]
-        top_ks: [10]
-        nqs: [10000]
-hdf5_path: /test/milvus/ann_hdf5/

tests/milvus_ann_acc/test.py

@@ -1,33 +0,0 @@
-import time
-from influxdb import InfluxDBClient
-
-INFLUXDB_HOST = "192.168.1.194"
-INFLUXDB_PORT = 8086
-INFLUXDB_USER = "admin"
-INFLUXDB_PASSWD = "admin"
-INFLUXDB_NAME = "test_result"
-
-client = InfluxDBClient(host=INFLUXDB_HOST, port=INFLUXDB_PORT, username=INFLUXDB_USER, password=INFLUXDB_PASSWD, database=INFLUXDB_NAME)
-
-print(client.get_list_database())
-acc_record = [{
-    "measurement": "accuracy",
-    "tags": {
-        "server_version": "0.4.3",
-        "dataset": "test",
-        "index_type": "test",
-        "nlist": 12,
-        "search_nprobe": 12,
-        "top_k": 1,
-        "nq": 1
-    },
-    "time": time.ctime(),
-    "fields": {
-        "accuracy": 0.1
-    }
-}]
-try:
-	res = client.write_points(acc_record)
-	print(res)
-except Exception as e:
-    print(str(e))

tests/milvus_benchmark/README.md

@@ -1,23 +1,42 @@
-# Requirements
+# Quick start
 
-- python 3.6+
-- pip install -r requirements.txt
+## 运行
 
-# How to use this Test Project
+### 运行说明：
 
-This project is used to test performance / accuracy /  stability of milvus server
+- 用于进行大数据集的准确性、性能、以及稳定性等相关测试
+- 可以运行两种模式：基于K8S+Jenkins的测试模式，以及local模式
 
-1. update your test configuration in suites_*.yaml
-2. run command
+### 运行示例：
 
-```shell
-### docker mode:
-python main.py --image=milvusdb/milvus:latest --run-count=2 --run-type=performance
+1. 基于K8S+Jenkins的测试方式：
 
-### local mode:
-python main.py --local --run-count=2 --run-type=performance --ip=127.0.0.1 --port=19530
-```
+   ![](assets/Parameters.png)
 
-# Contribution getting started
+2. 本地测试：
 
-- Follow PEP-8 for naming and black for formatting.
+   `python3 main.py --local --host=*.* --port=19530 --suite=suites/gpu_search_performance_random50m.yaml`
+
+### 测试集配置文件：
+
+在进行自定义的测试集或测试参数时，需要编写测试集配置文件。
+
+下面是搜索性能的测试集配置文件：
+
+![](assets/gpu_search_performance_random50m-yaml.png)
+
+1. search_performance: 定义测试类型，还包括`build_performance`,`insert_performance`,`accuracy`,`stability`,`search_stability`
+2. tables: 定义测试集列表
+3. 对于`search_performance`这类测试，每个table下都包含：
+   - server: milvus的server_config
+   - table_name: 表名，当前框架仅支持单表操作
+   - run_count: 搜索运行次数，并取搜索时间的最小值作为指标
+   - search_params: 向量搜索参数
+
+## 测试结果：
+
+搜索性能的结果输出：
+
+![](assets/milvus-nightly-performance-new-jenkins.png)
+
+在K8S测试模式下时，除打印上面的输出外，还会进行数据上报

tests/milvus_benchmark/ci/jenkinsfile/cleanup.groovy

@@ -0,0 +1,13 @@
+try {
+    def result = sh script: "helm status benchmark-test-${env.JOB_NAME}-${env.BUILD_NUMBER}", returnStatus: true
+    if (!result) {
+        sh "helm del --purge benchmark-test-${env.JOB_NAME}-${env.BUILD_NUMBER}"
+    }
+} catch (exc) {
+    def result = sh script: "helm status benchmark-test-${env.JOB_NAME}-${env.BUILD_NUMBER}", returnStatus: true
+    if (!result) {
+        sh "helm del --purge benchmark-test-${env.JOB_NAME}-${env.BUILD_NUMBER}"
+    }
+    throw exc
+}
+

tests/milvus_benchmark/ci/jenkinsfile/deploy_test.groovy

@@ -0,0 +1,20 @@
+timeout(time: 1440, unit: 'MINUTES') {
+    try {
+        dir ("milvus-helm") {
+            sh 'helm init --client-only --skip-refresh --stable-repo-url https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts'
+            sh 'helm repo update'
+            checkout([$class: 'GitSCM', branches: [[name: "${HELM_BRANCH}"]], userRemoteConfigs: [[url: "${HELM_URL}", name: 'origin', refspec: "+refs/heads/${HELM_BRANCH}:refs/remotes/origin/${HELM_BRANCH}"]]])
+        }
+        dir ("milvus_benchmark") {
+            print "Git clone url: ${TEST_URL}:${TEST_BRANCH}"
+            checkout([$class: 'GitSCM', branches: [[name: "${TEST_BRANCH}"]], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: "${params.GIT_USER}", url: "${TEST_URL}", name: 'origin', refspec: "+refs/heads/${TEST_BRANCH}:refs/remotes/origin/${TEST_BRANCH}"]]])
+            print "Install requirements"
+            sh "python3 -m pip install -r requirements.txt -i http://pypi.douban.com/simple --trusted-host pypi.douban.com"
+            sh "python3 -m pip install git+${TEST_LIB_URL}"
+            sh "python3 main.py --hostname=${params.SERVER_HOST} --image-tag=${IMAGE_TAG} --image-type=${params.IMAGE_TYPE} --suite=suites/${params.SUITE}"
+        }
+    } catch (exc) {
+        echo 'Deploy Test Failed !'
+        throw exc
+    }
+}

tests/milvus_benchmark/ci/main_jenkinsfile

@@ -6,9 +6,10 @@ pipeline {
     }
 
     parameters{
-        choice choices: ['cpu', 'gpu'], description: 'cpu or gpu version', name: 'IMAGE_TYPE'
-        string defaultValue: '0.6.0', description: 'server image version', name: 'IMAGE_VERSION', trim: true
-        string defaultValue: 'suite.yaml', description: 'test suite config yaml', name: 'SUITE', trim: true
+        choice choices: ['gpu', 'cpu'], description: 'cpu or gpu version', name: 'IMAGE_TYPE'
+        string defaultValue: 'master', description: 'server image version', name: 'IMAGE_VERSION', trim: true
+        choice choices: ['poseidon', 'eros', 'apollo', 'athena'], description: 'server host', name: 'SERVER_HOST'
+        string defaultValue: 'gpu_search_performance_sift1m.yaml', description: 'test suite config yaml', name: 'SUITE', trim: true
         string defaultValue: '09509e53-9125-4f5d-9ce8-42855987ad67', description: 'git credentials', name: 'GIT_USER', trim: true
     }
 
@@ -16,15 +17,16 @@ pipeline {
         IMAGE_TAG = "${params.IMAGE_VERSION}-${params.IMAGE_TYPE}-ubuntu18.04-release"
         HELM_URL = "https://github.com/milvus-io/milvus-helm.git"
         HELM_BRANCH = "0.6.0"
-        TEST_URL = "git@192.168.1.105:Test/milvus_ann_acc.git"
-        TEST_BRANCH = "0.6.0"
+        TEST_URL = "git@192.168.1.105:Test/milvus_benchmark.git"
+        TEST_BRANCH = "master"
+        TEST_LIB_URL = "http://192.168.1.105:6060/Test/milvus_metrics.git"
     }
 
     stages {
         stage("Setup env") {
             agent {
                 kubernetes {
-                    label 'dev-test'
+                    label "test-benchmark-${env.JOB_NAME}-${env.BUILD_NUMBER}"
                     defaultContainer 'jnlp'
                     yaml """
                         apiVersion: v1
@@ -44,14 +46,26 @@ pipeline {
                             - name: kubeconf
                               mountPath: /root/.kube/
                               readOnly: true
-                            - name: hdf5-path
-                              mountPath: /test
+                            - name: raw-data-path
+                              mountPath: /poc
                               readOnly: true
+                            - name: db-data-path
+                              mountPath: /test
+                              readOnly: false
                           volumes:
                           - name: kubeconf
                             secret:
                               secretName: test-cluster-config
-                          - name: hdf5-path
+                          - name: raw-data-path
+                            flexVolume:
+                              driver: "fstab/cifs"
+                              fsType: "cifs"
+                              secretRef:
+                                name: "cifs-test-secret"
+                              options:
+                                networkPath: "//192.168.1.126/poc"
+                                mountOptions: "vers=1.0"
+                          - name: db-data-path
                             flexVolume:
                               driver: "fstab/cifs"
                               fsType: "cifs"
@@ -65,30 +79,19 @@ pipeline {
             }
 
             stages {
-                stage("Deploy Default Server") {
+                stage("Deploy Test") {
                     steps {
-                        gitlabCommitStatus(name: 'Accuracy Test') {
+                        gitlabCommitStatus(name: 'Deploy Test') {
                             container('milvus-testframework') {
                                 script {
-                                    print "In Deploy Default Server Stage"
-                                    load "${env.WORKSPACE}/ci/jenkinsfile/deploy_default_server.groovy"
-                                }
-                            }
-                        }
-                    }
-                }
-                stage("Acc Test") {
-                    steps {
-                        gitlabCommitStatus(name: 'Accuracy Test') {
-                            container('milvus-testframework') {
-                                script {
-                                    print "In Acc test stage"
-                                    load "${env.WORKSPACE}/ci/jenkinsfile/acc_test.groovy"
+                                    print "In Deploy Test Stage"
+                                    load "${env.WORKSPACE}/ci/jenkinsfile/deploy_test.groovy"
                                 }
                             }
                         }
                     }
                 }
+
                 stage ("Cleanup Env") {
                     steps {
                         gitlabCommitStatus(name: 'Cleanup Env') {
@@ -111,17 +114,17 @@ pipeline {
                 }
                 success {
                     script {
-                        echo "Milvus ann-accuracy test success !"
+                        echo "Milvus benchmark test success !"
                     }
                 }
                 aborted {
                     script {
-                        echo "Milvus ann-accuracy test aborted !"
+                        echo "Milvus benchmark test aborted !"
                     }
                 }
                 failure {
                     script {
-                        echo "Milvus ann-accuracy test failed !"
+                        echo "Milvus benchmark test failed !"
                     }
                 }
             }

tests/milvus_benchmark/client.py

@@ -1,8 +1,8 @@
+import sys
 import pdb
 import random
 import logging
 import json
-import sys
 import time, datetime
 from multiprocessing import Process
 from milvus import Milvus, IndexType, MetricType
@@ -12,7 +12,14 @@ logger = logging.getLogger("milvus_benchmark.client")
 SERVER_HOST_DEFAULT = "127.0.0.1"
 # SERVER_HOST_DEFAULT = "192.168.1.130"
 SERVER_PORT_DEFAULT = 19530
-
+INDEX_MAP = {
+    "flat": IndexType.FLAT,
+    "ivf_flat": IndexType.IVFLAT,
+    "ivf_sq8": IndexType.IVF_SQ8,
+    "nsg": IndexType.RNSG,
+    "ivf_sq8h": IndexType.IVF_SQ8H,
+    "ivf_pq": IndexType.IVF_PQ
+}
 
 def time_wrapper(func):
     """
@@ -28,18 +35,30 @@ def time_wrapper(func):
 
 
 class MilvusClient(object):
-    def __init__(self, table_name=None, ip=None, port=None):
+    def __init__(self, table_name=None, ip=None, port=None, timeout=60):
         self._milvus = Milvus()
         self._table_name = table_name
         try:
+            i = 1
+            start_time = time.time()
             if not ip:
                 self._milvus.connect(
                     host = SERVER_HOST_DEFAULT,
                     port = SERVER_PORT_DEFAULT)
             else:
-                self._milvus.connect(
-                    host = ip,
-                    port = port)
+                # retry connect for remote server
+                while time.time() < start_time + timeout:
+                    try:
+                        self._milvus.connect(
+                            host = ip,
+                            port = port)
+                        if self._milvus.connected() is True:
+                            logger.debug("Try connect times: %d, %s" % (i, round(time.time() - start_time, 2)))
+                            break
+                    except Exception as e:
+                        logger.debug("Milvus connect failed")
+                        i = i + 1
+
         except Exception as e:
             raise e
 
@@ -49,7 +68,7 @@ class MilvusClient(object):
     def check_status(self, status):
         if not status.OK():
             logger.error(status.message)
-            raise Exception("Status not ok")
+            # raise Exception("Status not ok")
 
     def create_table(self, table_name, dimension, index_file_size, metric_type):
         if not self._table_name:
@@ -58,6 +77,10 @@ class MilvusClient(object):
             metric_type = MetricType.L2
         elif metric_type == "ip":
             metric_type = MetricType.IP
+        elif metric_type == "jaccard":
+            metric_type = MetricType.JACCARD
+        elif metric_type == "hamming":
+            metric_type = MetricType.HAMMING
         else:
             logger.error("Not supported metric_type: %s" % metric_type)
         create_param = {'table_name': table_name,
@@ -75,20 +98,8 @@ class MilvusClient(object):
 
     @time_wrapper
     def create_index(self, index_type, nlist):
-        if index_type == "flat":
-            index_type = IndexType.FLAT
-        elif index_type == "ivf_flat":
-            index_type = IndexType.IVFLAT
-        elif index_type == "ivf_sq8":
-            index_type = IndexType.IVF_SQ8
-        elif index_type == "nsg":
-            index_type = IndexType.NSG
-        elif index_type == "ivf_sq8h":
-            index_type = IndexType.IVF_SQ8H
-        elif index_type == "ivf_pq":
-            index_type = IndexType.IVF_PQ
         index_params = {
-            "index_type": index_type,
+            "index_type": INDEX_MAP[index_type],
             "nlist": nlist,
         }
         logger.info("Building index start, table_name: %s, index_params: %s" % (self._table_name, json.dumps(index_params)))
@@ -96,7 +107,18 @@ class MilvusClient(object):
         self.check_status(status)
 
     def describe_index(self):
-        return self._milvus.describe_index(self._table_name)
+        status, result = self._milvus.describe_index(self._table_name)
+        index_type = None
+        for k, v in INDEX_MAP.items():
+            if result._index_type == v:
+                index_type = k
+                break
+        nlist = result._nlist
+        res = {
+            "index_type": index_type,
+            "nlist": nlist
+        }
+        return res
 
     def drop_index(self):
         logger.info("Drop index: %s" % self._table_name)
@@ -106,7 +128,7 @@ class MilvusClient(object):
     def query(self, X, top_k, nprobe):
         status, result = self._milvus.search_vectors(self._table_name, top_k, nprobe, X)
         self.check_status(status)
-        return status, result
+        return result
 
     def count(self):
         return self._milvus.get_table_row_count(self._table_name)[1]
@@ -122,7 +144,7 @@ class MilvusClient(object):
                 continue
             else:
                 break
-        if i < timeout:
+        if i >= timeout:
             logger.error("Delete table timeout")
 
     def describe(self):
@@ -131,8 +153,10 @@ class MilvusClient(object):
     def show_tables(self):
         return self._milvus.show_tables()
 
-    def exists_table(self):
-        status, res = self._milvus.has_table(self._table_name)
+    def exists_table(self, table_name=None):
+        if table_name is None:
+            table_name = self._table_name
+        status, res = self._milvus.has_table(table_name)
         self.check_status(status)
         return res
 
@@ -140,6 +164,33 @@ class MilvusClient(object):
     def preload_table(self):
         return self._milvus.preload_table(self._table_name, timeout=3000)
 
+    def get_server_version(self):
+        status, res = self._milvus.server_version()
+        return res
+
+    def get_server_mode(self):
+        return self.cmd("mode")
+
+    def get_server_commit(self):
+        return self.cmd("build_commit_id")
+
+    def get_server_config(self):
+        return json.loads(self.cmd("get_config *"))
+
+    def get_mem_info(self):
+        result = json.loads(self.cmd("get_system_info"))
+        result_human = {
+            # unit: Gb
+            "memory_used": round(int(result["memory_used"]) / (1024*1024*1024), 2)
+        }
+        return result_human
+
+    def cmd(self, command):
+        status, res = self._milvus._cmd(command)
+        logger.info("Server command: %s, result: %s" % (command, res))
+        self.check_status(status)
+        return res
+
 
 def fit(table_name, X):
     milvus = Milvus()
@@ -265,6 +316,12 @@ if __name__ == "__main__":
     # data = mmap_fvecs("/poc/deep1b/deep1B_queries.fvecs")
     # data = sklearn.preprocessing.normalize(data, axis=1, norm='l2')
     # np.save("/test/milvus/deep1b/query.npy", data)
+    dimension = 4096
+    insert_xb = 10000
+    insert_vectors = [[random.random() for _ in range(dimension)] for _ in range(insert_xb)]
+    data = sklearn.preprocessing.normalize(insert_vectors, axis=1, norm='l2')
+    np.save("/test/milvus/raw_data/random/query_%d.npy" % dimension, data)
+    sys.exit()
 
     total_size = 100000000
     # total_size = 1000000000

tests/milvus_benchmark/demo.py

@@ -1,51 +0,0 @@
-import os
-import logging
-import pdb
-import time
-import random
-from multiprocessing import Process
-import numpy as np
-from client import MilvusClient
-
-nq = 100000
-dimension = 128
-run_count = 1
-table_name = "sift_10m_1024_128_ip"
-insert_vectors = [[random.random() for _ in range(dimension)] for _ in range(nq)]
-
-def do_query(milvus, table_name, top_ks, nqs, nprobe, run_count):
-    bi_res = []
-    for index, nq in enumerate(nqs):
-        tmp_res = []
-        for top_k in top_ks:
-            avg_query_time = 0.0
-            total_query_time = 0.0
-            vectors = insert_vectors[0:nq]
-            for i in range(run_count):
-                start_time = time.time()
-                status, query_res = milvus.query(vectors, top_k, nprobe)
-                total_query_time = total_query_time + (time.time() - start_time)
-                if status.code:
-                    print(status.message)
-            avg_query_time = round(total_query_time / run_count, 2)
-            tmp_res.append(avg_query_time)
-        bi_res.append(tmp_res)
-    return bi_res
-
-while 1:
-    milvus_instance = MilvusClient(table_name, ip="192.168.1.197", port=19530)
-    top_ks = random.sample([x for x in range(1, 100)], 4)
-    nqs = random.sample([x for x in range(1, 1000)], 3)
-    nprobe = random.choice([x for x in range(1, 500)])
-    res = do_query(milvus_instance, table_name, top_ks, nqs, nprobe, run_count)
-    status, res = milvus_instance.insert(insert_vectors, ids=[x for x in range(len(insert_vectors))])
-    if not status.OK():
-        logger.error(status.message)
-
-    # status = milvus_instance.drop_index()
-    if not status.OK():
-        print(status.message)
-    index_type = "ivf_sq8"
-    status = milvus_instance.create_index(index_type, 16384)
-    if not status.OK():
-        print(status.message)

tests/milvus_benchmark/docker_runner.py

@@ -53,7 +53,6 @@ class DockerRunner(Runner):
                         # milvus.create_index("ivf_sq8", 16384)
                         res = self.do_insert(milvus, table_name, data_type, dimension, table_size, param["ni_per"])
                         logger.info(res)
-
                         # wait for file merge
                         time.sleep(table_size * dimension / 5000000)
                         # Clear up
@@ -71,7 +70,7 @@ class DockerRunner(Runner):
                         container = utils.run_server(self.image, test_type="remote", volume_name=volume_name, db_slave=None)
                         time.sleep(2)
                         milvus = MilvusClient(table_name)
-                        logger.debug(milvus._milvus.show_tables())
+                        logger.debug(milvus.show_tables())
                         # Check has table or not
                         if not milvus.exists_table():
                             logger.warning("Table %s not existed, continue exec next params ..." % table_name)
@@ -104,6 +103,92 @@ class DockerRunner(Runner):
                                     utils.print_table(headers, nqs, res)
                         utils.remove_container(container)
 
+        elif run_type == "insert_performance":
+            for op_type, op_value in definition.items():
+                # run docker mode
+                run_count = op_value["run_count"]
+                run_params = op_value["params"]
+                container = None
+                if not run_params:
+                    logger.debug("No run params")
+                    continue
+                for index, param in enumerate(run_params):
+                    logger.info("Definition param: %s" % str(param))
+                    table_name = param["table_name"]
+                    volume_name = param["db_path_prefix"]
+                    print(table_name)
+                    (data_type, table_size, index_file_size, dimension, metric_type) = parser.table_parser(table_name)
+                    for k, v in param.items():
+                        if k.startswith("server."):
+                            # Update server config
+                            utils.modify_config(k, v, type="server", db_slave=None)
+                    container = utils.run_server(self.image, test_type="remote", volume_name=volume_name, db_slave=None)
+                    time.sleep(2)
+                    milvus = MilvusClient(table_name)
+                    # Check has table or not
+                    if milvus.exists_table():
+                        milvus.delete()
+                        time.sleep(10)
+                    milvus.create_table(table_name, dimension, index_file_size, metric_type)
+                    # debug
+                    # milvus.create_index("ivf_sq8", 16384)
+                    res = self.do_insert(milvus, table_name, data_type, dimension, table_size, param["ni_per"])
+                    logger.info(res)
+                    # wait for file merge
+                    time.sleep(table_size * dimension / 5000000)
+                    # Clear up
+                    utils.remove_container(container)
+
+        elif run_type == "search_performance":
+            for op_type, op_value in definition.items():
+                # run docker mode
+                run_count = op_value["run_count"]
+                run_params = op_value["params"]
+                container = None
+                for index, param in enumerate(run_params):
+                    logger.info("Definition param: %s" % str(param))
+                    table_name = param["dataset"]
+                    volume_name = param["db_path_prefix"]
+                    (data_type, table_size, index_file_size, dimension, metric_type) = parser.table_parser(table_name)
+                    for k, v in param.items():
+                        if k.startswith("server."):                   
+                            utils.modify_config(k, v, type="server")
+                    container = utils.run_server(self.image, test_type="remote", volume_name=volume_name, db_slave=None)
+                    time.sleep(2)
+                    milvus = MilvusClient(table_name)
+                    logger.debug(milvus.show_tables())
+                    # Check has table or not
+                    if not milvus.exists_table():
+                        logger.warning("Table %s not existed, continue exec next params ..." % table_name)
+                        continue
+                    # parse index info
+                    index_types = param["index.index_types"]
+                    nlists = param["index.nlists"]
+                    # parse top-k, nq, nprobe
+                    top_ks, nqs, nprobes = parser.search_params_parser(param)
+                    for index_type in index_types:
+                        for nlist in nlists:
+                            result = milvus.describe_index()
+                            logger.info(result)
+                            # milvus.drop_index()
+                            # milvus.create_index(index_type, nlist)
+                            result = milvus.describe_index()
+                            logger.info(result)
+                            logger.info(milvus.count())
+                            # preload index
+                            milvus.preload_table()
+                            logger.info("Start warm up query")
+                            res = self.do_query(milvus, table_name, [1], [1], 1, 1)
+                            logger.info("End warm up query")
+                            # Run query test
+                            for nprobe in nprobes:
+                                logger.info("index_type: %s, nlist: %s, metric_type: %s, nprobe: %s" % (index_type, nlist, metric_type, nprobe))
+                                res = self.do_query(milvus, table_name, top_ks, nqs, nprobe, run_count)
+                                headers = ["Nq/Top-k"]
+                                headers.extend([str(top_k) for top_k in top_ks])
+                                utils.print_table(headers, nqs, res)
+                    utils.remove_container(container)
+
         elif run_type == "accuracy":
             """
             {
@@ -149,11 +234,9 @@ class DockerRunner(Runner):
                     nlists = param["index.nlists"]
                     # parse top-k, nq, nprobe
                     top_ks, nqs, nprobes = parser.search_params_parser(param)
-
                     if sift_acc is True:
                         # preload groundtruth data
                         true_ids_all = self.get_groundtruth_ids(table_size)
-
                     acc_dict = {}
                     for index_type in index_types:
                         for nlist in nlists:

tests/milvus_benchmark/k8s_runner.py

@@ -0,0 +1,473 @@
+import os
+import logging
+import pdb
+import time
+import re
+import random
+import traceback
+from multiprocessing import Process
+import numpy as np
+from client import MilvusClient
+import utils
+import parser
+from runner import Runner
+from milvus_metrics.api import report
+from milvus_metrics.models import Env, Hardware, Server, Metric
+import utils
+
+logger = logging.getLogger("milvus_benchmark.k8s_runner")
+namespace = "milvus"
+DELETE_INTERVAL_TIME = 5
+# INSERT_INTERVAL = 100000
+INSERT_INTERVAL = 50000
+timestamp = int(time.time())
+default_path = "/var/lib/milvus"
+
+class K8sRunner(Runner):
+    """run docker mode"""
+    def __init__(self):
+        super(K8sRunner, self).__init__()
+        self.name = utils.get_unique_name()
+        self.host = None
+        self.ip = None
+        self.hostname = None
+        self.env_value = None
+        
+    def init_env(self, server_config, args):
+        self.hostname = args.hostname
+        # update server_config
+        helm_path = os.path.join(os.getcwd(), "../milvus-helm/milvus")
+        server_config_file = helm_path+"/ci/config/sqlite/%s/server_config.yaml" % (args.image_type)
+        if not os.path.exists(server_config_file):
+            raise Exception("File %s not existed" % server_config_file)
+        if server_config:
+            logger.debug("Update server config")
+            utils.update_server_config(server_config_file, server_config)
+        # update log_config
+        log_config_file = helm_path+"/config/log_config.conf"
+        if not os.path.exists(log_config_file):
+            raise Exception("File %s not existed" % log_config_file)
+        src_log_config_file = helm_path+"/config/log_config.conf.src"
+        if not os.path.exists(src_log_config_file):
+            # copy
+            os.system("cp %s %s" % (log_config_file, src_log_config_file))
+        else:
+            # reset
+            os.system("cp %s %s" % (src_log_config_file, log_config_file))
+        if "db_config.primary_path" in server_config:
+            os.system("sed -i 's#%s#%s#g' %s" % (default_path, server_config["db_config.primary_path"], log_config_file))
+        
+        # with open(log_config_file, "r+") as fd:
+        #     for line in fd.readlines():
+        #         fd.write(re.sub(r'^%s' % default_path, server_config["db_config.primary_path"], line))
+        # update values
+        values_file_path = helm_path+"/values.yaml"
+        if not os.path.exists(values_file_path):
+            raise Exception("File %s not existed" % values_file_path)
+        utils.update_values(values_file_path, args.hostname)
+        try:
+            logger.debug("Start install server")
+            self.host, self.ip = utils.helm_install_server(helm_path, args.image_tag, args.image_type, self.name, namespace)
+        except Exception as e:
+            logger.error("Helm install server failed: %s" % str(e))
+            logger.error(traceback.format_exc())
+            self.clean_up()
+            return False
+        # for debugging
+        # self.host = "192.168.1.101"
+        if not self.host:
+            logger.error("Helm install server failed")
+            self.clean_up()
+            return False
+        return True
+
+    def clean_up(self):
+        logger.debug(self.name)
+        utils.helm_del_server(self.name)
+
+    def report_wrapper(self, milvus_instance, env_value, hostname, table_info, index_info, search_params):
+        metric = Metric()
+        metric.set_run_id(timestamp)
+        metric.env = Env(env_value)
+        metric.env.OMP_NUM_THREADS = 0
+        metric.hardware = Hardware(name=hostname)
+        server_version = milvus_instance.get_server_version()
+        server_mode = milvus_instance.get_server_mode()
+        commit = milvus_instance.get_server_commit()
+        metric.server = Server(version=server_version, mode=server_mode, build_commit=commit)
+        metric.table = table_info
+        metric.index = index_info
+        metric.search = search_params
+        return metric
+
+    def run(self, run_type, table):
+        logger.debug(run_type)
+        logger.debug(table)
+        table_name = table["table_name"]
+        milvus_instance = MilvusClient(table_name=table_name, ip=self.ip)
+        self.env_value = milvus_instance.get_server_config()
+        if run_type == "insert_performance":
+            (data_type, table_size, index_file_size, dimension, metric_type) = parser.table_parser(table_name)
+            ni_per = table["ni_per"]
+            build_index = table["build_index"]
+            if milvus_instance.exists_table():
+                milvus_instance.delete()
+                time.sleep(10)
+            index_info = {}
+            search_params = {}
+            milvus_instance.create_table(table_name, dimension, index_file_size, metric_type)
+            if build_index is True:
+                index_type = table["index_type"]
+                nlist = table["nlist"]
+                index_info = {
+                    "index_type": index_type,
+                    "index_nlist": nlist
+                }
+                milvus_instance.create_index(index_type, nlist)
+            res = self.do_insert(milvus_instance, table_name, data_type, dimension, table_size, ni_per)
+            logger.info(res)
+            table_info = {
+                "dimension": dimension,
+                "metric_type": metric_type,
+                "dataset_name": table_name
+            }
+            metric = self.report_wrapper(milvus_instance, self.env_value, self.hostname, table_info, index_info, search_params)
+            metric.metrics = {
+                "type": "insert_performance",
+                "value": {
+                    "total_time": res["total_time"],
+                    "qps": res["qps"],
+                    "ni_time": res["ni_time"]
+                } 
+            }
+            report(metric)
+            logger.debug("Wait for file merge")
+            time.sleep(120)
+
+        elif run_type == "build_performance":
+            (data_type, table_size, index_file_size, dimension, metric_type) = parser.table_parser(table_name)
+            index_type = table["index_type"]
+            nlist = table["nlist"]
+            table_info = {
+                "dimension": dimension,
+                "metric_type": metric_type,
+                "dataset_name": table_name
+            }
+            index_info = {
+                "index_type": index_type,
+                "index_nlist": nlist
+            }
+            if not milvus_instance.exists_table():
+                logger.error("Table name: %s not existed" % table_name)
+                return
+            search_params = {}
+            start_time = time.time()
+            # drop index
+            logger.debug("Drop index")
+            milvus_instance.drop_index()
+            start_mem_usage = milvus_instance.get_mem_info()["memory_used"]
+            milvus_instance.create_index(index_type, nlist)
+            logger.debug(milvus_instance.describe_index())
+            end_time = time.time()
+            end_mem_usage = milvus_instance.get_mem_info()["memory_used"]
+            metric = self.report_wrapper(milvus_instance, self.env_value, self.hostname, table_info, index_info, search_params)
+            metric.metrics = {
+                "type": "build_performance",
+                "value": {
+                    "build_time": round(end_time - start_time, 1),
+                    "start_mem_usage": start_mem_usage,
+                    "end_mem_usage": end_mem_usage,
+                    "diff_mem": end_mem_usage - start_mem_usage
+                } 
+            }
+            report(metric)
+
+        elif run_type == "search_performance":
+            (data_type, table_size, index_file_size, dimension, metric_type) = parser.table_parser(table_name)
+            run_count = table["run_count"]
+            search_params = table["search_params"]
+            table_info = {
+                "dimension": dimension,
+                "metric_type": metric_type,
+                "dataset_name": table_name
+            }
+            if not milvus_instance.exists_table():
+                logger.error("Table name: %s not existed" % table_name)
+                return
+            logger.info(milvus_instance.count())
+            result = milvus_instance.describe_index()
+            index_info = {
+                "index_type": result["index_type"],
+                "index_nlist": result["nlist"]
+            }
+            logger.info(index_info)
+            nprobes = search_params["nprobes"]
+            top_ks = search_params["top_ks"]
+            nqs = search_params["nqs"]
+            milvus_instance.preload_table()
+            logger.info("Start warm up query")
+            res = self.do_query(milvus_instance, table_name, [1], [1], 1, 2)
+            logger.info("End warm up query")
+            for nprobe in nprobes:
+                logger.info("Search nprobe: %s" % nprobe)
+                res = self.do_query(milvus_instance, table_name, top_ks, nqs, nprobe, run_count)
+                headers = ["Nq/Top-k"]
+                headers.extend([str(top_k) for top_k in top_ks])
+                utils.print_table(headers, nqs, res)
+                for index_nq, nq in enumerate(nqs):
+                    for index_top_k, top_k in enumerate(top_ks):
+                        search_param = {
+                            "nprobe": nprobe,
+                            "nq": nq,
+                            "topk": top_k
+                        }
+                        search_time = res[index_nq][index_top_k]
+                        metric = self.report_wrapper(milvus_instance, self.env_value, self.hostname, table_info, index_info, search_param)
+                        metric.metrics = {
+                            "type": "search_performance",
+                            "value": {
+                                "search_time": search_time
+                            } 
+                        }
+                        report(metric)
+
+        # for sift/deep datasets
+        elif run_type == "accuracy":
+            (data_type, table_size, index_file_size, dimension, metric_type) = parser.table_parser(table_name)
+            search_params = table["search_params"]
+            table_info = {
+                "dimension": dimension,
+                "metric_type": metric_type,
+                "dataset_name": table_name
+            }
+            if not milvus_instance.exists_table():
+                logger.error("Table name: %s not existed" % table_name)
+                return
+            logger.info(milvus_instance.count())
+            result = milvus_instance.describe_index()
+            index_info = {
+                "index_type": result["index_type"],
+                "index_nlist": result["nlist"]
+            }
+            logger.info(index_info)
+            nprobes = search_params["nprobes"]
+            top_ks = search_params["top_ks"]
+            nqs = search_params["nqs"]
+            milvus_instance.preload_table()
+            true_ids_all = self.get_groundtruth_ids(table_size)
+            for nprobe in nprobes:
+                logger.info("Search nprobe: %s" % nprobe)
+                for top_k in top_ks:
+                    for nq in nqs:
+                        total = 0
+                        search_param = {
+                            "nprobe": nprobe,
+                            "nq": nq,
+                            "topk": top_k
+                        }
+                        result_ids, result_distances = self.do_query_ids(milvus_instance, table_name, top_k, nq, nprobe)
+                        acc_value = self.get_recall_value(true_ids_all[:nq, :top_k].tolist(), result_ids)
+                        logger.info("Query accuracy: %s" % acc_value)
+                        metric = self.report_wrapper(milvus_instance, self.env_value, self.hostname, table_info, index_info, search_param)
+                        metric.metrics = {
+                            "type": "accuracy",
+                            "value": {
+                                "acc": acc_value
+                            } 
+                        }
+                        report(metric)
+
+        elif run_type == "ann_accuracy":
+            hdf5_source_file = table["source_file"]
+            table_name = table["table_name"]
+            index_file_sizes = table["index_file_sizes"]
+            index_types = table["index_types"]
+            nlists = table["nlists"]
+            search_params = table["search_params"]
+            nprobes = search_params["nprobes"]
+            top_ks = search_params["top_ks"]
+            nqs = search_params["nqs"]
+            data_type, dimension, metric_type = parser.parse_ann_table_name(table_name)
+            table_info = {
+                "dimension": dimension,
+                "metric_type": metric_type,
+                "dataset_name": table_name
+            }
+            dataset = utils.get_dataset(hdf5_source_file)
+            if milvus_instance.exists_table(table_name):
+                logger.info("Re-create table: %s" % table_name)
+                milvus_instance.delete(table_name)
+                time.sleep(DELETE_INTERVAL_TIME)
+            true_ids = np.array(dataset["neighbors"])
+            for index_file_size in index_file_sizes:
+                milvus_instance.create_table(table_name, dimension, index_file_size, metric_type)
+                logger.info(milvus_instance.describe())
+                insert_vectors = self.normalize(metric_type, np.array(dataset["train"]))
+                # Insert batch once
+                # milvus_instance.insert(insert_vectors)
+                loops = len(insert_vectors) // INSERT_INTERVAL + 1
+                for i in range(loops):
+                    start = i*INSERT_INTERVAL
+                    end = min((i+1)*INSERT_INTERVAL, len(insert_vectors))
+                    tmp_vectors = insert_vectors[start:end]
+                    if start < end:
+                        if not isinstance(tmp_vectors, list):
+                            milvus_instance.insert(tmp_vectors.tolist(), ids=[i for i in range(start, end)])
+                        else:
+                            milvus_instance.insert(tmp_vectors, ids=[i for i in range(start, end)])
+                time.sleep(20)
+                logger.info("Table: %s, row count: %s" % (table_name, milvus_instance.count()))
+                if milvus_instance.count() != len(insert_vectors):
+                    logger.error("Table row count is not equal to insert vectors")
+                    return
+                for index_type in index_types:
+                    for nlist in nlists:
+                        milvus_instance.create_index(index_type, nlist)
+                        # logger.info(milvus_instance.describe_index())
+                        logger.info("Start preload table: %s, index_type: %s, nlist: %s" % (table_name, index_type, nlist))
+                        milvus_instance.preload_table()
+                        index_info = {
+                            "index_type": index_type,
+                            "index_nlist": nlist
+                        }
+                        for nprobe in nprobes:
+                            for nq in nqs:
+                                query_vectors = self.normalize(metric_type, np.array(dataset["test"][:nq]))
+                                for top_k in top_ks:
+                                    search_params = {
+                                        "nq": len(query_vectors),
+                                        "nprobe": nprobe,
+                                        "topk": top_k
+                                    }
+                                    if not isinstance(query_vectors, list):
+                                        result = milvus_instance.query(query_vectors.tolist(), top_k, nprobe)
+                                    else:
+                                        result = milvus_instance.query(query_vectors, top_k, nprobe)
+                                    result_ids = result.id_array
+                                    acc_value = self.get_recall_value(true_ids[:nq, :top_k].tolist(), result_ids)
+                                    logger.info("Query ann_accuracy: %s" % acc_value)
+                                    metric = self.report_wrapper(milvus_instance, self.env_value, self.hostname, table_info, index_info, search_params)
+                                    metric.metrics = {
+                                        "type": "ann_accuracy",
+                                        "value": {
+                                            "acc": acc_value
+                                        } 
+                                    }
+                                    report(metric)
+                milvus_instance.delete()
+
+        elif run_type == "search_stability":
+            (data_type, table_size, index_file_size, dimension, metric_type) = parser.table_parser(table_name)
+            search_params = table["search_params"]
+            during_time = table["during_time"]
+            table_info = {
+                "dimension": dimension,
+                "metric_type": metric_type,
+                "dataset_name": table_name
+            }
+            if not milvus_instance.exists_table():
+                logger.error("Table name: %s not existed" % table_name)
+                return
+            logger.info(milvus_instance.count())
+            result = milvus_instance.describe_index()
+            index_info = {
+                "index_type": result["index_type"],
+                "index_nlist": result["nlist"]
+            }
+            search_param = {}
+            logger.info(index_info)
+            g_nprobe = int(search_params["nprobes"].split("-")[1])
+            g_top_k = int(search_params["top_ks"].split("-")[1])
+            g_nq = int(search_params["nqs"].split("-")[1])
+            l_nprobe = int(search_params["nprobes"].split("-")[0])
+            l_top_k = int(search_params["top_ks"].split("-")[0])
+            l_nq = int(search_params["nqs"].split("-")[0])
+            milvus_instance.preload_table()
+            start_mem_usage = milvus_instance.get_mem_info()["memory_used"]
+            logger.debug(start_mem_usage)
+            logger.info("Start warm up query")
+            res = self.do_query(milvus_instance, table_name, [1], [1], 1, 2)
+            logger.info("End warm up query")
+            start_time = time.time()
+            while time.time() < start_time + during_time * 60:
+                top_k = random.randint(l_top_k, g_top_k)
+                nq = random.randint(l_nq, g_nq)
+                nprobe = random.randint(l_nprobe, g_nprobe)
+                query_vectors = [[random.random() for _ in range(dimension)] for _ in range(nq)]
+                logger.debug("Query nprobe:%d, nq:%d, top-k:%d" % (nprobe, nq, top_k))
+                result = milvus_instance.query(query_vectors, top_k, nprobe)
+            end_mem_usage = milvus_instance.get_mem_info()["memory_used"]
+            metric = self.report_wrapper(milvus_instance, self.env_value, self.hostname, table_info, index_info, search_param)
+            metric.metrics = {
+                "type": "search_stability",
+                "value": {
+                    "during_time": during_time,
+                    "start_mem_usage": start_mem_usage,
+                    "end_mem_usage": end_mem_usage,
+                    "diff_mem": end_mem_usage - start_mem_usage
+                } 
+            }
+            report(metric)
+
+        elif run_type == "stability":
+            (data_type, table_size, index_file_size, dimension, metric_type) = parser.table_parser(table_name)
+            search_params = table["search_params"]
+            insert_xb = table["insert_xb"]
+            insert_interval = table["insert_interval"]
+            during_time = table["during_time"]
+            table_info = {
+                "dimension": dimension,
+                "metric_type": metric_type,
+                "dataset_name": table_name
+            }
+            if not milvus_instance.exists_table():
+                logger.error("Table name: %s not existed" % table_name)
+                return
+            logger.info(milvus_instance.count())
+            result = milvus_instance.describe_index()
+            index_info = {
+                "index_type": result["index_type"],
+                "index_nlist": result["nlist"]
+            }
+            search_param = {}
+            logger.info(index_info)
+            g_nprobe = int(search_params["nprobes"].split("-")[1])
+            g_top_k = int(search_params["top_ks"].split("-")[1])
+            g_nq = int(search_params["nqs"].split("-")[1])
+            l_nprobe = int(search_params["nprobes"].split("-")[0])
+            l_top_k = int(search_params["top_ks"].split("-")[0])
+            l_nq = int(search_params["nqs"].split("-")[0])
+            milvus_instance.preload_table()
+            logger.info("Start warm up query")
+            res = self.do_query(milvus_instance, table_name, [1], [1], 1, 2)
+            logger.info("End warm up query")
+            start_mem_usage = milvus_instance.get_mem_info()["memory_used"]
+            start_row_count = milvus_instance.count()
+            start_time = time.time()
+            i = 0
+            while time.time() < start_time + during_time * 60:
+                i = i + 1
+                for j in range(insert_interval):
+                    top_k = random.randint(l_top_k, g_top_k)
+                    nq = random.randint(l_nq, g_nq)
+                    nprobe = random.randint(l_nprobe, g_nprobe)
+                    query_vectors = [[random.random() for _ in range(dimension)] for _ in range(nq)]
+                    logger.debug("Query nprobe:%d, nq:%d, top-k:%d" % (nprobe, nq, top_k))
+                    result = milvus_instance.query(query_vectors, top_k, nprobe)
+                insert_vectors = [[random.random() for _ in range(dimension)] for _ in range(insert_xb)]
+                status, res = milvus_instance.insert(insert_vectors, ids=[x for x in range(len(insert_vectors))])
+                logger.debug("%d, row_count: %d" % (i, milvus_instance.count()))
+            end_mem_usage = milvus_instance.get_mem_info()["memory_used"]
+            end_row_count = milvus_instance.count()
+            metric = self.report_wrapper(milvus_instance, self.env_value, self.hostname, table_info, index_info, search_param)
+            metric.metrics = {
+                "type": "stability",
+                "value": {
+                    "during_time": during_time,
+                    "start_mem_usage": start_mem_usage,
+                    "end_mem_usage": end_mem_usage,
+                    "diff_mem": end_mem_usage - start_mem_usage,
+                    "row_count_increments": end_row_count - start_row_count
+                } 
+            }
+            report(metric)

tests/milvus_benchmark/local_runner.py

@@ -48,6 +48,8 @@ class LocalRunner(Runner):
                         milvus = MilvusClient(table_name, ip=self.ip, port=self.port)
                         logger.info(milvus.describe())
                         logger.info(milvus.describe_index())
+                        logger.info(milvus.count())
+                        logger.info(milvus.show_tables())
                         # parse index info
                         index_types = param["index.index_types"]
                         nlists = param["index.nlists"]
@@ -64,7 +66,7 @@ class LocalRunner(Runner):
                                 logger.info("End preloading table")
                                 # Run query test
                                 logger.info("Start warm up query")
-                                res = self.do_query(milvus, table_name, [1], [1], 1, 1)
+                                res = self.do_query(milvus, table_name, [1], [1], 1, 2)
                                 logger.info("End warm up query")
                                 for nprobe in nprobes:
                                     logger.info("index_type: %s, nlist: %s, metric_type: %s, nprobe: %s" % (index_type, nlist, metric_type, nprobe))
@@ -204,12 +206,14 @@ class LocalRunner(Runner):
                         #     p.join()
                         i = i + 1
                         milvus_instance = MilvusClient(table_name, ip=self.ip, port=self.port)
-                        top_ks = random.sample([x for x in range(1, 100)], 4)
-                        nqs = random.sample([x for x in range(1, 200)], 3)
+                        top_ks = random.sample([x for x in range(1, 100)], 1)
+                        nqs = random.sample([x for x in range(1, 200)], 2)
                         nprobe = random.choice([x for x in range(1, 100)])
                         res = self.do_query(milvus_instance, table_name, top_ks, nqs, nprobe, run_count)
                         # milvus_instance = MilvusClient(table_name)
                         status, res = milvus_instance.insert(insert_vectors, ids=[x for x in range(len(insert_vectors))])
                         if not status.OK():
                             logger.error(status.message)
+                        logger.debug(milvus.count())
+                        res = self.do_query(milvus_instance, table_name, top_ks, nqs, nprobe, run_count)
                         # status = milvus_instance.create_index(index_type, 16384)

tests/milvus_benchmark/main.py

@@ -4,26 +4,31 @@ import time
 import pdb
 import argparse
 import logging
-import utils
-from yaml import load, dump
+import traceback
 from logging import handlers
+
+from yaml import full_load, dump
 from parser import operations_parser
 from local_runner import LocalRunner
 from docker_runner import DockerRunner
+from k8s_runner import K8sRunner
 
 DEFAULT_IMAGE = "milvusdb/milvus:latest"
-LOG_FOLDER = "benchmark_logs"
-logger = logging.getLogger("milvus_benchmark")
+LOG_FOLDER = "logs"
 
-formatter = logging.Formatter('[%(asctime)s] [%(levelname)-4s] [%(pathname)s:%(lineno)d] %(message)s')
-if not os.path.exists(LOG_FOLDER):
-    os.system('mkdir -p %s' % LOG_FOLDER)
-fileTimeHandler = handlers.TimedRotatingFileHandler(os.path.join(LOG_FOLDER, 'milvus_benchmark'), "D", 1, 10)
-fileTimeHandler.suffix = "%Y%m%d.log"
-fileTimeHandler.setFormatter(formatter)
-logging.basicConfig(level=logging.DEBUG)
-fileTimeHandler.setFormatter(formatter)
-logger.addHandler(fileTimeHandler)
+# formatter = logging.Formatter('[%(asctime)s] [%(levelname)-4s] [%(pathname)s:%(lineno)d] %(message)s')
+# if not os.path.exists(LOG_FOLDER):
+#     os.system('mkdir -p %s' % LOG_FOLDER)
+# fileTimeHandler = handlers.TimedRotatingFileHandler(os.path.join(LOG_FOLDER, 'milvus_benchmark'), "D", 1, 10)
+# fileTimeHandler.suffix = "%Y%m%d.log"
+# fileTimeHandler.setFormatter(formatter)
+# logging.basicConfig(level=logging.DEBUG)
+# fileTimeHandler.setFormatter(formatter)
+# logger.addHandler(fileTimeHandler)
+logging.basicConfig(format='%(asctime)s,%(msecs)d %(levelname)-8s [%(filename)s:%(lineno)d] %(message)s',
+    datefmt='%Y-%m-%d:%H:%M:%S',
+    level=logging.DEBUG)
+logger = logging.getLogger("milvus_benchmark")
 
 
 def positive_int(s):
@@ -51,30 +56,39 @@ def main():
     parser = argparse.ArgumentParser(
         formatter_class=argparse.ArgumentDefaultsHelpFormatter)
     parser.add_argument(
-        '--image',
-        help='use the given image')
+        "--hostname",
+        default="eros",
+        help="server host name")
+    parser.add_argument(
+        "--image-tag",
+        default="",
+        help="image tag")
+    parser.add_argument(
+        "--image-type",
+        default="",
+        help="image type")
+    # parser.add_argument(
+    #     "--run-count",
+    #     default=1,
+    #     type=positive_int,
+    #     help="run times for each test")
+    # # performance / stability / accuracy test
+    # parser.add_argument(
+    #     "--run-type",
+    #     default="search_performance",
+    #     help="run type, default performance")
+    parser.add_argument(
+        '--suite',
+        metavar='FILE',
+        help='load test suite from FILE',
+        default='suites/suite.yaml')
     parser.add_argument(
         '--local',
         action='store_true',
         help='use local milvus server')
     parser.add_argument(
-        "--run-count",
-        default=1,
-        type=positive_int,
-        help="run each db operation times")
-    # performance / stability / accuracy test
-    parser.add_argument(
-        "--run-type",
-        default="performance",
-        help="run type, default performance")
-    parser.add_argument(
-        '--suites',
-        metavar='FILE',
-        help='load test suites from FILE',
-        default='suites.yaml')
-    parser.add_argument(
-        '--ip',
-        help='server ip param for local mode',
+        '--host',
+        help='server host ip param for local mode',
         default='127.0.0.1')
     parser.add_argument(
         '--port',
@@ -83,43 +97,60 @@ def main():
 
     args = parser.parse_args()
 
-    operations = None
     # Get all benchmark test suites
-    if args.suites:
-        with open(args.suites) as f:
-            suites_dict = load(f)
+    if args.suite:
+        with open(args.suite) as f:
+            suite_dict = full_load(f)
             f.close()
         # With definition order
-        operations = operations_parser(suites_dict, run_type=args.run_type)
+        run_type, run_params = operations_parser(suite_dict)
 
     # init_env()
-    run_params = {"run_count": args.run_count}
+    # run_params = {"run_count": args.run_count}
 
-    if args.image:
+    if args.image_tag:
+        namespace = "milvus"
+        logger.debug(args)
         # for docker mode
         if args.local:
-            logger.error("Local mode and docker mode are incompatible arguments")
+            logger.error("Local mode and docker mode are incompatible")
             sys.exit(-1)
         # Docker pull image
-        if not utils.pull_image(args.image):
-            raise Exception('Image %s pull failed' % image)
-
+        # if not utils.pull_image(args.image):
+        #     raise Exception('Image %s pull failed' % image)
         # TODO: Check milvus server port is available
-        logger.info("Init: remove all containers created with image: %s" % args.image)
-        utils.remove_all_containers(args.image)
-        runner = DockerRunner(args.image)
-        for operation_type in operations:
-            logger.info("Start run test, test type: %s" % operation_type)
-            run_params["params"] = operations[operation_type]
-            runner.run({operation_type: run_params}, run_type=args.run_type)
-            logger.info("Run params: %s" % str(run_params))
+        # logger.info("Init: remove all containers created with image: %s" % args.image)
+        # utils.remove_all_containers(args.image)
+        # runner = DockerRunner(args)
+        tables = run_params["tables"]
+        for table in tables:
+            # run tests
+            server_config = table["server"]
+            logger.debug(server_config)
+            runner = K8sRunner()
+            if runner.init_env(server_config, args):
+                logger.debug("Start run tests")
+                try:
+                    runner.run(run_type, table)
+                except Exception as e:
+                    logger.error(str(e))
+                    logger.error(traceback.format_exc())
+                finally:
+                    runner.clean_up()
+            else:
+                logger.error("Runner init failed")
+        # for operation_type in operations:
+        #     logger.info("Start run test, test type: %s" % operation_type)
+        #     run_params["params"] = operations[operation_type]
+        #     runner.run({operation_type: run_params}, run_type=args.run_type)
+        #     logger.info("Run params: %s" % str(run_params))
 
     if args.local:
         # for local mode
-        ip = args.ip
+        host = args.host
         port = args.port
 
-        runner = LocalRunner(ip, port)
+        runner = LocalRunner(host, port)
         for operation_type in operations:
             logger.info("Start run local mode test, test type: %s" % operation_type)
             run_params["params"] = operations[operation_type]

tests/milvus_benchmark/parser.py

@@ -4,9 +4,12 @@ import logging
 logger = logging.getLogger("milvus_benchmark.parser")
 
 
-def operations_parser(operations, run_type="performance"):
-    definitions = operations[run_type]
-    return definitions
+def operations_parser(operations):
+    if not operations:
+        raise Exception("No operations in suite defined")
+    for run_type, run_params in operations.items():
+        logger.debug(run_type)
+        return (run_type, run_params)
 
 
 def table_parser(table_name):
@@ -26,6 +29,23 @@ def table_parser(table_name):
     return (data_type, table_size, index_file_size, dimension, metric_type)
 
 
+def parse_ann_table_name(table_name):
+    data_type = table_name.split("_")[0]
+    dimension = int(table_name.split("_")[1])
+    metric = table_name.split("_")[-1]
+    # metric = table_name.attrs['distance']
+    # dimension = len(table_name["train"][0])
+    if metric == "euclidean":
+        metric_type = "l2"
+    elif metric  == "angular":
+        metric_type = "ip"
+    elif metric  == "jaccard":
+        metric_type = "jaccard"
+    elif metric == "hamming":
+        metric_type = "hamming"
+    return ("ann_"+data_type, dimension, metric_type)
+
+
 def search_params_parser(param):
     # parse top-k, set default value if top-k not in param
     if "top_ks" not in param:

tests/milvus_benchmark/requirements.txt

@@ -2,8 +2,9 @@ numpy==1.16.3
 pymilvus-test>=0.2.0
 scikit-learn==0.19.1
 h5py==2.7.1
-influxdb==5.2.2
+# influxdb==5.2.2
 pyyaml>=5.1
 tableprint==0.8.0
 ansicolors==1.1.8
 scipy==1.3.1
+kubernetes==10.0.1

tests/milvus_benchmark/runner.py

@@ -5,24 +5,26 @@ import time
 import random
 from multiprocessing import Process
 import numpy as np
+import sklearn.preprocessing
 from client import MilvusClient
 import utils
 import parser
 
 logger = logging.getLogger("milvus_benchmark.runner")
 
-SERVER_HOST_DEFAULT = "127.0.0.1"
-SERVER_PORT_DEFAULT = 19530
 VECTORS_PER_FILE = 1000000
 SIFT_VECTORS_PER_FILE = 100000
+JACCARD_VECTORS_PER_FILE = 2000000
+
 MAX_NQ = 10001
 FILE_PREFIX = "binary_"
 
 # FOLDER_NAME = 'ann_1000m/source_data'
-SRC_BINARY_DATA_DIR = '/poc/yuncong/yunfeng/random_data'
-SRC_BINARY_DATA_DIR_high = '/test/milvus/raw_data/random'
-SIFT_SRC_DATA_DIR = '/poc/yuncong/ann_1000m/'
-SIFT_SRC_BINARY_DATA_DIR = SIFT_SRC_DATA_DIR + 'source_data'
+SRC_BINARY_DATA_DIR = '/test/milvus/raw_data/random/'
+SIFT_SRC_DATA_DIR = '/test/milvus/raw_data/sift1b/'
+DEEP_SRC_DATA_DIR = '/test/milvus/raw_data/deep1b/'
+JACCARD_SRC_DATA_DIR = '/test/milvus/raw_data/jaccard/'
+HAMMING_SRC_DATA_DIR = '/test/milvus/raw_data/jaccard/'
 SIFT_SRC_GROUNDTRUTH_DATA_DIR = SIFT_SRC_DATA_DIR + 'gnd'
 
 WARM_TOP_K = 1
@@ -44,16 +46,19 @@ GROUNDTRUTH_MAP = {
 }
 
 
-def gen_file_name(idx, table_dimension, data_type):
+def gen_file_name(idx, dimension, data_type):
     s = "%05d" % idx
-    fname = FILE_PREFIX + str(table_dimension) + "d_" + s + ".npy"
+    fname = FILE_PREFIX + str(dimension) + "d_" + s + ".npy"
     if data_type == "random":
-        if table_dimension == 512:
-            fname = SRC_BINARY_DATA_DIR+'/'+fname
-        elif table_dimension >= 4096:
-            fname = SRC_BINARY_DATA_DIR_high+'/'+fname
+        fname = SRC_BINARY_DATA_DIR+fname
     elif data_type == "sift":
-        fname = SIFT_SRC_BINARY_DATA_DIR+'/'+fname
+        fname = SIFT_SRC_DATA_DIR+fname
+    elif data_type == "deep":
+        fname = DEEP_SRC_DATA_DIR+fname
+    elif data_type == "jaccard":
+        fname = JACCARD_SRC_DATA_DIR+fname
+    elif data_type == "hamming":
+        fname = HAMMING_SRC_DATA_DIR+fname
     return fname
 
 
@@ -62,9 +67,15 @@ def get_vectors_from_binary(nq, dimension, data_type):
     if nq > MAX_NQ:
         raise Exception("Over size nq")
     if data_type == "random":
-        file_name = gen_file_name(0, dimension, data_type)
+        file_name = SRC_BINARY_DATA_DIR+'query_%d.npy' % dimension
     elif data_type == "sift":
-        file_name = SIFT_SRC_DATA_DIR+'/'+'query.npy'
+        file_name = SIFT_SRC_DATA_DIR+'query.npy'
+    elif data_type == "deep":
+        file_name = DEEP_SRC_DATA_DIR+'query.npy'
+    elif data_type == "jaccard":
+        file_name = JACCARD_SRC_DATA_DIR+'query.npy'
+    elif data_type == "hamming":
+        file_name = HAMMING_SRC_DATA_DIR+'query.npy'
     data = np.load(file_name)
     vectors = data[0:nq].tolist()
     return vectors
@@ -74,6 +85,21 @@ class Runner(object):
     def __init__(self):
         pass
 
+    def normalize(self, metric_type, X):
+        if metric_type == "ip":
+            logger.info("Set normalize for metric_type: %s" % metric_type)
+            X = sklearn.preprocessing.normalize(X, axis=1, norm='l2')
+            X = X.astype(np.float32)
+        elif metric_type == "l2":
+            X = X.astype(np.float32)
+        elif metric_type == "jaccard" or metric_type == "hamming":
+            tmp = []
+            for index, item in enumerate(X):
+                new_vector = bytes(np.packbits(item, axis=-1).tolist())
+                tmp.append(new_vector)
+            X = tmp
+        return X
+
     def do_insert(self, milvus, table_name, data_type, dimension, size, ni):
         '''
         @params:
@@ -101,14 +127,18 @@ class Runner(object):
                 vectors_per_file = 10000
         elif data_type == "sift":
             vectors_per_file = SIFT_VECTORS_PER_FILE
+        elif data_type == "jaccard" or data_type == "hamming":
+            vectors_per_file = JACCARD_VECTORS_PER_FILE
+        else:
+            raise Exception("data_type: %s not supported" % data_type)
         if size % vectors_per_file or ni > vectors_per_file:
             raise Exception("Not invalid table size or ni")
         file_num = size // vectors_per_file
         for i in range(file_num):
             file_name = gen_file_name(i, dimension, data_type)
-            logger.info("Load npy file: %s start" % file_name)
+            # logger.info("Load npy file: %s start" % file_name)
             data = np.load(file_name)
-            logger.info("Load npy file: %s end" % file_name)
+            # logger.info("Load npy file: %s end" % file_name)
             loops = vectors_per_file // ni
             for j in range(loops):
                 vectors = data[j*ni:(j+1)*ni].tolist()
@@ -130,28 +160,26 @@ class Runner(object):
         bi_res["ni_time"] = ni_time
         return bi_res
 
-    def do_query(self, milvus, table_name, top_ks, nqs, nprobe, run_count):
+    def do_query(self, milvus, table_name, top_ks, nqs, nprobe, run_count=1):
+        bi_res = []
         (data_type, table_size, index_file_size, dimension, metric_type) = parser.table_parser(table_name)
         base_query_vectors = get_vectors_from_binary(MAX_NQ, dimension, data_type)
-
-        bi_res = []
-        for index, nq in enumerate(nqs):
+        for nq in nqs:
             tmp_res = []
             vectors = base_query_vectors[0:nq]
             for top_k in top_ks:
                 avg_query_time = 0.0
-                total_query_time = 0.0
+                min_query_time = 0.0
                 logger.info("Start query, query params: top-k: {}, nq: {}, actually length of vectors: {}".format(top_k, nq, len(vectors)))
                 for i in range(run_count):
                     logger.info("Start run query, run %d of %s" % (i+1, run_count))
                     start_time = time.time()
-                    status, query_res = milvus.query(vectors, top_k, nprobe)
-                    total_query_time = total_query_time + (time.time() - start_time)
-                    if status.code:
-                        logger.error("Query failed with message: %s" % status.message)
-                avg_query_time = round(total_query_time / run_count, 2)
-                logger.info("Avarage query time: %.2f" % avg_query_time)
-                tmp_res.append(avg_query_time)
+                    query_res = milvus.query(vectors, top_k, nprobe)
+                    interval_time = time.time() - start_time
+                    if (i == 0) or (min_query_time > interval_time):
+                        min_query_time = interval_time
+                logger.info("Min query time: %.2f" % min_query_time)
+                tmp_res.append(round(min_query_time, 2))
             bi_res.append(tmp_res)
         return bi_res
 
@@ -160,10 +188,7 @@ class Runner(object):
         base_query_vectors = get_vectors_from_binary(MAX_NQ, dimension, data_type)
         vectors = base_query_vectors[0:nq]
         logger.info("Start query, query params: top-k: {}, nq: {}, actually length of vectors: {}".format(top_k, nq, len(vectors)))
-        status, query_res = milvus.query(vectors, top_k, nprobe)
-        if not status.OK():
-            msg = "Query failed with message: %s" % status.message
-            raise Exception(msg)
+        query_res = milvus.query(vectors, top_k, nprobe)
         result_ids = []
         result_distances = []
         for result in query_res:
@@ -181,10 +206,7 @@ class Runner(object):
         base_query_vectors = get_vectors_from_binary(MAX_NQ, dimension, data_type)
         vectors = base_query_vectors[0:nq]
         logger.info("Start query, query params: top-k: {}, nq: {}, actually length of vectors: {}".format(top_k, nq, len(vectors)))
-        status, query_res = milvus.query(vectors, top_k, nprobe)
-        if not status.OK():
-            msg = "Query failed with message: %s" % status.message
-            raise Exception(msg)
+        query_res = milvus.query(vectors, top_k, nprobe)
         # if file existed, cover it
         if os.path.isfile(id_store_name):
             os.remove(id_store_name)
@@ -212,15 +234,17 @@ class Runner(object):
         # get the accuracy
         return self.get_recall_value(flat_id_list, index_id_list)
 
-    def get_recall_value(self, flat_id_list, index_id_list):
+    def get_recall_value(self, true_ids, result_ids):
         """
         Use the intersection length
         """
         sum_radio = 0.0
-        for index, item in enumerate(index_id_list):
-            tmp = set(item).intersection(set(flat_id_list[index]))
+        for index, item in enumerate(result_ids):
+            # tmp = set(item).intersection(set(flat_id_list[index]))
+            tmp = set(true_ids[index]).intersection(set(item))
             sum_radio = sum_radio + len(tmp) / len(item)
-        return round(sum_radio / len(index_id_list), 3)
+            # logger.debug(sum_radio)
+        return round(sum_radio / len(result_ids), 3)
 
     """
     Implementation based on:

tests/milvus_benchmark/scripts/default_config.json

@@ -0,0 +1,86 @@
+[
+    {
+        "job_name": "milvus-apollo",
+        "build_params": {
+            "SUITE": "cpu_accuracy_ann.yaml",
+            "IMAGE_TYPE": "cpu",
+            "IMAGE_VERSION": "master",
+            "SERVER_HOST": "apollo"
+        }
+    },
+    {
+        "job_name": "milvus-apollo",
+        "build_params": {
+            "SUITE": "cpu_stability_sift50m.yaml",
+            "IMAGE_TYPE": "cpu",
+            "IMAGE_VERSION": "master",
+            "SERVER_HOST": "apollo"
+        }
+    },
+
+    {
+        "job_name": "milvus-eros",
+        "build_params": {
+            "SUITE": "gpu_accuracy_ann.yaml",
+            "IMAGE_TYPE": "gpu",
+            "IMAGE_VERSION": "master",
+            "SERVER_HOST": "eros"
+        }
+    },
+    {
+        "job_name": "milvus-eros",
+        "build_params": {
+            "SUITE": "gpu_search_stability.yaml",
+            "IMAGE_TYPE": "gpu",
+            "IMAGE_VERSION": "master",
+            "SERVER_HOST": "eros"
+        }
+    },
+    {
+        "job_name": "milvus-eros",
+        "build_params": {
+            "SUITE": "gpu_build_performance.yaml",
+            "IMAGE_TYPE": "gpu",
+            "IMAGE_VERSION": "master",
+            "SERVER_HOST": "eros"
+        }
+    },
+
+    {
+        "job_name": "milvus-poseidon",
+        "build_params": {
+            "SUITE": "gpu_search_performance.yaml",
+            "IMAGE_TYPE": "gpu",
+            "IMAGE_VERSION": "master",
+            "SERVER_HOST": "poseidon"
+        }
+    },
+    {
+        "job_name": "milvus-poseidon",
+        "build_params": {
+            "SUITE": "cpu_search_performance.yaml",
+            "IMAGE_TYPE": "cpu",
+            "IMAGE_VERSION": "master",
+            "SERVER_HOST": "poseidon"
+        }
+    },
+    {
+        "job_name": "milvus-poseidon",
+        "build_params": {
+            "SUITE": "insert_performance.yaml",
+            "IMAGE_TYPE": "gpu",
+            "IMAGE_VERSION": "master",
+            "SERVER_HOST": "poseidon"
+        }
+    },
+    {
+        "job_name": "milvus-poseidon",
+        "build_params": {
+            "SUITE": "gpu_accuracy.yaml",
+            "IMAGE_TYPE": "gpu",
+            "IMAGE_VERSION": "master",
+            "SERVER_HOST": "poseidon"
+        }
+    }
+
+]

tests/milvus_benchmark/scripts/scheduler.py

@@ -0,0 +1,28 @@
+import json
+from pprint import pprint
+import jenkins
+
+JENKINS_URL = "****"
+server = jenkins.Jenkins(JENKINS_URL, username='****', password='****')
+user = server.get_whoami()
+version = server.get_version()
+print('Hello %s from Jenkins %s' % (user['fullName'], version))
+
+
+# print(job_config)
+# build_params = {
+#     "SUITE": "gpu_accuracy_ann_debug.yaml",
+#     "IMAGE_TYPE": "cpu",
+#     "IMAGE_VERSION": "tanimoto_distance",
+#     "SERVER_HOST": "eros"
+# }
+# print(server.build_job(job_name, build_params))
+
+
+with open("default_config.json") as json_file:
+	data = json.load(json_file)
+	for config in data:
+		build_params = config["build_params"]
+		job_name = config["job_name"]
+		res = server.build_job(job_name, build_params)
+		print(job_name, res)

tests/milvus_benchmark/suites.yaml

@@ -1,38 +0,0 @@
-# data sets
-datasets:
-  hf5:
-    gist-960,sift-128
-  npy:
-    50000000-512, 100000000-512
-
-operations:
-  # interface: search_vectors
-  query:
-    # dataset: table name you have already created
-    # key starts with "server." need to reconfig and restart server, including nprpbe/nlist/use_blas_threshold/..
-    [
-      # debug
-      # {"dataset": "ip_ivfsq8_1000", "top_ks": [16], "nqs": [1], "server.nprobe": 1, "server.use_blas_threshold": 800, "server.cpu_cache_capacity": 110},
-      
-      {"dataset": "ip_ivfsq8_1000", "top_ks": [1, 2, 4, 8, 16, 32, 64, 128, 256, 512], "nqs": [1, 10, 100, 500, 800, 1000], "server.nprobe": 1, "server.use_blas_threshold": 800, "server.cpu_cache_capacity": 110},
-      {"dataset": "ip_ivfsq8_1000", "top_ks": [1, 2, 4, 8, 16, 32, 64, 128, 256, 512], "nqs": [1, 10, 100, 500, 800, 1000], "server.nprobe": 10, "server.use_blas_threshold": 20, "server.cpu_cache_capacity": 110},
-      {"dataset": "ip_ivfsq8_5000", "top_ks": [1, 2, 4, 8, 16, 32, 64, 128, 256, 512], "nqs": [1, 10, 100, 500, 800, 1000], "server.nprobe": 1, "server.use_blas_threshold": 800, "server.cpu_cache_capacity": 110},
-      {"dataset": "ip_ivfsq8_5000", "top_ks": [1, 2, 4, 8, 16, 32, 64, 128, 256, 512], "nqs": [1, 10, 100, 500, 800, 1000], "server.nprobe": 10, "server.use_blas_threshold": 20, "server.cpu_cache_capacity": 110},
-      {"dataset": "ip_ivfsq8_40000", "top_ks": [1, 2, 4, 8, 16, 32, 64, 128, 256, 512], "nqs": [1, 10, 100, 500, 800, 1000], "server.nprobe": 1, "server.use_blas_threshold": 800, "server.cpu_cache_capacity": 110},
-      # {"dataset": "ip_ivfsq8_40000", "top_ks": [1, 2, 4, 8, 16, 32, 64, 128, 256], "nqs": [1, 10, 100, 1000], "server.nprobe": 10, "server.use_blas_threshold": 20, "server.cpu_cache_capacity": 110},
-    ]
-
-  # interface: add_vectors
-  insert:
-    # index_type: flat/ivf_flat/ivf_sq8
-    [
-      # debug
-
-
-      {"table_name": "ip_ivf_flat_20m_1024", "table.index_type": "ivf_flat", "server.index_building_threshold": 1024, "table.size": 20000000, "table.ni": 100000, "table.dim": 512, "server.cpu_cache_capacity": 110},
-      {"table_name": "ip_ivf_sq8_50m_1024", "table.index_type": "ivf_sq8", "server.index_building_threshold": 1024, "table.size": 50000000, "table.ni": 100000, "table.dim": 512, "server.cpu_cache_capacity": 110},
-    ]
-
-  # TODO: interface: build_index
-  build: []
-

tests/milvus_benchmark/suites/cpu_accuracy.yaml

@@ -0,0 +1,119 @@
+accuracy:
+  tables:
+    # sift1b
+    -
+      server:
+        db_config.primary_path: /test/milvus/db_data_gpu/sift_1b_2048_128_l2_sq8
+        cache_config.cpu_cache_capacity: 150
+        engine_config.use_blas_threshold: 1100
+        engine_config.gpu_search_threshold: 1
+        gpu_resource_config.enable: false
+        gpu_resource_config.cache_capacity: 4
+        gpu_resource_config.search_resources:
+          - gpu0
+          - gpu1
+        gpu_resource_config.build_index_resources:
+          - gpu0
+          - gpu1
+      table_name: sift_1b_2048_128_l2
+      search_params:
+        nprobes: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]
+        top_ks: [64]
+        nqs: [1000]
+    -
+      server:
+        db_config.primary_path: /test/milvus/db_data_gpu/sift_1b_2048_128_l2_pq
+        cache_config.cpu_cache_capacity: 150
+        engine_config.use_blas_threshold: 1100
+        engine_config.gpu_search_threshold: 1
+        gpu_resource_config.enable: false
+        gpu_resource_config.cache_capacity: 4
+        gpu_resource_config.search_resources:
+          - gpu0
+          - gpu1
+        gpu_resource_config.build_index_resources:
+          - gpu0
+          - gpu1
+      table_name: sift_1b_2048_128_l2
+      search_params:
+        nprobes: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]
+        top_ks: [64]
+        nqs: [1000]
+
+    # sift50m
+    -
+      server:
+        db_config.primary_path: /test/milvus/db_data_gpu/sift_50m_1024_128_l2_ivf
+        cache_config.cpu_cache_capacity: 60
+        engine_config.use_blas_threshold: 1100
+        engine_config.gpu_search_threshold: 1
+        gpu_resource_config.enable: false
+        gpu_resource_config.cache_capacity: 4
+        gpu_resource_config.search_resources:
+          - gpu0
+          - gpu1
+        gpu_resource_config.build_index_resources:
+          - gpu0
+          - gpu1
+      table_name: sift_50m_1024_128_l2
+      search_params:
+        nprobes: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]
+        top_ks: [64]
+        nqs: [1000]
+    -
+      server:
+        db_config.primary_path: /test/milvus/db_data_gpu/sift_50m_1024_128_l2_sq8
+        cache_config.cpu_cache_capacity: 30
+        engine_config.use_blas_threshold: 1100
+        engine_config.gpu_search_threshold: 1
+        gpu_resource_config.enable: false
+        gpu_resource_config.cache_capacity: 4
+        gpu_resource_config.search_resources:
+          - gpu0
+          - gpu1
+        gpu_resource_config.build_index_resources:
+          - gpu0
+          - gpu1
+      table_name: sift_50m_1024_128_l2
+      search_params:
+        nprobes: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]
+        top_ks: [64]
+        nqs: [1000]
+    -
+      server:
+        db_config.primary_path: /test/milvus/db_data_gpu/sift_50m_1024_128_l2_pq
+        cache_config.cpu_cache_capacity: 30
+        engine_config.use_blas_threshold: 1100
+        engine_config.gpu_search_threshold: 1
+        gpu_resource_config.enable: false
+        gpu_resource_config.cache_capacity: 4
+        gpu_resource_config.search_resources:
+          - gpu0
+          - gpu1
+        gpu_resource_config.build_index_resources:
+          - gpu0
+          - gpu1
+      table_name: sift_50m_1024_128_l2
+      search_params:
+        nprobes: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]
+        top_ks: [64]
+        nqs: [1000]
+    -
+      server:
+        db_config.primary_path: /test/milvus/db_data_gpu/sift_50m_1024_128_l2_nsg
+        cache_config.cpu_cache_capacity: 100
+        engine_config.use_blas_threshold: 1100
+        engine_config.gpu_search_threshold: 1
+        gpu_resource_config.enable: false
+        gpu_resource_config.cache_capacity: 4
+        gpu_resource_config.search_resources:
+          - gpu0
+          - gpu1
+        gpu_resource_config.build_index_resources:
+          - gpu0
+          - gpu1
+      table_name: sift_50m_1024_128_l2
+      search_params:
+        nprobes: [1]
+        top_ks: [64]
+        nqs: [1000]

tests/milvus_benchmark/suites/cpu_accuracy_ann.yaml

@@ -0,0 +1,68 @@
+ann_accuracy:
+  tables:
+    -
+      server:
+        cache_config.cpu_cache_capacity: 16
+        engine_config.use_blas_threshold: 1100
+        engine_config.gpu_search_threshold: 1
+        gpu_resource_config.enable: true
+        gpu_resource_config.cache_capacity: 4
+        gpu_resource_config.search_resources:
+          - gpu0
+          - gpu1
+        gpu_resource_config.build_index_resources:
+          - gpu0
+          - gpu1
+      source_file: /test/milvus/ann_hdf5/sift-128-euclidean.hdf5
+      table_name: sift_128_euclidean
+      index_file_sizes: [1024]
+      index_types: ['flat', 'ivf_flat', 'ivf_sq8', 'ivf_pq', 'nsg']
+      nlists: [16384]
+      search_params:
+        nprobes: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]
+        top_ks: [10]
+        nqs: [10000]
+    -
+      server:
+        cache_config.cpu_cache_capacity: 16
+        engine_config.use_blas_threshold: 1100
+        engine_config.gpu_search_threshold: 1
+        gpu_resource_config.enable: true
+        gpu_resource_config.cache_capacity: 4
+        gpu_resource_config.search_resources:
+          - gpu0
+          - gpu1
+        gpu_resource_config.build_index_resources:
+          - gpu0
+          - gpu1
+      source_file: /test/milvus/ann_hdf5/gist-960-euclidean.hdf5
+      table_name: gist_960_euclidean
+      index_file_sizes: [1024]
+      index_types: ['flat', 'ivf_flat', 'ivf_sq8', 'ivf_pq', 'nsg']
+      nlists: [16384]
+      search_params:
+        nprobes: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]
+        top_ks: [10]
+        nqs: [10000]
+    -
+      server:
+        cache_config.cpu_cache_capacity: 16
+        engine_config.use_blas_threshold: 1100
+        engine_config.gpu_search_threshold: 1
+        gpu_resource_config.enable: true
+        gpu_resource_config.cache_capacity: 4
+        gpu_resource_config.search_resources:
+          - gpu0
+          - gpu1
+        gpu_resource_config.build_index_resources:
+          - gpu0
+          - gpu1
+      source_file: /test/milvus/ann_hdf5/glove-200-angular.hdf5
+      table_name: glove_200_angular
+      index_file_sizes: [1024]
+      index_types: ['flat', 'ivf_flat', 'ivf_sq8', 'ivf_pq', 'nsg']
+      nlists: [16384]
+      search_params:
+        nprobes: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]
+        top_ks: [10]
+        nqs: [10000]

tests/milvus_benchmark/suites/cpu_build_performance.yaml

@@ -0,0 +1,36 @@
+build_performance:
+  tables:
+    # -
+    #   server:
+    #     db_config.primary_path: /test/milvus/db_data_gpu/sift_50m_1024_128_l2_sq8_4096
+    #     cache_config.cpu_cache_capacity: 32
+    #     engine_config.use_blas_threshold: 1100
+    #     engine_config.gpu_search_threshold: 1
+    #     gpu_resource_config.enable: false
+    #     gpu_resource_config.cache_capacity: 6
+    #     gpu_resource_config.search_resources:
+    #       - gpu0
+    #       - gpu1
+    #     gpu_resource_config.build_index_resources:
+    #       - gpu0
+    #       - gpu1
+    #   table_name: sift_50m_1024_128_l2
+    #   index_type: ivf_sq8
+    #   nlist: 4096
+    -
+      server:
+        db_config.primary_path: /test/milvus/db_data_gpu/sift_50m_1024_128_l2_sq8_8192
+        cache_config.cpu_cache_capacity: 32
+        engine_config.use_blas_threshold: 1100
+        engine_config.gpu_search_threshold: 1
+        gpu_resource_config.enable: false
+        gpu_resource_config.cache_capacity: 6
+        gpu_resource_config.search_resources:
+          - gpu0
+          - gpu1
+        gpu_resource_config.build_index_resources:
+          - gpu0
+          - gpu1
+      table_name: sift_50m_1024_128_l2
+      index_type: ivf_sq8
+      nlist: 8192

tests/milvus_benchmark/suites/cpu_search_performance.yaml

@@ -0,0 +1,169 @@
+search_performance:
+  tables:
+    # sift_50m
+    - 
+      server:
+        db_config.primary_path: /test/milvus/db_data_gpu/sift_50m_1024_128_l2_ivf
+        cache_config.cpu_cache_capacity: 32
+        engine_config.use_blas_threshold: 1100
+        engine_config.gpu_search_threshold: 200
+        gpu_resource_config.enable: false
+        gpu_resource_config.cache_capacity: 4
+        gpu_resource_config.search_resources:
+          - gpu0
+        gpu_resource_config.build_index_resources:
+          - gpu0
+      table_name: sift_50m_1024_128_l2
+      run_count: 2
+      search_params:
+        nprobes: [8, 32]
+        top_ks: [1, 10, 100, 1000]
+        nqs: [1, 10, 100, 200, 500, 1000]
+    -
+      server:
+        db_config.primary_path: /test/milvus/db_data_gpu/sift_50m_1024_128_l2_sq8
+        cache_config.cpu_cache_capacity: 16
+        engine_config.use_blas_threshold: 1100
+        engine_config.gpu_search_threshold: 200
+        gpu_resource_config.enable: false
+        gpu_resource_config.cache_capacity: 4
+        gpu_resource_config.search_resources:
+          - gpu0
+        gpu_resource_config.build_index_resources:
+          - gpu0
+      table_name: sift_50m_1024_128_l2
+      run_count: 2
+      search_params:
+        nprobes: [8, 32]
+        top_ks: [1, 10, 100, 1000]
+        nqs: [1, 10, 100, 200, 500, 1000]
+    -
+      server:
+        db_config.primary_path: /test/milvus/db_data_gpu/sift_50m_1024_128_l2_pq
+        cache_config.cpu_cache_capacity: 32
+        engine_config.use_blas_threshold: 1100
+        engine_config.gpu_search_threshold: 200
+        gpu_resource_config.enable: false
+        gpu_resource_config.cache_capacity: 4
+        gpu_resource_config.search_resources:
+          - gpu0
+        gpu_resource_config.build_index_resources:
+          - gpu0
+      table_name: sift_50m_1024_128_l2
+      run_count: 2
+      search_params:
+        nprobes: [8, 32]
+        top_ks: [1, 10, 100, 1000]
+        nqs: [1, 10, 100, 200, 500, 1000]
+    -
+      server:
+        db_config.primary_path: /test/milvus/db_data_gpu/sift_50m_1024_128_l2_nsg
+        cache_config.cpu_cache_capacity: 50
+        engine_config.use_blas_threshold: 1100
+        engine_config.gpu_search_threshold: 200
+        gpu_resource_config.enable: false
+        gpu_resource_config.cache_capacity: 4
+        gpu_resource_config.search_resources:
+          - gpu0
+        gpu_resource_config.build_index_resources:
+          - gpu0
+      table_name: sift_50m_1024_128_l2
+      run_count: 2
+      search_params:
+        nprobes: [8]
+        top_ks: [1, 10, 100, 1000]
+        nqs: [1, 10, 100, 200, 500, 1000]
+
+    # random_50m
+    - 
+      server:
+        db_config.primary_path: /test/milvus/db_data_gpu/random_50m_1024_512_ip_ivf
+        cache_config.cpu_cache_capacity: 110
+        engine_config.use_blas_threshold: 1100
+        engine_config.gpu_search_threshold: 200
+        gpu_resource_config.enable: false
+        gpu_resource_config.cache_capacity: 4
+        gpu_resource_config.search_resources:
+          - gpu0
+        gpu_resource_config.build_index_resources:
+          - gpu0
+      table_name: random_50m_1024_512_ip
+      run_count: 2
+      search_params:
+        nprobes: [8, 32]
+        top_ks: [1, 10, 100, 1000]
+        nqs: [1, 10, 100, 200, 500, 1000]
+    -
+      server:
+        db_config.primary_path: /test/milvus/db_data_gpu/random_50m_1024_512_ip_sq8
+        cache_config.cpu_cache_capacity: 30
+        engine_config.use_blas_threshold: 1100
+        engine_config.gpu_search_threshold: 200
+        gpu_resource_config.enable: false
+        gpu_resource_config.cache_capacity: 4
+        gpu_resource_config.search_resources:
+          - gpu0
+        gpu_resource_config.build_index_resources:
+          - gpu0
+      table_name: random_50m_1024_512_ip
+      run_count: 2
+      search_params:
+        nprobes: [8, 32]
+        top_ks: [1, 10, 100, 1000]
+        nqs: [1, 10, 100, 200, 500, 1000]
+    -
+      server:
+        db_config.primary_path: /test/milvus/db_data_gpu/random_50m_1024_512_ip_nsg
+        cache_config.cpu_cache_capacity: 200
+        engine_config.use_blas_threshold: 1100
+        engine_config.gpu_search_threshold: 200
+        gpu_resource_config.enable: false
+        gpu_resource_config.cache_capacity: 4
+        gpu_resource_config.search_resources:
+          - gpu0
+        gpu_resource_config.build_index_resources:
+          - gpu0
+      table_name: random_50m_1024_512_ip
+      run_count: 2
+      search_params:
+        nprobes: [8]
+        top_ks: [1, 10, 100, 1000]
+        nqs: [1, 10, 100, 200, 500, 1000]
+
+    # sift_1b
+    - 
+      server:
+        db_config.primary_path: /test/milvus/db_data_gpu/sift_1b_2048_128_l2_sq8
+        cache_config.cpu_cache_capacity: 150
+        engine_config.use_blas_threshold: 1100
+        engine_config.gpu_search_threshold: 200
+        gpu_resource_config.enable: false
+        gpu_resource_config.cache_capacity: 4
+        gpu_resource_config.search_resources:
+          - gpu0
+        gpu_resource_config.build_index_resources:
+          - gpu0
+      table_name: sift_1b_2048_128_l2
+      run_count: 2
+      search_params:
+        nprobes: [8, 32]
+        top_ks: [1, 10, 100, 1000]
+        nqs: [1, 10, 100, 200, 500, 1000]
+    -
+      server:
+        db_config.primary_path: /test/milvus/db_data_gpu/sift_1b_2048_128_l2_pq
+        cache_config.cpu_cache_capacity: 150
+        engine_config.use_blas_threshold: 1100
+        engine_config.gpu_search_threshold: 200
+        gpu_resource_config.enable: false
+        gpu_resource_config.cache_capacity: 4
+        gpu_resource_config.search_resources:
+          - gpu0
+        gpu_resource_config.build_index_resources:
+          - gpu0
+      table_name: sift_1b_2048_128_l2
+      run_count: 2
+      search_params:
+        nprobes: [8, 32]
+        top_ks: [1, 10, 100, 1000]
+        nqs: [1, 10, 100, 200, 500, 1000]

tests/milvus_benchmark/suites/cpu_search_stability.yaml

@@ -0,0 +1,20 @@
+search_stability:
+  tables:
+    - 
+      server:
+        db_config.primary_path: /test/milvus/db_data_gpu/sift_50m_1024_128_l2_pq
+        cache_config.cpu_cache_capacity: 50
+        engine_config.use_blas_threshold: 1100
+        engine_config.gpu_search_threshold: 100
+        gpu_resource_config.enable: false
+        gpu_resource_config.cache_capacity: 4
+        gpu_resource_config.search_resources:
+          - gpu0
+        gpu_resource_config.build_index_resources:
+          - gpu0
+      table_name: sift_50m_1024_128_l2
+      during_time: 240
+      search_params:
+        nprobes: 1-200
+        top_ks: 1-200
+        nqs: 1-200

tests/milvus_benchmark/suites/cpu_stability_sift50m.yaml

@@ -0,0 +1,27 @@
+stability:
+  tables:
+    - 
+      server:
+        db_config.primary_path: /test/milvus/db_data_gpu/sift_50m_1024_128_l2_sq8_8192_stability
+        cache_config.cpu_cache_capacity: 64
+        cache_config.cache_insert_data: true
+        engine_config.use_blas_threshold: 1100
+        engine_config.gpu_search_threshold: 100
+        gpu_resource_config.enable: true
+        gpu_resource_config.cache_capacity: 4
+        gpu_resource_config.search_resources:
+          - gpu0
+          - gpu1
+        gpu_resource_config.build_index_resources:
+          - gpu0
+          - gpu1
+      table_name: sift_50m_1024_128_l2
+      during_time: 480
+      search_params:
+        nprobes: 1-200
+        top_ks: 1-200
+        nqs: 1-200
+      # length of insert vectors
+      insert_xb: 100000
+      # insert after search 4 times
+      insert_interval: 4

tests/milvus_benchmark/suites/gpu_accuracy.yaml

@@ -0,0 +1,157 @@
+accuracy:
+  tables:
+    # sift1b
+    -
+      server:
+        db_config.primary_path: /test/milvus/db_data_gpu/sift_1b_2048_128_l2_sq8
+        cache_config.cpu_cache_capacity: 150
+        engine_config.use_blas_threshold: 1100
+        engine_config.gpu_search_threshold: 1
+        gpu_resource_config.enable: true
+        gpu_resource_config.cache_capacity: 4
+        gpu_resource_config.search_resources:
+          - gpu0
+          - gpu1
+        gpu_resource_config.build_index_resources:
+          - gpu0
+          - gpu1
+      table_name: sift_1b_2048_128_l2
+      search_params:
+        nprobes: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]
+        top_ks: [64]
+        nqs: [1000]
+    -
+      server:
+        db_config.primary_path: /test/milvus/db_data_gpu/sift_1b_2048_128_l2_sq8h
+        cache_config.cpu_cache_capacity: 150
+        engine_config.use_blas_threshold: 1100
+        engine_config.gpu_search_threshold: 1
+        gpu_resource_config.enable: true
+        gpu_resource_config.cache_capacity: 4
+        gpu_resource_config.search_resources:
+          - gpu0
+          - gpu1
+        gpu_resource_config.build_index_resources:
+          - gpu0
+          - gpu1
+      table_name: sift_1b_2048_128_l2
+      search_params:
+        nprobes: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]
+        top_ks: [64]
+        nqs: [1000]
+    # -
+    #   server:
+    #     db_config.primary_path: /test/milvus/db_data_gpu/sift_1b_2048_128_l2_pq
+    #     cache_config.cpu_cache_capacity: 150
+    #     engine_config.use_blas_threshold: 1100
+    #     engine_config.gpu_search_threshold: 1
+    #     gpu_resource_config.enable: true
+    #     gpu_resource_config.cache_capacity: 4
+    #     gpu_resource_config.search_resources:
+    #       - gpu0
+    #       - gpu1
+    #     gpu_resource_config.build_index_resources:
+    #       - gpu0
+    #       - gpu1
+    #   table_name: sift_1b_2048_128_l2
+    #   search_params:
+    #     nprobes: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]
+    #     top_ks: [64]
+    #     nqs: [1000]
+
+    # sift50m
+    -
+      server:
+        db_config.primary_path: /test/milvus/db_data_gpu/sift_50m_1024_128_l2_ivf
+        cache_config.cpu_cache_capacity: 60
+        engine_config.use_blas_threshold: 1100
+        engine_config.gpu_search_threshold: 1
+        gpu_resource_config.enable: true
+        gpu_resource_config.cache_capacity: 4
+        gpu_resource_config.search_resources:
+          - gpu0
+          - gpu1
+        gpu_resource_config.build_index_resources:
+          - gpu0
+          - gpu1
+      table_name: sift_50m_1024_128_l2
+      search_params:
+        nprobes: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]
+        top_ks: [64]
+        nqs: [1000]
+    -
+      server:
+        db_config.primary_path: /test/milvus/db_data_gpu/sift_50m_1024_128_l2_sq8
+        cache_config.cpu_cache_capacity: 30
+        engine_config.use_blas_threshold: 1100
+        engine_config.gpu_search_threshold: 1
+        gpu_resource_config.enable: true
+        gpu_resource_config.cache_capacity: 4
+        gpu_resource_config.search_resources:
+          - gpu0
+          - gpu1
+        gpu_resource_config.build_index_resources:
+          - gpu0
+          - gpu1
+      table_name: sift_50m_1024_128_l2
+      search_params:
+        nprobes: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]
+        top_ks: [64]
+        nqs: [1000]
+    -
+      server:
+        db_config.primary_path: /test/milvus/db_data_gpu/sift_50m_1024_128_l2_sq8h
+        cache_config.cpu_cache_capacity: 30
+        engine_config.use_blas_threshold: 1100
+        engine_config.gpu_search_threshold: 1
+        gpu_resource_config.enable: true
+        gpu_resource_config.cache_capacity: 4
+        gpu_resource_config.search_resources:
+          - gpu0
+          - gpu1
+        gpu_resource_config.build_index_resources:
+          - gpu0
+          - gpu1
+      table_name: sift_50m_1024_128_l2
+      search_params:
+        nprobes: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]
+        top_ks: [64]
+        nqs: [1000]
+    # -
+    #   server:
+    #     db_config.primary_path: /test/milvus/db_data_gpu/sift_50m_1024_128_l2_pq
+    #     cache_config.cpu_cache_capacity: 30
+    #     engine_config.use_blas_threshold: 1100
+    #     engine_config.gpu_search_threshold: 1
+    #     gpu_resource_config.enable: true
+    #     gpu_resource_config.cache_capacity: 4
+    #     gpu_resource_config.search_resources:
+    #       - gpu0
+    #       - gpu1
+    #     gpu_resource_config.build_index_resources:
+    #       - gpu0
+    #       - gpu1
+    #   table_name: sift_50m_1024_128_l2
+    #   search_params:
+    #     nprobes: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]
+    #     top_ks: [64]
+    #     nqs: [1000]
+    -
+      server:
+        db_config.primary_path: /test/milvus/db_data_gpu/sift_50m_1024_128_l2_nsg
+        cache_config.cpu_cache_capacity: 100
+        engine_config.use_blas_threshold: 1100
+        engine_config.gpu_search_threshold: 1
+        gpu_resource_config.enable: true
+        gpu_resource_config.cache_capacity: 4
+        gpu_resource_config.search_resources:
+          - gpu0
+          - gpu1
+        gpu_resource_config.build_index_resources:
+          - gpu0
+          - gpu1
+      table_name: sift_50m_1024_128_l2
+      search_params:
+        nprobes: [1]
+        top_ks: [64]
+        nqs: [1000]

tests/milvus_benchmark/suites/gpu_accuracy_ann.yaml

@@ -0,0 +1,68 @@
+ann_accuracy:
+  tables:
+    -
+      server:
+        cache_config.cpu_cache_capacity: 16
+        engine_config.use_blas_threshold: 1100
+        engine_config.gpu_search_threshold: 1
+        gpu_resource_config.enable: true
+        gpu_resource_config.cache_capacity: 4
+        gpu_resource_config.search_resources:
+          - gpu0
+          - gpu1
+        gpu_resource_config.build_index_resources:
+          - gpu0
+          - gpu1
+      source_file: /test/milvus/ann_hdf5/sift-128-euclidean.hdf5
+      table_name: sift_128_euclidean
+      index_file_sizes: [1024]
+      index_types: ['flat', 'ivf_flat', 'ivf_sq8', 'ivf_sq8h', 'ivf_pq', 'nsg']
+      nlists: [16384]
+      search_params:
+        nprobes: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]
+        top_ks: [10]
+        nqs: [10000]
+    -
+      server:
+        cache_config.cpu_cache_capacity: 16
+        engine_config.use_blas_threshold: 1100
+        engine_config.gpu_search_threshold: 1
+        gpu_resource_config.enable: true
+        gpu_resource_config.cache_capacity: 4
+        gpu_resource_config.search_resources:
+          - gpu0
+          - gpu1
+        gpu_resource_config.build_index_resources:
+          - gpu0
+          - gpu1
+      source_file: /test/milvus/ann_hdf5/gist-960-euclidean.hdf5
+      table_name: gist_960_euclidean
+      index_file_sizes: [1024]
+      index_types: ['flat', 'ivf_flat', 'ivf_sq8', 'ivf_sq8h', 'ivf_pq', 'nsg']
+      nlists: [16384]
+      search_params:
+        nprobes: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]
+        top_ks: [10]
+        nqs: [10000]
+    -
+      server:
+        cache_config.cpu_cache_capacity: 16
+        engine_config.use_blas_threshold: 1100
+        engine_config.gpu_search_threshold: 1
+        gpu_resource_config.enable: true
+        gpu_resource_config.cache_capacity: 4
+        gpu_resource_config.search_resources:
+          - gpu0
+          - gpu1
+        gpu_resource_config.build_index_resources:
+          - gpu0
+          - gpu1
+      source_file: /test/milvus/ann_hdf5/glove-200-angular.hdf5
+      table_name: glove_200_angular
+      index_file_sizes: [1024]
+      index_types: ['flat', 'ivf_flat', 'ivf_sq8', 'ivf_sq8h', 'nsg']
+      nlists: [16384]
+      search_params:
+        nprobes: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]
+        top_ks: [10]
+        nqs: [10000]

tests/milvus_benchmark/suites/gpu_accuracy_ann_debug.yaml

@@ -0,0 +1,47 @@
+ann_accuracy:
+  tables:
+    -
+      server:
+        cache_config.cpu_cache_capacity: 16
+        engine_config.use_blas_threshold: 1100
+        engine_config.gpu_search_threshold: 1
+        gpu_resource_config.enable: true
+        gpu_resource_config.cache_capacity: 4
+        gpu_resource_config.search_resources:
+          - gpu0
+          - gpu1
+        gpu_resource_config.build_index_resources:
+          - gpu0
+          - gpu1
+      source_file: /test/milvus/ann_hdf5/kosarak-27983-jaccard.hdf5
+      table_name: kosarak_27984_jaccard
+      index_file_sizes: [1024]
+      index_types: ['flat', 'ivf_flat']
+      nlists: [2048]
+      search_params:
+        nprobes: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]
+        top_ks: [10]
+        nqs: [10000]
+
+    -
+      server:
+        cache_config.cpu_cache_capacity: 16
+        engine_config.use_blas_threshold: 1100
+        engine_config.gpu_search_threshold: 1
+        gpu_resource_config.enable: true
+        gpu_resource_config.cache_capacity: 4
+        gpu_resource_config.search_resources:
+          - gpu0
+          - gpu1
+        gpu_resource_config.build_index_resources:
+          - gpu0
+          - gpu1
+      source_file: /test/milvus/ann_hdf5/sift-256-hamming.hdf5
+      table_name: sift_256_hamming
+      index_file_sizes: [1024]
+      index_types: ['flat', 'ivf_flat']
+      nlists: [2048]
+      search_params:
+        nprobes: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]
+        top_ks: [100]
+        nqs: [1000]