milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge branch '0.5.1' into master 

Former-commit-id: 6c76b96da5353eb746d5c80f8272744e90998a84
pull/191/head
Jin Hai 2019-10-30 14:01:11 +08:00 committed by GitHub
parent 53f36927e4 8970efae49
commit 7d2e405d32
135 changed files with 2916 additions and 1814 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

15
CHANGELOG.md
View File

@ -5,10 +5,21 @@ Please mark all change in change log and use the ticket from JIRA.
# Milvus 0.5.1 (TODO)
## Bug
## Improvement
- \#64 - Improvement dump function in scheduler
## Feature
- \#90 - The server start error messages could be improved to enhance user experience
- \#104 - test_scheduler core dump
- \#115 - Using new structure for tasktable
## Improvement
- \#64 - Improvement dump function in scheduler
- \#80 - Print version information into log during server start
- \#82 - Move easyloggingpp into "external" directory
- \#92 - Speed up CMake build process
- \#96 - Remove .a file in milvus/lib for docker-version
- \#118 - Using shared_ptr instead of weak_ptr to avoid performance loss
- \#122 - Add unique id for Job
## Task
# Milvus 0.5.0 (2019-10-21)

33
README.md
View File

@ -1,9 +1,12 @@
![Milvuslogo](https://github.com/milvus-io/docs/blob/master/assets/milvus_logo.png)
![LICENSE](https://img.shields.io/badge/license-Apache--2.0-brightgreen)
![Language](https://img.shields.io/badge/language-C%2B%2B-blue)
[![codebeat badge](https://codebeat.co/badges/e030a4f6-b126-4475-a938-4723d54ec3a7?style=plastic)](https://codebeat.co/projects/github-com-jinhai-cn-milvus-master)
![Release](https://img.shields.io/badge/release-v0.5.0-orange)
![Release_date](https://img.shields.io/badge/release_date-October-yellowgreen)
- [Slack Community](https://join.slack.com/t/milvusio/shared_invite/enQtNzY1OTQ0NDI3NjMzLWNmYmM1NmNjOTQ5MGI5NDhhYmRhMGU5M2NhNzhhMDMzY2MzNDdlYjM5ODQ5MmE3ODFlYzU3YjJkNmVlNDQ2ZTk)
- [Twitter](https://twitter.com/milvusio)
@ -17,15 +20,15 @@
## What is Milvus
Milvus is an open source similarity search engine for massive feature vectors. Designed with heterogeneous computing architecture for the best cost efficiency. Searches over billion-scale vectors take only milliseconds with minimum computing resources.
Milvus is an open source similarity search engine for massive-scale feature vectors. Built with heterogeneous computing architecture for the best cost efficiency. Searches over billion-scale vectors take only milliseconds with minimum computing resources.
Milvus provides stable Python, Java and C++ APIs.
Keep up-to-date with newest releases and latest updates by reading Milvus [release notes](https://milvus.io/docs/en/releases/v0.5.0/).
Keep up-to-date with newest releases and latest updates by reading Milvus [release notes](https://milvus.io/docs/en/release/v0.5.0/).
- Heterogeneous computing
Milvus is designed with heterogeneous computing architecture for the best performance and cost efficiency.
Milvus is built with heterogeneous computing architecture for the best performance and cost efficiency.
- Multiple indexes
@ -61,14 +64,14 @@ Keep up-to-date with newest releases and latest updates by reading Milvus [relea
## Get started
### Hardware Requirements
### Hardware requirements
| Component | Recommended configuration |
| --------- | ----------------------------------- |
| CPU | Intel CPU Haswell or higher |
| GPU | NVIDIA Pascal series or higher |
| Memory | 8 GB or more (depends on data size) |
| Storage | SATA 3.0 SSD or higher |
| RAM | 8 GB or more (depends on data size) |
| Hard drive| SATA 3.0 SSD or higher |
### Install using docker
@ -89,7 +92,7 @@ Use Docker to install Milvus is a breeze. See the [Milvus install guide](https:/
```shell
$ cd [Milvus sourcecode path]/core
./ubuntu_build_deps.sh
$ ./ubuntu_build_deps.sh
```
##### Step 2 Build
@ -165,9 +168,13 @@ Make sure Java 8 or higher is already installed.
Refer to [this link](https://github.com/milvus-io/milvus-sdk-java/tree/master/examples) for the example code.
## Milvus roadmap
Please read our [roadmap](https://milvus.io/docs/en/roadmap/) to learn about upcoming features.
## Contribution guidelines
Contributions are welcomed and greatly appreciated. If you want to contribute to Milvus, please read our [contribution guidelines](CONTRIBUTING.md). This project adheres to the [code of conduct](CODE_OF_CONDUCT.md) of Milvus. By participating, you are expected to uphold this code.
Contributions are welcomed and greatly appreciated. Please read our [contribution guidelines](CONTRIBUTING.md) for detailed contribution workflow. This project adheres to the [code of conduct](CODE_OF_CONDUCT.md) of Milvus. By participating, you are expected to uphold this code.
We use [GitHub issues](https://github.com/milvus-io/milvus/issues/new/choose) to track issues and bugs. For general questions and public discussions, please join our community.
@ -175,9 +182,11 @@ We use [GitHub issues](https://github.com/milvus-io/milvus/issues/new/choose) to
To connect with other users and contributors, welcome to join our [slack channel](https://join.slack.com/t/milvusio/shared_invite/enQtNzY1OTQ0NDI3NjMzLWNmYmM1NmNjOTQ5MGI5NDhhYmRhMGU5M2NhNzhhMDMzY2MzNDdlYjM5ODQ5MmE3ODFlYzU3YjJkNmVlNDQ2ZTk).
## Milvus Roadmap
## Thanks
Please read our [roadmap](https://milvus.io/docs/en/roadmap/) to learn about upcoming features.
We greatly appreciate the help of the following people.
- [akihoni](https://github.com/akihoni) found a broken link and a small typo in the README file.
## Resources
@ -193,8 +202,8 @@ Please read our [roadmap](https://milvus.io/docs/en/roadmap/) to learn about upc
[Milvus roadmap](https://milvus.io/docs/en/roadmap/)
## License
[Apache 2.0 license](LICENSE)
[Apache License 2.0](LICENSE)

22
ci/jenkins/Jenkinsfile vendored
View File

@ -1,6 +1,15 @@
String cron_timezone = "TZ=Asia/Shanghai"
String cron_string = BRANCH_NAME == "master" ? "H 0 * * * " : ""
cron_string = BRANCH_NAME == "0.5.1" ? "H 1 * * * " : cron_string
pipeline {
agent none
triggers {
cron """${cron_timezone}
${cron_string}"""
}
options {
timestamps()
}
@ -119,7 +128,12 @@ pipeline {
steps {
container('milvus-test-env') {
script {
load "${env.WORKSPACE}/ci/jenkins/jenkinsfile/singleDevTest.groovy"
boolean isNightlyTest = isTimeTriggeredBuild()
if (isNightlyTest) {
load "${env.WORKSPACE}/ci/jenkins/jenkinsfile/singleDevNightlyTest.groovy"
} else {
load "${env.WORKSPACE}/ci/jenkins/jenkinsfile/singleDevTest.groovy"
}
}
}
}
@ -150,3 +164,9 @@ pipeline {
}
}
boolean isTimeTriggeredBuild() {
if (currentBuild.getBuildCauses('hudson.triggers.TimerTrigger$TimerTriggerCause').size() != 0) {
return true
}
return false
}

5
ci/jenkins/jenkinsfile/cleanupSingleDev.groovy
View File

@ -1,5 +1,8 @@
try {
sh "helm del --purge ${env.PIPELINE_NAME}-${env.BUILD_NUMBER}-single-gpu"
def helmResult = sh script: "helm status ${env.PIPELINE_NAME}-${env.BUILD_NUMBER}-single-gpu", returnStatus: true
if (!helmResult) {
sh "helm del --purge ${env.PIPELINE_NAME}-${env.BUILD_NUMBER}-single-gpu"
}
} catch (exc) {
def helmResult = sh script: "helm status ${env.PIPELINE_NAME}-${env.BUILD_NUMBER}-single-gpu", returnStatus: true
if (!helmResult) {

2
ci/jenkins/jenkinsfile/coverage.groovy
View File

@ -1,4 +1,4 @@
timeout(time: 60, unit: 'MINUTES') {
timeout(time: 30, unit: 'MINUTES') {
dir ("ci/jenkins/scripts") {
sh "./coverage.sh -o /opt/milvus -u root -p 123456 -t \$POD_IP"
// Set some env variables so codecov detection script works correctly

19
ci/jenkins/jenkinsfile/deploySingle2Dev.groovy
View File

@ -1,14 +1,9 @@
try {
sh 'helm init --client-only --skip-refresh --stable-repo-url https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts'
sh 'helm repo update'
dir ('milvus-helm') {
checkout([$class: 'GitSCM', branches: [[name: "0.5.0"]], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: "${params.GIT_CREDENTIALS_ID}", url: "https://github.com/milvus-io/milvus-helm.git", name: 'origin', refspec: "+refs/heads/0.5.0:refs/remotes/origin/0.5.0"]]])
dir ("milvus-gpu") {
sh "helm install --wait --timeout 300 --set engine.image.tag=${DOCKER_VERSION} --set expose.type=clusterIP --name ${env.PIPELINE_NAME}-${env.BUILD_NUMBER}-single-gpu -f ci/values.yaml -f ci/filebeat/values.yaml --namespace milvus ."
}
sh 'helm init --client-only --skip-refresh --stable-repo-url https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts'
sh 'helm repo update'
dir ('milvus-helm') {
checkout([$class: 'GitSCM', branches: [[name: "0.5.0"]], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: "${params.GIT_CREDENTIALS_ID}", url: "https://github.com/milvus-io/milvus-helm.git", name: 'origin', refspec: "+refs/heads/0.5.0:refs/remotes/origin/0.5.0"]]])
dir ("milvus-gpu") {
sh "helm install --wait --timeout 300 --set engine.image.tag=${DOCKER_VERSION} --set expose.type=clusterIP --name ${env.PIPELINE_NAME}-${env.BUILD_NUMBER}-single-gpu -f ci/db_backend/sqlite_values.yaml -f ci/filebeat/values.yaml --namespace milvus ."
}
} catch (exc) {
echo 'Helm running failed!'
sh "helm del --purge ${env.PIPELINE_NAME}-${env.BUILD_NUMBER}-single-gpu"
throw exc
}

22
ci/jenkins/jenkinsfile/singleDevNightlyTest.groovy Normal file
View File

@ -0,0 +1,22 @@
timeout(time: 90, unit: 'MINUTES') {
dir ("tests/milvus_python_test") {
sh 'python3 -m pip install -r requirements.txt'
sh "pytest . --alluredir=\"test_out/dev/single/sqlite\" --ip ${env.PIPELINE_NAME}-${env.BUILD_NUMBER}-single-gpu-milvus-gpu-engine.milvus.svc.cluster.local"
}
// mysql database backend test
load "${env.WORKSPACE}/ci/jenkins/jenkinsfile/cleanupSingleDev.groovy"
if (!fileExists('milvus-helm')) {
dir ("milvus-helm") {
checkout([$class: 'GitSCM', branches: [[name: "0.5.0"]], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: "${params.GIT_CREDENTIALS_ID}", url: "https://github.com/milvus-io/milvus-helm.git", name: 'origin', refspec: "+refs/heads/0.5.0:refs/remotes/origin/0.5.0"]]])
}
}
dir ("milvus-helm") {
dir ("milvus-gpu") {
sh "helm install --wait --timeout 300 --set engine.image.tag=${DOCKER_VERSION} --set expose.type=clusterIP --name ${env.PIPELINE_NAME}-${env.BUILD_NUMBER}-single-gpu -f ci/db_backend/mysql_values.yaml -f ci/filebeat/values.yaml --namespace milvus ."
}
}
dir ("tests/milvus_python_test") {
sh "pytest . --alluredir=\"test_out/dev/single/mysql\" --ip ${env.PIPELINE_NAME}-${env.BUILD_NUMBER}-single-gpu-milvus-gpu-engine.milvus.svc.cluster.local"
}
}

32
ci/jenkins/jenkinsfile/singleDevTest.groovy
View File

@ -1,22 +1,24 @@
timeout(time: 30, unit: 'MINUTES') {
timeout(time: 60, unit: 'MINUTES') {
dir ("tests/milvus_python_test") {
sh 'python3 -m pip install -r requirements.txt'
sh "pytest . --alluredir=\"test_out/dev/single/sqlite\" --level=1 --ip ${env.PIPELINE_NAME}-${env.BUILD_NUMBER}-single-gpu-milvus-gpu-engine.milvus.svc.cluster.local"
}
// mysql database backend test
load "${env.WORKSPACE}/ci/jenkins/jenkinsfile/cleanupSingleDev.groovy"
// load "${env.WORKSPACE}/ci/jenkins/jenkinsfile/cleanupSingleDev.groovy"
if (!fileExists('milvus-helm')) {
dir ("milvus-helm") {
checkout([$class: 'GitSCM', branches: [[name: "0.5.0"]], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: "${params.GIT_CREDENTIALS_ID}", url: "https://github.com/milvus-io/milvus-helm.git", name: 'origin', refspec: "+refs/heads/0.5.0:refs/remotes/origin/0.5.0"]]])
}
}
dir ("milvus-helm") {
dir ("milvus-gpu") {
sh "helm install --wait --timeout 300 --set engine.image.tag=${DOCKER_VERSION} --set expose.type=clusterIP --name ${env.PIPELINE_NAME}-${env.BUILD_NUMBER}-single-gpu -f ci/db_backend/mysql_values.yaml -f ci/filebeat/values.yaml --namespace milvus ."
}
}
dir ("tests/milvus_python_test") {
sh "pytest . --alluredir=\"test_out/dev/single/mysql\" --level=1 --ip ${env.PIPELINE_NAME}-${env.BUILD_NUMBER}-single-gpu-milvus-gpu-engine.milvus.svc.cluster.local"
}
// Remove mysql-version tests: 10-28
// if (!fileExists('milvus-helm')) {
// dir ("milvus-helm") {
// checkout([$class: 'GitSCM', branches: [[name: "0.5.0"]], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: "${params.GIT_CREDENTIALS_ID}", url: "https://github.com/milvus-io/milvus-helm.git", name: 'origin', refspec: "+refs/heads/0.5.0:refs/remotes/origin/0.5.0"]]])
// }
// }
// dir ("milvus-helm") {
// dir ("milvus-gpu") {
// sh "helm install --wait --timeout 300 --set engine.image.tag=${DOCKER_VERSION} --set expose.type=clusterIP --name ${env.PIPELINE_NAME}-${env.BUILD_NUMBER}-single-gpu -f ci/db_backend/mysql_values.yaml -f ci/filebeat/values.yaml --namespace milvus ."
// }
// }
// dir ("tests/milvus_python_test") {
// sh "pytest . --alluredir=\"test_out/dev/single/mysql\" --level=1 --ip ${env.PIPELINE_NAME}-${env.BUILD_NUMBER}-single-gpu-milvus-gpu-engine.milvus.svc.cluster.local"
// }
}

10
ci/jenkins/scripts/coverage.sh
View File

@ -109,6 +109,7 @@ for test in `ls ${DIR_UNITTEST}`; do
if [ $? -ne 0 ]; then
echo ${args}
echo ${DIR_UNITTEST}/${test} "run failed"
exit -1
fi
done
@ -131,8 +132,13 @@ ${LCOV_CMD} -r "${FILE_INFO_OUTPUT}" -o "${FILE_INFO_OUTPUT_NEW}" \
"*/src/server/Server.cpp" \
"*/src/server/DBWrapper.cpp" \
"*/src/server/grpc_impl/GrpcServer.cpp" \
"*/src/utils/easylogging++.h" \
"*/src/utils/easylogging++.cc"
"*/src/external/easyloggingpp/easylogging++.h" \
"*/src/external/easyloggingpp/easylogging++.cc"
if [ $? -ne 0 ]; then
echo "gen ${FILE_INFO_OUTPUT_NEW} failed"
exit -2
fi
# gen html report
# ${LCOV_GEN_CMD} "${FILE_INFO_OUTPUT_NEW}" --output-directory ${DIR_LCOV_OUTPUT}/

2
ci/jenkinsfile/milvus_build.groovy
View File

@ -14,7 +14,7 @@ container('milvus-build-env') {
sh "export JFROG_ARTFACTORY_URL='${params.JFROG_ARTFACTORY_URL}' \
&& export JFROG_USER_NAME='${USERNAME}' \
&& export JFROG_PASSWORD='${PASSWORD}' \
&& export FAISS_URL='http://192.168.1.105:6060/jinhai/faiss/-/archive/branch-0.2.1/faiss-branch-0.2.1.tar.gz' \
&& export FAISS_URL='http://192.168.1.105:6060/jinhai/faiss/-/archive/branch-0.3.0/faiss-branch-0.3.0.tar.gz' \
&& ./build.sh -t ${params.BUILD_TYPE} -d /opt/milvus -j -u -c"
sh "./coverage.sh -u root -p 123456 -t \$POD_IP"

2
ci/jenkinsfile/milvus_build_no_ut.groovy
View File

@ -14,7 +14,7 @@ container('milvus-build-env') {
sh "export JFROG_ARTFACTORY_URL='${params.JFROG_ARTFACTORY_URL}' \
&& export JFROG_USER_NAME='${USERNAME}' \
&& export JFROG_PASSWORD='${PASSWORD}' \
&& export FAISS_URL='http://192.168.1.105:6060/jinhai/faiss/-/archive/branch-0.2.1/faiss-branch-0.2.1.tar.gz' \
&& export FAISS_URL='http://192.168.1.105:6060/jinhai/faiss/-/archive/branch-0.3.0/faiss-branch-0.3.0.tar.gz' \
&& ./build.sh -t ${params.BUILD_TYPE} -j -d /opt/milvus"
}
}

2
core/CMakeLists.txt
View File

@ -71,7 +71,7 @@ if(MILVUS_VERSION_MAJOR STREQUAL ""
endif()
message(STATUS "Build version = ${MILVUS_VERSION}")
configure_file(${CMAKE_CURRENT_SOURCE_DIR}/version.h.macro ${CMAKE_CURRENT_SOURCE_DIR}/version.h)
configure_file(${CMAKE_CURRENT_SOURCE_DIR}/src/version.h.macro ${CMAKE_CURRENT_SOURCE_DIR}/src/version.h)
message(STATUS "Milvus version: "
"${MILVUS_VERSION_MAJOR}.${MILVUS_VERSION_MINOR}.${MILVUS_VERSION_PATCH} "

1
core/build-support/lint_exclusions.txt
View File

@ -6,4 +6,5 @@
*easylogging++*
*SqliteMetaImpl.cpp
*src/grpc*
*src/external*
*milvus/include*

23
core/cmake/DefineOptions.cmake
View File

@ -55,21 +55,10 @@ define_option_string(MILVUS_DEPENDENCY_SOURCE
define_option(MILVUS_VERBOSE_THIRDPARTY_BUILD
"Show output from ExternalProjects rather than just logging to files" ON)
define_option(MILVUS_BOOST_VENDORED "Use vendored Boost instead of existing Boost. \
Note that this requires linking Boost statically" OFF)
define_option(MILVUS_BOOST_HEADER_ONLY "Use only BOOST headers" OFF)
define_option(MILVUS_WITH_BZ2 "Build with BZ2 compression" ON)
define_option(MILVUS_WITH_EASYLOGGINGPP "Build with Easylogging++ library" ON)
define_option(MILVUS_WITH_LZ4 "Build with lz4 compression" ON)
define_option(MILVUS_WITH_PROMETHEUS "Build with PROMETHEUS library" ON)
define_option(MILVUS_WITH_SNAPPY "Build with Snappy compression" ON)
define_option(MILVUS_WITH_SQLITE "Build with SQLite library" ON)
define_option(MILVUS_WITH_SQLITE_ORM "Build with SQLite ORM library" ON)
@ -78,16 +67,6 @@ define_option(MILVUS_WITH_MYSQLPP "Build with MySQL++" ON)
define_option(MILVUS_WITH_YAMLCPP "Build with yaml-cpp library" ON)
define_option(MILVUS_WITH_ZLIB "Build with zlib compression" ON)
if(CMAKE_VERSION VERSION_LESS 3.7)
set(MILVUS_WITH_ZSTD_DEFAULT OFF)
else()
# ExternalProject_Add(SOURCE_SUBDIR) is available since CMake 3.7.
set(MILVUS_WITH_ZSTD_DEFAULT ON)
endif()
define_option(MILVUS_WITH_ZSTD "Build with zstd compression" ${MILVUS_WITH_ZSTD_DEFAULT})
if (MILVUS_ENABLE_PROFILING STREQUAL "ON")
define_option(MILVUS_WITH_LIBUNWIND "Build with libunwind" ON)
define_option(MILVUS_WITH_GPERFTOOLS "Build with gperftools" ON)
@ -95,6 +74,8 @@ endif()
define_option(MILVUS_WITH_GRPC "Build with GRPC" ON)
define_option(MILVUS_WITH_ZLIB "Build with zlib compression" ON)
#----------------------------------------------------------------------
if(MSVC)
set_option_category("MSVC")

707
core/cmake/ThirdPartyPackages.cmake
View File

@ -16,21 +16,16 @@
set(MILVUS_THIRDPARTY_DEPENDENCIES
BOOST
BZip2
GTest
Lz4
MySQLPP
Prometheus
Snappy
SQLite
SQLite_ORM
yaml-cpp
ZLIB
ZSTD
libunwind
gperftools
GRPC)
GRPC
ZLIB)
message(STATUS "Using ${MILVUS_DEPENDENCY_SOURCE} approach to find dependencies")
@ -42,34 +37,26 @@ foreach(DEPENDENCY ${MILVUS_THIRDPARTY_DEPENDENCIES})
endforeach()
macro(build_dependency DEPENDENCY_NAME)
if("${DEPENDENCY_NAME}" STREQUAL "BZip2")
build_bzip2()
elseif ("${DEPENDENCY_NAME}" STREQUAL "GTest")
if ("${DEPENDENCY_NAME}" STREQUAL "GTest")
build_gtest()
elseif("${DEPENDENCY_NAME}" STREQUAL "Lz4")
build_lz4()
elseif ("${DEPENDENCY_NAME}" STREQUAL "MySQLPP")
build_mysqlpp()
elseif ("${DEPENDENCY_NAME}" STREQUAL "Prometheus")
build_prometheus()
elseif ("${DEPENDENCY_NAME}" STREQUAL "Snappy")
build_snappy()
elseif ("${DEPENDENCY_NAME}" STREQUAL "SQLite")
build_sqlite()
elseif ("${DEPENDENCY_NAME}" STREQUAL "SQLite_ORM")
build_sqlite_orm()
elseif("${DEPENDENCY_NAME}" STREQUAL "yaml-cpp")
build_yamlcpp()
elseif("${DEPENDENCY_NAME}" STREQUAL "ZLIB")
build_zlib()
elseif("${DEPENDENCY_NAME}" STREQUAL "ZSTD")
build_zstd()
elseif("${DEPENDENCY_NAME}" STREQUAL "libunwind")
build_libunwind()
elseif("${DEPENDENCY_NAME}" STREQUAL "gperftools")
build_gperftools()
elseif("${DEPENDENCY_NAME}" STREQUAL "GRPC")
build_grpc()
elseif("${DEPENDENCY_NAME}" STREQUAL "ZLIB")
build_zlib()
else()
message(FATAL_ERROR "Unknown thirdparty dependency to build: ${DEPENDENCY_NAME}")
endif ()
@ -263,23 +250,6 @@ foreach(_VERSION_ENTRY ${TOOLCHAIN_VERSIONS_TXT})
set(${_LIB_NAME} "${_LIB_VERSION}")
endforeach()
if(DEFINED ENV{MILVUS_BOOST_URL})
set(BOOST_SOURCE_URL "$ENV{MILVUS_BOOST_URL}")
else()
string(REPLACE "." "_" BOOST_VERSION_UNDERSCORES ${BOOST_VERSION})
set(BOOST_SOURCE_URL
"https://nchc.dl.sourceforge.net/project/boost/boost/${BOOST_VERSION}/boost_${BOOST_VERSION_UNDERSCORES}.tar.gz")
#"https://dl.bintray.com/boostorg/release/${BOOST_VERSION}/source/boost_${BOOST_VERSION_UNDERSCORES}.tar.gz")
endif()
set(BOOST_MD5 "fea771fe8176828fabf9c09242ee8c26")
if(DEFINED ENV{MILVUS_BZIP2_URL})
set(BZIP2_SOURCE_URL "$ENV{MILVUS_BZIP2_URL}")
else()
set(BZIP2_SOURCE_URL "https://sourceware.org/pub/bzip2/bzip2-${BZIP2_VERSION}.tar.gz")
endif()
set(BZIP2_MD5 "00b516f4704d4a7cb50a1d97e6e8e15b")
if (DEFINED ENV{MILVUS_GTEST_URL})
set(GTEST_SOURCE_URL "$ENV{MILVUS_GTEST_URL}")
else ()
@ -288,13 +258,6 @@ else ()
endif()
set(GTEST_MD5 "2e6fbeb6a91310a16efe181886c59596")
if(DEFINED ENV{MILVUS_LZ4_URL})
set(LZ4_SOURCE_URL "$ENV{MILVUS_LZ4_URL}")
else()
set(LZ4_SOURCE_URL "https://github.com/lz4/lz4/archive/${LZ4_VERSION}.tar.gz")
endif()
set(LZ4_MD5 "a80f28f2a2e5fe59ebfe8407f793da22")
if(DEFINED ENV{MILVUS_MYSQLPP_URL})
set(MYSQLPP_SOURCE_URL "$ENV{MILVUS_MYSQLPP_URL}")
else()
@ -309,14 +272,6 @@ else ()
https://github.com/jupp0r/prometheus-cpp.git)
endif()
if(DEFINED ENV{MILVUS_SNAPPY_URL})
set(SNAPPY_SOURCE_URL "$ENV{MILVUS_SNAPPY_URL}")
else()
set(SNAPPY_SOURCE_URL
"https://github.com/google/snappy/archive/${SNAPPY_VERSION}.tar.gz")
endif()
set(SNAPPY_MD5 "ee9086291c9ae8deb4dac5e0b85bf54a")
if(DEFINED ENV{MILVUS_SQLITE_URL})
set(SQLITE_SOURCE_URL "$ENV{MILVUS_SQLITE_URL}")
else()
@ -329,7 +284,6 @@ if(DEFINED ENV{MILVUS_SQLITE_ORM_URL})
set(SQLITE_ORM_SOURCE_URL "$ENV{MILVUS_SQLITE_ORM_URL}")
else()
set(SQLITE_ORM_SOURCE_URL
# "http://192.168.1.105:6060/Test/sqlite_orm/-/archive/master/sqlite_orm-master.zip")
"https://github.com/fnc12/sqlite_orm/archive/${SQLITE_ORM_VERSION}.zip")
endif()
set(SQLITE_ORM_MD5 "ba9a405a8a1421c093aa8ce988ff8598")
@ -341,20 +295,6 @@ else()
endif()
set(YAMLCPP_MD5 "5b943e9af0060d0811148b037449ef82")
if(DEFINED ENV{MILVUS_ZLIB_URL})
set(ZLIB_SOURCE_URL "$ENV{MILVUS_ZLIB_URL}")
else()
set(ZLIB_SOURCE_URL "https://github.com/madler/zlib/archive/${ZLIB_VERSION}.tar.gz")
endif()
set(ZLIB_MD5 "0095d2d2d1f3442ce1318336637b695f")
if(DEFINED ENV{MILVUS_ZSTD_URL})
set(ZSTD_SOURCE_URL "$ENV{MILVUS_ZSTD_URL}")
else()
set(ZSTD_SOURCE_URL "https://github.com/facebook/zstd/archive/${ZSTD_VERSION}.tar.gz")
endif()
set(ZSTD_MD5 "340c837db48354f8d5eafe74c6077120")
if(DEFINED ENV{MILVUS_LIBUNWIND_URL})
set(LIBUNWIND_SOURCE_URL "$ENV{MILVUS_LIBUNWIND_URL}")
else()
@ -379,202 +319,12 @@ else()
endif()
set(GRPC_MD5 "0362ba219f59432c530070b5f5c3df73")
# ----------------------------------------------------------------------
# Add Boost dependencies (code adapted from Apache Kudu (incubating))
set(Boost_USE_MULTITHREADED ON)
set(Boost_ADDITIONAL_VERSIONS
"1.70.0"
"1.70"
"1.69.0"
"1.69"
"1.68.0"
"1.68"
"1.67.0"
"1.67"
"1.66.0"
"1.66"
"1.65.0"
"1.65"
"1.64.0"
"1.64"
"1.63.0"
"1.63"
"1.62.0"
"1.61"
"1.61.0"
"1.62"
"1.60.0"
"1.60")
if(MILVUS_BOOST_VENDORED)
set(BOOST_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/boost_ep-prefix/src/boost_ep")
set(BOOST_LIB_DIR "${BOOST_PREFIX}/stage/lib")
set(BOOST_BUILD_LINK "static")
set(BOOST_STATIC_SYSTEM_LIBRARY
"${BOOST_LIB_DIR}/${CMAKE_STATIC_LIBRARY_PREFIX}boost_system${CMAKE_STATIC_LIBRARY_SUFFIX}"
)
set(BOOST_STATIC_FILESYSTEM_LIBRARY
"${BOOST_LIB_DIR}/${CMAKE_STATIC_LIBRARY_PREFIX}boost_filesystem${CMAKE_STATIC_LIBRARY_SUFFIX}"
)
set(BOOST_STATIC_SERIALIZATION_LIBRARY
"${BOOST_LIB_DIR}/${CMAKE_STATIC_LIBRARY_PREFIX}boost_serialization${CMAKE_STATIC_LIBRARY_SUFFIX}"
)
set(BOOST_SYSTEM_LIBRARY boost_system_static)
set(BOOST_FILESYSTEM_LIBRARY boost_filesystem_static)
set(BOOST_SERIALIZATION_LIBRARY boost_serialization_static)
if(MILVUS_BOOST_HEADER_ONLY)
set(BOOST_BUILD_PRODUCTS)
set(BOOST_CONFIGURE_COMMAND "")
set(BOOST_BUILD_COMMAND "")
else()
set(BOOST_BUILD_PRODUCTS ${BOOST_STATIC_SYSTEM_LIBRARY}
${BOOST_STATIC_FILESYSTEM_LIBRARY} ${BOOST_STATIC_SERIALIZATION_LIBRARY})
set(BOOST_CONFIGURE_COMMAND "./bootstrap.sh" "--prefix=${BOOST_PREFIX}"
"--with-libraries=filesystem,serialization,system")
if("${CMAKE_BUILD_TYPE}" STREQUAL "DEBUG")
set(BOOST_BUILD_VARIANT "debug")
else()
set(BOOST_BUILD_VARIANT "release")
endif()
set(BOOST_BUILD_COMMAND
"./b2"
"link=${BOOST_BUILD_LINK}"
"variant=${BOOST_BUILD_VARIANT}"
"cxxflags=-fPIC")
add_thirdparty_lib(boost_system STATIC_LIB "${BOOST_STATIC_SYSTEM_LIBRARY}")
add_thirdparty_lib(boost_filesystem STATIC_LIB "${BOOST_STATIC_FILESYSTEM_LIBRARY}")
add_thirdparty_lib(boost_serialization STATIC_LIB "${BOOST_STATIC_SERIALIZATION_LIBRARY}")
set(MILVUS_BOOST_LIBS ${BOOST_SYSTEM_LIBRARY} ${BOOST_FILESYSTEM_LIBRARY} ${BOOST_STATIC_SERIALIZATION_LIBRARY})
endif()
externalproject_add(boost_ep
URL
${BOOST_SOURCE_URL}
BUILD_BYPRODUCTS
${BOOST_BUILD_PRODUCTS}
BUILD_IN_SOURCE
1
CONFIGURE_COMMAND
${BOOST_CONFIGURE_COMMAND}
BUILD_COMMAND
${BOOST_BUILD_COMMAND}
INSTALL_COMMAND
""
${EP_LOG_OPTIONS})
set(Boost_INCLUDE_DIR "${BOOST_PREFIX}")
set(Boost_INCLUDE_DIRS "${Boost_INCLUDE_DIR}")
add_dependencies(boost_system_static boost_ep)
add_dependencies(boost_filesystem_static boost_ep)
add_dependencies(boost_serialization_static boost_ep)
endif()
include_directories(SYSTEM ${Boost_INCLUDE_DIR})
link_directories(SYSTEM ${BOOST_LIB_DIR})
# ----------------------------------------------------------------------
# bzip2
macro(build_bzip2)
message(STATUS "Building BZip2-${BZIP2_VERSION} from source")
set(BZIP2_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/bzip2_ep-prefix/src/bzip2_ep")
set(BZIP2_INCLUDE_DIR "${BZIP2_PREFIX}/include")
set(BZIP2_STATIC_LIB
"${BZIP2_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}bz2${CMAKE_STATIC_LIBRARY_SUFFIX}")
if(USE_JFROG_CACHE STREQUAL "ON")
set(BZIP2_CACHE_PACKAGE_NAME "bzip2_${BZIP2_MD5}.tar.gz")
set(BZIP2_CACHE_URL "${JFROG_ARTFACTORY_CACHE_URL}/${BZIP2_CACHE_PACKAGE_NAME}")
set(BZIP2_CACHE_PACKAGE_PATH "${THIRDPARTY_PACKAGE_CACHE}/${BZIP2_CACHE_PACKAGE_NAME}")
execute_process(COMMAND wget -q --method HEAD ${BZIP2_CACHE_URL} RESULT_VARIABLE return_code)
message(STATUS "Check the remote cache file ${BZIP2_CACHE_URL}. return code = ${return_code}")
if (NOT return_code EQUAL 0)
externalproject_add(bzip2_ep
${EP_LOG_OPTIONS}
CONFIGURE_COMMAND
""
BUILD_IN_SOURCE
1
BUILD_COMMAND
${MAKE}
${MAKE_BUILD_ARGS}
CFLAGS=${EP_C_FLAGS}
INSTALL_COMMAND
${MAKE}
install
PREFIX=${BZIP2_PREFIX}
CFLAGS=${EP_C_FLAGS}
INSTALL_DIR
${BZIP2_PREFIX}
URL
${BZIP2_SOURCE_URL}
BUILD_BYPRODUCTS
"${BZIP2_STATIC_LIB}")
ExternalProject_Create_Cache(bzip2_ep ${BZIP2_CACHE_PACKAGE_PATH} "${CMAKE_CURRENT_BINARY_DIR}/bzip2_ep-prefix" ${JFROG_USER_NAME} ${JFROG_PASSWORD} ${BZIP2_CACHE_URL})
else()
file(DOWNLOAD ${BZIP2_CACHE_URL} ${BZIP2_CACHE_PACKAGE_PATH} STATUS status)
list(GET status 0 status_code)
message(STATUS "DOWNLOADING FROM ${BZIP2_CACHE_URL} TO ${BZIP2_CACHE_PACKAGE_PATH}. STATUS = ${status_code}")
if (status_code EQUAL 0)
ExternalProject_Use_Cache(bzip2_ep ${BZIP2_CACHE_PACKAGE_PATH} ${CMAKE_CURRENT_BINARY_DIR})
endif()
endif()
else()
externalproject_add(bzip2_ep
${EP_LOG_OPTIONS}
CONFIGURE_COMMAND
""
BUILD_IN_SOURCE
1
BUILD_COMMAND
${MAKE}
${MAKE_BUILD_ARGS}
CFLAGS=${EP_C_FLAGS}
INSTALL_COMMAND
${MAKE}
install
PREFIX=${BZIP2_PREFIX}
CFLAGS=${EP_C_FLAGS}
INSTALL_DIR
${BZIP2_PREFIX}
URL
${BZIP2_SOURCE_URL}
BUILD_BYPRODUCTS
"${BZIP2_STATIC_LIB}")
endif()
file(MAKE_DIRECTORY "${BZIP2_INCLUDE_DIR}")
add_library(bzip2 STATIC IMPORTED)
set_target_properties(
bzip2
PROPERTIES IMPORTED_LOCATION "${BZIP2_STATIC_LIB}"
INTERFACE_INCLUDE_DIRECTORIES "${BZIP2_INCLUDE_DIR}")
add_dependencies(bzip2 bzip2_ep)
endmacro()
if(MILVUS_WITH_BZ2)
resolve_dependency(BZip2)
if(NOT TARGET bzip2)
add_library(bzip2 UNKNOWN IMPORTED)
set_target_properties(bzip2
PROPERTIES IMPORTED_LOCATION "${BZIP2_LIBRARIES}"
INTERFACE_INCLUDE_DIRECTORIES "${BZIP2_INCLUDE_DIR}")
endif()
link_directories(SYSTEM ${BZIP2_PREFIX}/lib/)
include_directories(SYSTEM "${BZIP2_INCLUDE_DIR}")
if(DEFINED ENV{MILVUS_ZLIB_URL})
set(ZLIB_SOURCE_URL "$ENV{MILVUS_ZLIB_URL}")
else()
set(ZLIB_SOURCE_URL "https://github.com/madler/zlib/archive/${ZLIB_VERSION}.tar.gz")
endif()
set(ZLIB_MD5 "0095d2d2d1f3442ce1318336637b695f")
# ----------------------------------------------------------------------
# Google gtest
@ -689,95 +439,6 @@ if (MILVUS_BUILD_TESTS)
include_directories(SYSTEM ${GTEST_INCLUDE_DIR})
endif()
# ----------------------------------------------------------------------
# lz4
macro(build_lz4)
message(STATUS "Building lz4-${LZ4_VERSION} from source")
set(LZ4_BUILD_DIR "${CMAKE_CURRENT_BINARY_DIR}/lz4_ep-prefix/src/lz4_ep")
set(LZ4_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/lz4_ep-prefix/")
set(LZ4_STATIC_LIB "${LZ4_BUILD_DIR}/lib/liblz4.a")
set(LZ4_BUILD_COMMAND BUILD_COMMAND ${MAKE} ${MAKE_BUILD_ARGS} CFLAGS=${EP_C_FLAGS})
# We need to copy the header in lib to directory outside of the build
if(USE_JFROG_CACHE STREQUAL "ON")
set(LZ4_CACHE_PACKAGE_NAME "lz4_${LZ4_MD5}.tar.gz")
set(LZ4_CACHE_URL "${JFROG_ARTFACTORY_CACHE_URL}/${LZ4_CACHE_PACKAGE_NAME}")
set(LZ4_CACHE_PACKAGE_PATH "${THIRDPARTY_PACKAGE_CACHE}/${LZ4_CACHE_PACKAGE_NAME}")
execute_process(COMMAND wget -q --method HEAD ${LZ4_CACHE_URL} RESULT_VARIABLE return_code)
message(STATUS "Check the remote file ${LZ4_CACHE_URL}. return code = ${return_code}")
if (NOT return_code EQUAL 0)
externalproject_add(lz4_ep
URL
${LZ4_SOURCE_URL}
${EP_LOG_OPTIONS}
UPDATE_COMMAND
${CMAKE_COMMAND}
-E
copy_directory
"${LZ4_BUILD_DIR}/lib"
"${LZ4_PREFIX}/include"
${LZ4_PATCH_COMMAND}
CONFIGURE_COMMAND
""
INSTALL_COMMAND
""
BINARY_DIR
${LZ4_BUILD_DIR}
BUILD_BYPRODUCTS
${LZ4_STATIC_LIB}
${LZ4_BUILD_COMMAND})
ExternalProject_Create_Cache(lz4_ep ${LZ4_CACHE_PACKAGE_PATH} "${CMAKE_CURRENT_BINARY_DIR}/lz4_ep-prefix" ${JFROG_USER_NAME} ${JFROG_PASSWORD} ${LZ4_CACHE_URL})
else()
file(DOWNLOAD ${LZ4_CACHE_URL} ${LZ4_CACHE_PACKAGE_PATH} STATUS status)
list(GET status 0 status_code)
message(STATUS "DOWNLOADING FROM ${LZ4_CACHE_URL} TO ${LZ4_CACHE_PACKAGE_PATH}. STATUS = ${status_code}")
if (status_code EQUAL 0)
ExternalProject_Use_Cache(lz4_ep ${LZ4_CACHE_PACKAGE_PATH} ${CMAKE_CURRENT_BINARY_DIR})
endif()
endif()
else()
externalproject_add(lz4_ep
URL
${LZ4_SOURCE_URL}
${EP_LOG_OPTIONS}
UPDATE_COMMAND
${CMAKE_COMMAND}
-E
copy_directory
"${LZ4_BUILD_DIR}/lib"
"${LZ4_PREFIX}/include"
${LZ4_PATCH_COMMAND}
CONFIGURE_COMMAND
""
INSTALL_COMMAND
""
BINARY_DIR
${LZ4_BUILD_DIR}
BUILD_BYPRODUCTS
${LZ4_STATIC_LIB}
${LZ4_BUILD_COMMAND})
endif()
file(MAKE_DIRECTORY "${LZ4_PREFIX}/include")
add_library(lz4 STATIC IMPORTED)
set_target_properties(lz4
PROPERTIES IMPORTED_LOCATION "${LZ4_STATIC_LIB}"
INTERFACE_INCLUDE_DIRECTORIES "${LZ4_PREFIX}/include")
add_dependencies(lz4 lz4_ep)
endmacro()
if(MILVUS_WITH_LZ4)
resolve_dependency(Lz4)
get_target_property(LZ4_INCLUDE_DIR lz4 INTERFACE_INCLUDE_DIRECTORIES)
link_directories(SYSTEM ${LZ4_BUILD_DIR}/lib/)
include_directories(SYSTEM ${LZ4_INCLUDE_DIR})
endif()
# ----------------------------------------------------------------------
# MySQL++
@ -996,93 +657,6 @@ if(MILVUS_WITH_PROMETHEUS)
endif()
# ----------------------------------------------------------------------
# Snappy
macro(build_snappy)
message(STATUS "Building snappy-${SNAPPY_VERSION} from source")
set(SNAPPY_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/snappy_ep-prefix/src/snappy_ep")
set(SNAPPY_INCLUDE_DIRS "${SNAPPY_PREFIX}/include")
set(SNAPPY_STATIC_LIB_NAME snappy)
set(SNAPPY_STATIC_LIB
"${SNAPPY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}${SNAPPY_STATIC_LIB_NAME}${CMAKE_STATIC_LIBRARY_SUFFIX}"
)
set(SNAPPY_CMAKE_ARGS
${EP_COMMON_CMAKE_ARGS}
-DCMAKE_INSTALL_LIBDIR=lib
-DSNAPPY_BUILD_TESTS=OFF
"-DCMAKE_INSTALL_PREFIX=${SNAPPY_PREFIX}")
if(USE_JFROG_CACHE STREQUAL "ON")
set(SNAPPY_CACHE_PACKAGE_NAME "snappy_${SNAPPY_MD5}.tar.gz")
set(SNAPPY_CACHE_URL "${JFROG_ARTFACTORY_CACHE_URL}/${SNAPPY_CACHE_PACKAGE_NAME}")
set(SNAPPY_CACHE_PACKAGE_PATH "${THIRDPARTY_PACKAGE_CACHE}/${SNAPPY_CACHE_PACKAGE_NAME}")
execute_process(COMMAND wget -q --method HEAD ${SNAPPY_CACHE_URL} RESULT_VARIABLE return_code)
message(STATUS "Check the remote file ${SNAPPY_CACHE_URL}. return code = ${return_code}")
if (NOT return_code EQUAL 0)
externalproject_add(snappy_ep
${EP_LOG_OPTIONS}
BUILD_COMMAND
${MAKE}
${MAKE_BUILD_ARGS}
BUILD_IN_SOURCE
1
INSTALL_DIR
${SNAPPY_PREFIX}
URL
${SNAPPY_SOURCE_URL}
CMAKE_ARGS
${SNAPPY_CMAKE_ARGS}
BUILD_BYPRODUCTS
"${SNAPPY_STATIC_LIB}")
ExternalProject_Create_Cache(snappy_ep ${SNAPPY_CACHE_PACKAGE_PATH} "${CMAKE_CURRENT_BINARY_DIR}/snappy_ep-prefix" ${JFROG_USER_NAME} ${JFROG_PASSWORD} ${SNAPPY_CACHE_URL})
else()
file(DOWNLOAD ${SNAPPY_CACHE_URL} ${SNAPPY_CACHE_PACKAGE_PATH} STATUS status)
list(GET status 0 status_code)
message(STATUS "DOWNLOADING FROM ${SNAPPY_CACHE_URL} TO ${SNAPPY_CACHE_PACKAGE_PATH}. STATUS = ${status_code}")
if (status_code EQUAL 0)
ExternalProject_Use_Cache(snappy_ep ${SNAPPY_CACHE_PACKAGE_PATH} ${CMAKE_CURRENT_BINARY_DIR})
endif()
endif()
else()
externalproject_add(snappy_ep
${EP_LOG_OPTIONS}
BUILD_COMMAND
${MAKE}
${MAKE_BUILD_ARGS}
BUILD_IN_SOURCE
1
INSTALL_DIR
${SNAPPY_PREFIX}
URL
${SNAPPY_SOURCE_URL}
CMAKE_ARGS
${SNAPPY_CMAKE_ARGS}
BUILD_BYPRODUCTS
"${SNAPPY_STATIC_LIB}")
endif()
file(MAKE_DIRECTORY "${SNAPPY_INCLUDE_DIR}")
add_library(snappy STATIC IMPORTED)
set_target_properties(snappy
PROPERTIES IMPORTED_LOCATION "${SNAPPY_STATIC_LIB}"
INTERFACE_INCLUDE_DIRECTORIES
"${SNAPPY_INCLUDE_DIR}")
add_dependencies(snappy snappy_ep)
endmacro()
if(MILVUS_WITH_SNAPPY)
resolve_dependency(Snappy)
get_target_property(SNAPPY_INCLUDE_DIRS snappy INTERFACE_INCLUDE_DIRECTORIES)
link_directories(SYSTEM ${SNAPPY_PREFIX}/lib/)
include_directories(SYSTEM ${SNAPPY_INCLUDE_DIRS})
endif()
# ----------------------------------------------------------------------
# SQLite
@ -1265,176 +839,6 @@ if(MILVUS_WITH_YAMLCPP)
include_directories(SYSTEM ${YAMLCPP_INCLUDE_DIR})
endif()
# ----------------------------------------------------------------------
# zlib
macro(build_zlib)
message(STATUS "Building ZLIB-${ZLIB_VERSION} from source")
set(ZLIB_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/zlib_ep-prefix/src/zlib_ep")
set(ZLIB_STATIC_LIB_NAME libz.a)
set(ZLIB_STATIC_LIB "${ZLIB_PREFIX}/lib/${ZLIB_STATIC_LIB_NAME}")
set(ZLIB_INCLUDE_DIR "${ZLIB_PREFIX}/include")
set(ZLIB_CMAKE_ARGS ${EP_COMMON_CMAKE_ARGS} "-DCMAKE_INSTALL_PREFIX=${ZLIB_PREFIX}"
-DBUILD_SHARED_LIBS=OFF)
if(USE_JFROG_CACHE STREQUAL "ON")
set(ZLIB_CACHE_PACKAGE_NAME "zlib_${ZLIB_MD5}.tar.gz")
set(ZLIB_CACHE_URL "${JFROG_ARTFACTORY_CACHE_URL}/${ZLIB_CACHE_PACKAGE_NAME}")
set(ZLIB_CACHE_PACKAGE_PATH "${THIRDPARTY_PACKAGE_CACHE}/${ZLIB_CACHE_PACKAGE_NAME}")
execute_process(COMMAND wget -q --method HEAD ${ZLIB_CACHE_URL} RESULT_VARIABLE return_code)
message(STATUS "Check the remote file ${ZLIB_CACHE_URL}. return code = ${return_code}")
if (NOT return_code EQUAL 0)
externalproject_add(zlib_ep
URL
${ZLIB_SOURCE_URL}
${EP_LOG_OPTIONS}
BUILD_COMMAND
${MAKE}
${MAKE_BUILD_ARGS}
BUILD_BYPRODUCTS
"${ZLIB_STATIC_LIB}"
CMAKE_ARGS
${ZLIB_CMAKE_ARGS})
ExternalProject_Create_Cache(zlib_ep ${ZLIB_CACHE_PACKAGE_PATH} "${CMAKE_CURRENT_BINARY_DIR}/zlib_ep-prefix" ${JFROG_USER_NAME} ${JFROG_PASSWORD} ${ZLIB_CACHE_URL})
else()
file(DOWNLOAD ${ZLIB_CACHE_URL} ${ZLIB_CACHE_PACKAGE_PATH} STATUS status)
list(GET status 0 status_code)
message(STATUS "DOWNLOADING FROM ${ZLIB_CACHE_URL} TO ${ZLIB_CACHE_PACKAGE_PATH}. STATUS = ${status_code}")
if (status_code EQUAL 0)
ExternalProject_Use_Cache(zlib_ep ${ZLIB_CACHE_PACKAGE_PATH} ${CMAKE_CURRENT_BINARY_DIR})
endif()
endif()
else()
externalproject_add(zlib_ep
URL
${ZLIB_SOURCE_URL}
${EP_LOG_OPTIONS}
BUILD_COMMAND
${MAKE}
${MAKE_BUILD_ARGS}
BUILD_BYPRODUCTS
"${ZLIB_STATIC_LIB}"
CMAKE_ARGS
${ZLIB_CMAKE_ARGS})
endif()
file(MAKE_DIRECTORY "${ZLIB_INCLUDE_DIR}")
add_library(zlib STATIC IMPORTED)
set_target_properties(zlib
PROPERTIES IMPORTED_LOCATION "${ZLIB_STATIC_LIB}"
INTERFACE_INCLUDE_DIRECTORIES "${ZLIB_INCLUDE_DIR}")
add_dependencies(zlib zlib_ep)
endmacro()
if(MILVUS_WITH_ZLIB)
resolve_dependency(ZLIB)
get_target_property(ZLIB_INCLUDE_DIR zlib INTERFACE_INCLUDE_DIRECTORIES)
include_directories(SYSTEM ${ZLIB_INCLUDE_DIR})
endif()
# ----------------------------------------------------------------------
# zstd
macro(build_zstd)
message(STATUS "Building zstd-${ZSTD_VERSION} from source")
set(ZSTD_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/zstd_ep-prefix/src/zstd_ep")
set(ZSTD_CMAKE_ARGS
${EP_COMMON_TOOLCHAIN}
"-DCMAKE_INSTALL_PREFIX=${ZSTD_PREFIX}"
-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}
-DCMAKE_INSTALL_LIBDIR=lib #${CMAKE_INSTALL_LIBDIR}
-DZSTD_BUILD_PROGRAMS=off
-DZSTD_BUILD_SHARED=off
-DZSTD_BUILD_STATIC=on
-DZSTD_MULTITHREAD_SUPPORT=off)
set(ZSTD_STATIC_LIB "${ZSTD_PREFIX}/lib/libzstd.a")
set(ZSTD_INCLUDE_DIR "${ZSTD_PREFIX}/include")
set(ZSTD_CMAKE_ARGS
${ZSTD_CMAKE_ARGS}
-DCMAKE_C_COMPILER=${CMAKE_C_COMPILER}
-DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER}
-DCMAKE_C_FLAGS=${EP_C_FLAGS}
-DCMAKE_CXX_FLAGS=${EP_CXX_FLAGS})
if(CMAKE_VERSION VERSION_LESS 3.7)
message(FATAL_ERROR "Building zstd using ExternalProject requires at least CMake 3.7")
endif()
if(USE_JFROG_CACHE STREQUAL "ON")
set(ZSTD_CACHE_PACKAGE_NAME "zstd_${ZSTD_MD5}.tar.gz")
set(ZSTD_CACHE_URL "${JFROG_ARTFACTORY_CACHE_URL}/${ZSTD_CACHE_PACKAGE_NAME}")
set(ZSTD_CACHE_PACKAGE_PATH "${THIRDPARTY_PACKAGE_CACHE}/${ZSTD_CACHE_PACKAGE_NAME}")
execute_process(COMMAND wget -q --method HEAD ${ZSTD_CACHE_URL} RESULT_VARIABLE return_code)
message(STATUS "Check the remote file ${ZSTD_CACHE_URL}. return code = ${return_code}")
if (NOT return_code EQUAL 0)
externalproject_add(zstd_ep
${EP_LOG_OPTIONS}
CMAKE_ARGS
${ZSTD_CMAKE_ARGS}
SOURCE_SUBDIR
"build/cmake"
BUILD_COMMAND
${MAKE}
${MAKE_BUILD_ARGS}
INSTALL_DIR
${ZSTD_PREFIX}
URL
${ZSTD_SOURCE_URL}
BUILD_BYPRODUCTS
"${ZSTD_STATIC_LIB}")
ExternalProject_Create_Cache(zstd_ep ${ZSTD_CACHE_PACKAGE_PATH} "${CMAKE_CURRENT_BINARY_DIR}/zstd_ep-prefix" ${JFROG_USER_NAME} ${JFROG_PASSWORD} ${ZSTD_CACHE_URL})
else()
file(DOWNLOAD ${ZSTD_CACHE_URL} ${ZSTD_CACHE_PACKAGE_PATH} STATUS status)
list(GET status 0 status_code)
message(STATUS "DOWNLOADING FROM ${ZSTD_CACHE_URL} TO ${ZSTD_CACHE_PACKAGE_PATH}. STATUS = ${status_code}")
if (status_code EQUAL 0)
ExternalProject_Use_Cache(zstd_ep ${ZSTD_CACHE_PACKAGE_PATH} ${CMAKE_CURRENT_BINARY_DIR})
endif()
endif()
else()
externalproject_add(zstd_ep
${EP_LOG_OPTIONS}
CMAKE_ARGS
${ZSTD_CMAKE_ARGS}
SOURCE_SUBDIR
"build/cmake"
BUILD_COMMAND
${MAKE}
${MAKE_BUILD_ARGS}
INSTALL_DIR
${ZSTD_PREFIX}
URL
${ZSTD_SOURCE_URL}
BUILD_BYPRODUCTS
"${ZSTD_STATIC_LIB}")
endif()
file(MAKE_DIRECTORY "${ZSTD_INCLUDE_DIR}")
add_library(zstd STATIC IMPORTED)
set_target_properties(zstd
PROPERTIES IMPORTED_LOCATION "${ZSTD_STATIC_LIB}"
INTERFACE_INCLUDE_DIRECTORIES "${ZSTD_INCLUDE_DIR}")
add_dependencies(zstd zstd_ep)
endmacro()
if(MILVUS_WITH_ZSTD)
resolve_dependency(ZSTD)
get_target_property(ZSTD_INCLUDE_DIR zstd INTERFACE_INCLUDE_DIRECTORIES)
link_directories(SYSTEM ${ZSTD_PREFIX}/lib)
include_directories(SYSTEM ${ZSTD_INCLUDE_DIR})
endif()
# ----------------------------------------------------------------------
# libunwind
@ -1637,6 +1041,8 @@ macro(build_grpc)
${GRPC_PROTOBUF_STATIC_LIB}
${GRPC_PROTOC_STATIC_LIB})
ExternalProject_Add_StepDependencies(grpc_ep build zlib_ep)
ExternalProject_Create_Cache(grpc_ep ${GRPC_CACHE_PACKAGE_PATH} "${CMAKE_CURRENT_BINARY_DIR}/grpc_ep-prefix" ${JFROG_USER_NAME} ${JFROG_PASSWORD} ${GRPC_CACHE_URL})
else()
file(DOWNLOAD ${GRPC_CACHE_URL} ${GRPC_CACHE_PACKAGE_PATH} STATUS status)
@ -1665,6 +1071,9 @@ macro(build_grpc)
${GRPCPP_CHANNELZ_STATIC_LIB}
${GRPC_PROTOBUF_STATIC_LIB}
${GRPC_PROTOC_STATIC_LIB})
ExternalProject_Add_StepDependencies(grpc_ep build zlib_ep)
endif()
file(MAKE_DIRECTORY "${GRPC_INCLUDE_DIR}")
@ -1672,25 +1081,30 @@ macro(build_grpc)
add_library(grpc STATIC IMPORTED)
set_target_properties(grpc
PROPERTIES IMPORTED_LOCATION "${GRPC_STATIC_LIB}"
INTERFACE_INCLUDE_DIRECTORIES "${GRPC_INCLUDE_DIR}")
INTERFACE_INCLUDE_DIRECTORIES "${GRPC_INCLUDE_DIR}"
INTERFACE_LINK_LIBRARIES "zlib" )
add_library(grpc++ STATIC IMPORTED)
set_target_properties(grpc++
PROPERTIES IMPORTED_LOCATION "${GRPC++_STATIC_LIB}"
INTERFACE_INCLUDE_DIRECTORIES "${GRPC_INCLUDE_DIR}")
INTERFACE_INCLUDE_DIRECTORIES "${GRPC_INCLUDE_DIR}"
INTERFACE_LINK_LIBRARIES "zlib" )
add_library(grpcpp_channelz STATIC IMPORTED)
set_target_properties(grpcpp_channelz
PROPERTIES IMPORTED_LOCATION "${GRPCPP_CHANNELZ_STATIC_LIB}"
INTERFACE_INCLUDE_DIRECTORIES "${GRPC_INCLUDE_DIR}")
INTERFACE_INCLUDE_DIRECTORIES "${GRPC_INCLUDE_DIR}"
INTERFACE_LINK_LIBRARIES "zlib" )
add_library(grpc_protobuf STATIC IMPORTED)
set_target_properties(grpc_protobuf
PROPERTIES IMPORTED_LOCATION "${GRPC_PROTOBUF_STATIC_LIB}")
PROPERTIES IMPORTED_LOCATION "${GRPC_PROTOBUF_STATIC_LIB}"
INTERFACE_LINK_LIBRARIES "zlib" )
add_library(grpc_protoc STATIC IMPORTED)
set_target_properties(grpc_protoc
PROPERTIES IMPORTED_LOCATION "${GRPC_PROTOC_STATIC_LIB}")
PROPERTIES IMPORTED_LOCATION "${GRPC_PROTOC_STATIC_LIB}"
INTERFACE_LINK_LIBRARIES "zlib" )
add_dependencies(grpc grpc_ep)
add_dependencies(grpc++ grpc_ep)
@ -1710,3 +1124,74 @@ if(MILVUS_WITH_GRPC)
include_directories(SYSTEM ${GRPC_THIRD_PARTY_DIR}/protobuf/src)
link_directories(SYSTEM ${GRPC_PROTOBUF_LIB_DIR})
endif()
# ----------------------------------------------------------------------
# zlib
macro(build_zlib)
message(STATUS "Building ZLIB-${ZLIB_VERSION} from source")
set(ZLIB_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/zlib_ep-prefix/src/zlib_ep")
set(ZLIB_STATIC_LIB_NAME libz.a)
set(ZLIB_STATIC_LIB "${ZLIB_PREFIX}/lib/${ZLIB_STATIC_LIB_NAME}")
set(ZLIB_INCLUDE_DIR "${ZLIB_PREFIX}/include")
set(ZLIB_CMAKE_ARGS ${EP_COMMON_CMAKE_ARGS} "-DCMAKE_INSTALL_PREFIX=${ZLIB_PREFIX}"
-DBUILD_SHARED_LIBS=OFF)
if(USE_JFROG_CACHE STREQUAL "ON")
set(ZLIB_CACHE_PACKAGE_NAME "zlib_${ZLIB_MD5}.tar.gz")
set(ZLIB_CACHE_URL "${JFROG_ARTFACTORY_CACHE_URL}/${ZLIB_CACHE_PACKAGE_NAME}")
set(ZLIB_CACHE_PACKAGE_PATH "${THIRDPARTY_PACKAGE_CACHE}/${ZLIB_CACHE_PACKAGE_NAME}")
execute_process(COMMAND wget -q --method HEAD ${ZLIB_CACHE_URL} RESULT_VARIABLE return_code)
message(STATUS "Check the remote file ${ZLIB_CACHE_URL}. return code = ${return_code}")
if (NOT return_code EQUAL 0)
externalproject_add(zlib_ep
URL
${ZLIB_SOURCE_URL}
${EP_LOG_OPTIONS}
BUILD_COMMAND
${MAKE}
${MAKE_BUILD_ARGS}
BUILD_BYPRODUCTS
"${ZLIB_STATIC_LIB}"
CMAKE_ARGS
${ZLIB_CMAKE_ARGS})
ExternalProject_Create_Cache(zlib_ep ${ZLIB_CACHE_PACKAGE_PATH} "${CMAKE_CURRENT_BINARY_DIR}/zlib_ep-prefix" ${JFROG_USER_NAME} ${JFROG_PASSWORD} ${ZLIB_CACHE_URL})
else()
file(DOWNLOAD ${ZLIB_CACHE_URL} ${ZLIB_CACHE_PACKAGE_PATH} STATUS status)
list(GET status 0 status_code)
message(STATUS "DOWNLOADING FROM ${ZLIB_CACHE_URL} TO ${ZLIB_CACHE_PACKAGE_PATH}. STATUS = ${status_code}")
if (status_code EQUAL 0)
ExternalProject_Use_Cache(zlib_ep ${ZLIB_CACHE_PACKAGE_PATH} ${CMAKE_CURRENT_BINARY_DIR})
endif()
endif()
else()
externalproject_add(zlib_ep
URL
${ZLIB_SOURCE_URL}
${EP_LOG_OPTIONS}
BUILD_COMMAND
${MAKE}
${MAKE_BUILD_ARGS}
BUILD_BYPRODUCTS
"${ZLIB_STATIC_LIB}"
CMAKE_ARGS
${ZLIB_CMAKE_ARGS})
endif()
file(MAKE_DIRECTORY "${ZLIB_INCLUDE_DIR}")
add_library(zlib STATIC IMPORTED)
set_target_properties(zlib
PROPERTIES IMPORTED_LOCATION "${ZLIB_STATIC_LIB}"
INTERFACE_INCLUDE_DIRECTORIES "${ZLIB_INCLUDE_DIR}")
add_dependencies(zlib zlib_ep)
endmacro()
if(MILVUS_WITH_ZLIB)
resolve_dependency(ZLIB)
get_target_property(ZLIB_INCLUDE_DIR zlib INTERFACE_INCLUDE_DIRECTORIES)
include_directories(SYSTEM ${ZLIB_INCLUDE_DIR})
endif()

24
core/conf/server_config.template
View File

@ -2,9 +2,9 @@
server_config:
address: 0.0.0.0 # milvus server ip address (IPv4)
port: 19530 # port range: 1025 ~ 65534
port: 19530 # milvus server port, must in range [1025, 65534]
deploy_mode: single # deployment type: single, cluster_readonly, cluster_writable
time_zone: UTC+8
time_zone: UTC+8 # time zone, must be in format: UTC+X
db_config:
primary_path: @MILVUS_DB_PATH@ # path used to store data and meta
@ -14,30 +14,30 @@ db_config:
# Keep 'dialect://:@:/', and replace other texts with real values
# Replace 'dialect' with 'mysql' or 'sqlite'
insert_buffer_size: 4 # GB, maximum insert buffer size allowed
insert_buffer_size: 4 # GB, maximum insert buffer size allowed, must be a positive integer
# sum of insert_buffer_size and cpu_cache_capacity cannot exceed total memory
preload_table: # preload data at startup, '*' means load all tables, empty value means no preload
# you can specify preload tables like this: table1,table2,table3
metric_config:
enable_monitor: false # enable monitoring or not
enable_monitor: false # enable monitoring or not, must be a boolean
collector: prometheus # prometheus
prometheus_config:
port: 8080 # port prometheus uses to fetch metrics
port: 8080 # port prometheus uses to fetch metrics, must in range [1025, 65534]
cache_config:
cpu_cache_capacity: 16 # GB, CPU memory used for cache
cpu_cache_threshold: 0.85 # percentage of data that will be kept when cache cleanup is triggered
gpu_cache_capacity: 4 # GB, GPU memory used for cache
gpu_cache_threshold: 0.85 # percentage of data that will be kept when cache cleanup is triggered
cache_insert_data: false # whether to load inserted data into cache
cpu_cache_capacity: 16 # GB, CPU memory used for cache, must be a positive integer
cpu_cache_threshold: 0.85 # percentage of data that will be kept when cache cleanup is triggered, must be in range (0.0, 1.0]
gpu_cache_capacity: 4 # GB, GPU memory used for cache, must be a positive integer
gpu_cache_threshold: 0.85 # percentage of data that will be kept when cache cleanup is triggered, must be in range (0.0, 1.0]
cache_insert_data: false # whether to load inserted data into cache, must be a boolean
engine_config:
use_blas_threshold: 20 # if nq < use_blas_threshold, use SSE, faster with fluctuated response times
# if nq >= use_blas_threshold, use OpenBlas, slower with stable response times
resource_config:
search_resources: # define the GPUs used for search computation, valid value: gpux
search_resources: # define the GPUs used for search computation, must be in format: gpux
- gpu0
index_build_device: gpu0 # GPU used for building index
index_build_device: gpu0 # GPU used for building index, must be in format: gpux

10
core/coverage.sh
View File

@ -99,6 +99,7 @@ for test in `ls ${DIR_UNITTEST}`; do
if [ $? -ne 0 ]; then
echo ${args}
echo ${DIR_UNITTEST}/${test} "run failed"
exit -1
fi
done
@ -121,8 +122,13 @@ ${LCOV_CMD} -r "${FILE_INFO_OUTPUT}" -o "${FILE_INFO_OUTPUT_NEW}" \
"*/src/server/Server.cpp" \
"*/src/server/DBWrapper.cpp" \
"*/src/server/grpc_impl/GrpcServer.cpp" \
"*/src/utils/easylogging++.h" \
"*/src/utils/easylogging++.cc"
"*/easylogging++.h" \
"*/easylogging++.cc" \
"*/src/external/*"
if [ $? -ne 0 ]; then
echo "generate ${FILE_INFO_OUTPUT_NEW} failed"
exit -2
fi
# gen html report
${LCOV_GEN_CMD} "${FILE_INFO_OUTPUT_NEW}" --output-directory ${DIR_LCOV_OUTPUT}/

16
core/src/CMakeLists.txt
View File

@ -64,6 +64,13 @@ set(scheduler_files
${scheduler_task_files}
)
aux_source_directory(${MILVUS_ENGINE_SRC}/external/easyloggingpp external_easyloggingpp_files)
aux_source_directory(${MILVUS_ENGINE_SRC}/external/nlohmann external_nlohmann_files)
set(external_files
${external_easyloggingpp_files}
${external_nlohmann_files}
)
aux_source_directory(${MILVUS_ENGINE_SRC}/server server_files)
aux_source_directory(${MILVUS_ENGINE_SRC}/server/grpc_impl grpc_server_files)
aux_source_directory(${MILVUS_ENGINE_SRC}/utils utils_files)
@ -77,6 +84,7 @@ set(engine_files
${db_insert_files}
${db_meta_files}
${metrics_files}
${external_files}
${utils_files}
${wrapper_files}
)
@ -112,14 +120,10 @@ set(third_party_libs
${client_grpc_lib}
yaml-cpp
${prometheus_lib}
${boost_lib}
bzip2
lz4
snappy
zlib
zstd
${cuda_lib}
mysqlpp
zlib
${boost_lib}
)
if (MILVUS_ENABLE_PROFILING STREQUAL "ON")

6
core/src/db/DBImpl.cpp
View File

@ -136,7 +136,7 @@ DBImpl::DeleteTable(const std::string& table_id, const meta::DatesT& dates) {
// scheduler will determine when to delete table files
auto nres = scheduler::ResMgrInst::GetInstance()->GetNumOfComputeResource();
scheduler::DeleteJobPtr job = std::make_shared<scheduler::DeleteJob>(0, table_id, meta_ptr_, nres);
scheduler::DeleteJobPtr job = std::make_shared<scheduler::DeleteJob>(table_id, meta_ptr_, nres);
scheduler::JobMgrInst::GetInstance()->Put(job);
job->WaitAndDelete();
} else {
@ -439,7 +439,7 @@ DBImpl::QueryAsync(const std::string& table_id, const meta::TableFilesSchema& fi
// step 1: get files to search
ENGINE_LOG_DEBUG << "Engine query begin, index file count: " << files.size();
scheduler::SearchJobPtr job = std::make_shared<scheduler::SearchJob>(0, k, nq, nprobe, vectors);
scheduler::SearchJobPtr job = std::make_shared<scheduler::SearchJob>(k, nq, nprobe, vectors);
for (auto& file : files) {
scheduler::TableFileSchemaPtr file_ptr = std::make_shared<meta::TableFileSchema>(file);
job->AddIndexFile(file_ptr);
@ -754,7 +754,7 @@ DBImpl::BackgroundBuildIndex() {
Status status;
if (!to_index_files.empty()) {
scheduler::BuildIndexJobPtr job = std::make_shared<scheduler::BuildIndexJob>(0, meta_ptr_, options_);
scheduler::BuildIndexJobPtr job = std::make_shared<scheduler::BuildIndexJob>(meta_ptr_, options_);
// step 2: put build index task to scheduler
for (auto& file : to_index_files) {

57
core/src/db/engine/ExecutionEngineImpl.cpp
View File

@ -18,16 +18,13 @@
#include "db/engine/ExecutionEngineImpl.h"
#include "cache/CpuCacheMgr.h"
#include "cache/GpuCacheMgr.h"
#include "knowhere/common/Config.h"
#include "metrics/Metrics.h"
#include "scheduler/Utils.h"
#include "server/Config.h"
#include "utils/CommonUtil.h"
#include "utils/Exception.h"
#include "utils/Log.h"
#include "knowhere/common/Config.h"
#include "knowhere/common/Exception.h"
#include "knowhere/index/vector_index/IndexIVFSQHybrid.h"
#include "scheduler/Utils.h"
#include "server/Config.h"
#include "wrapper/ConfAdapter.h"
#include "wrapper/ConfAdapterMgr.h"
#include "wrapper/VecImpl.h"
@ -260,6 +257,54 @@ ExecutionEngineImpl::Load(bool to_cache) {
Status
ExecutionEngineImpl::CopyToGpu(uint64_t device_id, bool hybrid) {
if (hybrid) {
const std::string key = location_ + ".quantizer";
std::vector<uint64_t> gpus = scheduler::get_gpu_pool();
const int64_t NOT_FOUND = -1;
int64_t device_id = NOT_FOUND;
// cache hit
{
knowhere::QuantizerPtr quantizer = nullptr;
for (auto& gpu : gpus) {
auto cache = cache::GpuCacheMgr::GetInstance(gpu);
if (auto cached_quantizer = cache->GetIndex(key)) {
device_id = gpu;
quantizer = std::static_pointer_cast<CachedQuantizer>(cached_quantizer)->Data();
}
}
if (device_id != NOT_FOUND) {
// cache hit
auto config = std::make_shared<knowhere::QuantizerCfg>();
config->gpu_id = device_id;
config->mode = 2;
auto new_index = index_->LoadData(quantizer, config);
index_ = new_index;
}
}
if (device_id == NOT_FOUND) {
// cache miss
std::vector<int64_t> all_free_mem;
for (auto& gpu : gpus) {
auto cache = cache::GpuCacheMgr::GetInstance(gpu);
auto free_mem = cache->CacheCapacity() - cache->CacheUsage();
all_free_mem.push_back(free_mem);
}
auto max_e = std::max_element(all_free_mem.begin(), all_free_mem.end());
auto best_index = std::distance(all_free_mem.begin(), max_e);
device_id = gpus[best_index];
auto pair = index_->CopyToGpuWithQuantizer(device_id);
index_ = pair.first;
// cache
auto cached_quantizer = std::make_shared<CachedQuantizer>(pair.second);
cache::GpuCacheMgr::GetInstance(device_id)->InsertItem(key, cached_quantizer);
}
return Status::OK();
}

0
core/src/utils/easylogging++.cc → core/src/external/easyloggingpp/easylogging++.cc vendored
View File

0
core/src/utils/easylogging++.h → core/src/external/easyloggingpp/easylogging++.h vendored
View File

30
core/src/index/cmake/ThirdPartyPackagesCore.cmake
View File

@ -244,11 +244,12 @@ if(CUSTOMIZATION)
# set(FAISS_MD5 "21deb1c708490ca40ecb899122c01403") # commit-id 643e48f479637fd947e7b93fa4ca72b38ecc9a39 branch-0.2.0
# set(FAISS_MD5 "072db398351cca6e88f52d743bbb9fa0") # commit-id 3a2344d04744166af41ef1a74449d68a315bfe17 branch-0.2.1
# set(FAISS_MD5 "c89ea8e655f5cdf58f42486f13614714") # commit-id 9c28a1cbb88f41fa03b03d7204106201ad33276b branch-0.2.1
set(FAISS_MD5 "87fdd86351ffcaf3f80dc26ade63c44b") # commit-id 841a156e67e8e22cd8088e1b58c00afbf2efc30b branch-0.2.1
# set(FAISS_MD5 "87fdd86351ffcaf3f80dc26ade63c44b") # commit-id 841a156e67e8e22cd8088e1b58c00afbf2efc30b branch-0.2.1
set(FAISS_MD5 "f3b2ce3364c3fa7febd3aa7fdd0fe380") # commit-id 694e03458e6b69ce8a62502f71f69a614af5af8f branch-0.3.0
endif()
else()
set(FAISS_SOURCE_URL "https://github.com/facebookresearch/faiss/archive/v1.5.3.tar.gz")
set(FAISS_MD5 "0bc12737b23def156f6a1eb782050135")
set(FAISS_SOURCE_URL "https://github.com/milvus-io/faiss/archive/1.6.0.tar.gz")
set(FAISS_MD5 "eb96d84f98b078a9eec04a796f5c792e")
endif()
message(STATUS "FAISS URL = ${FAISS_SOURCE_URL}")
@ -299,12 +300,29 @@ macro(build_arrow)
${EP_COMMON_CMAKE_ARGS}
-DARROW_BUILD_STATIC=ON
-DARROW_BUILD_SHARED=OFF
-DARROW_PARQUET=OFF
-DARROW_USE_GLOG=OFF
-DCMAKE_INSTALL_PREFIX=${ARROW_PREFIX}
"-DCMAKE_LIBRARY_PATH=${CUDA_TOOLKIT_ROOT_DIR}/lib64/stubs"
-DARROW_CUDA=OFF
-DARROW_FLIGHT=OFF
-DARROW_GANDIVA=OFF
-DARROW_GANDIVA_JAVA=OFF
-DARROW_HDFS=OFF
-DARROW_HIVESERVER2=OFF
-DARROW_ORC=OFF
-DARROW_PARQUET=OFF
-DARROW_PLASMA=OFF
-DARROW_PLASMA_JAVA_CLIENT=OFF
-DARROW_PYTHON=OFF
-DARROW_WITH_BZ2=OFF
-DARROW_WITH_ZLIB=OFF
-DARROW_WITH_LZ4=OFF
-DARROW_WITH_SNAPPY=OFF
-DARROW_WITH_ZSTD=OFF
-DARROW_WITH_BROTLI=OFF
-DCMAKE_BUILD_TYPE=Release
-DARROW_DEPENDENCY_SOURCE=BUNDLED) #Build all arrow dependencies from source instead of calling find_package first
-DARROW_DEPENDENCY_SOURCE=BUNDLED #Build all arrow dependencies from source instead of calling find_package first
-DBOOST_SOURCE=AUTO #try to find BOOST in the system default locations and build from source if not found
)
if(USE_JFROG_CACHE STREQUAL "ON")

36
core/src/index/knowhere/CMakeLists.txt
View File

@ -81,27 +81,6 @@ target_link_libraries(
${depend_libs}
)
INSTALL(TARGETS
knowhere
SPTAGLibStatic
DESTINATION
lib)
INSTALL(FILES
${ARROW_STATIC_LIB}
${ARROW_PREFIX}/lib/libjemalloc_pic.a
${FAISS_STATIC_LIB}
${LAPACK_STATIC_LIB}
${BLAS_STATIC_LIB}
DESTINATION
lib
)
INSTALL(FILES ${OPENBLAS_REAL_STATIC_LIB}
RENAME "libopenblas.a"
DESTINATION lib
)
set(INDEX_INCLUDE_DIRS
${INDEX_SOURCE_DIR}/knowhere
${INDEX_SOURCE_DIR}/thirdparty
@ -112,17 +91,4 @@ set(INDEX_INCLUDE_DIRS
${LAPACK_INCLUDE_DIR}
)
set(INDEX_INCLUDE_DIRS ${INDEX_INCLUDE_DIRS} PARENT_SCOPE)
#INSTALL(DIRECTORY
# ${INDEX_SOURCE_DIR}/include/knowhere
# ${ARROW_INCLUDE_DIR}/arrow
# ${FAISS_PREFIX}/include/faiss
# ${OPENBLAS_INCLUDE_DIR}/
# DESTINATION
# include)
#
#INSTALL(DIRECTORY
# ${SPTAG_SOURCE_DIR}/AnnService/inc/
# DESTINATION
# include/SPTAG/AnnService/inc)
set(INDEX_INCLUDE_DIRS ${INDEX_INCLUDE_DIRS} PARENT_SCOPE)

2
core/src/index/knowhere/knowhere/common/Log.h
View File

@ -17,7 +17,7 @@
#pragma once
#include "utils/easylogging++.h"
#include "external/easyloggingpp/easylogging++.h"
namespace knowhere {

2
core/src/index/knowhere/knowhere/index/vector_index/FaissBaseIndex.h
View File

@ -38,7 +38,7 @@ class FaissBaseIndex {
virtual void
SealImpl();
protected:
public:
std::shared_ptr<faiss::Index> index_ = nullptr;
};

14
core/src/index/knowhere/knowhere/index/vector_index/IndexGPUIVF.cpp
View File

@ -15,12 +15,12 @@
// specific language governing permissions and limitations
// under the License.
#include <faiss/gpu/GpuAutoTune.h>
#include <faiss/gpu/GpuIndexFlat.h>
#include <memory>
#include <faiss/gpu/GpuCloner.h>
#include <faiss/gpu/GpuIndexIVF.h>
#include <faiss/gpu/GpuIndexIVFFlat.h>
#include <faiss/index_io.h>
#include <memory>
#include "knowhere/adapter/VectorAdapter.h"
#include "knowhere/common/Exception.h"
@ -86,7 +86,8 @@ GPUIVF::SerializeImpl() {
faiss::Index* index = index_.get();
faiss::Index* host_index = faiss::gpu::index_gpu_to_cpu(index);
SealImpl();
// TODO(linxj): support seal
// SealImpl();
faiss::write_index(host_index, &writer);
delete host_index;
@ -130,13 +131,12 @@ void
GPUIVF::search_impl(int64_t n, const float* data, int64_t k, float* distances, int64_t* labels, const Config& cfg) {
std::lock_guard<std::mutex> lk(mutex_);
// TODO(linxj): gpu index support GenParams
if (auto device_index = std::dynamic_pointer_cast<faiss::gpu::GpuIndexIVF>(index_)) {
auto search_cfg = std::dynamic_pointer_cast<IVFCfg>(cfg);
device_index->setNumProbes(search_cfg->nprobe);
device_index->nprobe = search_cfg->nprobe;
// assert(device_index->getNumProbes() == search_cfg->nprobe);
{
// TODO(linxj): allocate gpu mem
ResScope rs(res_, gpu_id_);
device_index->search(n, (float*)data, k, distances, labels);
}

4
core/src/index/knowhere/knowhere/index/vector_index/IndexGPUIVFPQ.cpp
View File

@ -16,8 +16,10 @@
// under the License.
#include <faiss/IndexIVFPQ.h>
#include <faiss/gpu/GpuAutoTune.h>
#include <faiss/gpu/GpuCloner.h>
#include <faiss/gpu/GpuIndexIVFPQ.h>
#include <faiss/index_factory.h>
#include <memory>
#include "knowhere/adapter/VectorAdapter.h"

14
core/src/index/knowhere/knowhere/index/vector_index/IndexGPUIVFSQ.cpp
View File

@ -15,9 +15,10 @@
// specific language governing permissions and limitations
// under the License.
#include <faiss/gpu/GpuAutoTune.h>
#include <faiss/gpu/GpuCloner.h>
#include <faiss/index_factory.h>
#include <memory>
#include <utility>
#include "knowhere/adapter/VectorAdapter.h"
#include "knowhere/common/Exception.h"
@ -71,13 +72,4 @@ GPUIVFSQ::CopyGpuToCpu(const Config& config) {
return std::make_shared<IVFSQ>(new_index);
}
void
GPUIVFSQ::search_impl(int64_t n, const float* data, int64_t k, float* distances, int64_t* labels, const Config& cfg) {
#ifdef CUSTOMIZATION
GPUIVF::search_impl(n, data, k, distances, labels, cfg);
#else
IVF::search_impl(n, data, k, distances, labels, cfg);
#endif
}
} // namespace knowhere

4
core/src/index/knowhere/knowhere/index/vector_index/IndexGPUIVFSQ.h
View File

@ -38,10 +38,6 @@ class GPUIVFSQ : public GPUIVF {
VectorIndexPtr
CopyGpuToCpu(const Config& config) override;
protected:
void
search_impl(int64_t n, const float* data, int64_t k, float* distances, int64_t* labels, const Config& cfg) override;
};
} // namespace knowhere

5
core/src/index/knowhere/knowhere/index/vector_index/IndexIDMAP.cpp
View File

@ -15,11 +15,12 @@
// specific language governing permissions and limitations
// under the License.
#include <faiss/AutoTune.h>
#include <faiss/IndexFlat.h>
#include <faiss/MetaIndexes.h>
#include <faiss/gpu/GpuAutoTune.h>
#include <faiss/gpu/GpuCloner.h>
#include <faiss/index_factory.h>
#include <faiss/index_io.h>
#include <vector>
#include "knowhere/adapter/VectorAdapter.h"

7
core/src/index/knowhere/knowhere/index/vector_index/IndexIVF.cpp
View File

@ -15,15 +15,12 @@
// specific language governing permissions and limitations
// under the License.
#include <faiss/AutoTune.h>
#include <faiss/AuxIndexStructures.h>
#include <faiss/IVFlib.h>
#include <faiss/IndexFlat.h>
#include <faiss/IndexIVF.h>
#include <faiss/IndexIVFFlat.h>
#include <faiss/IndexIVFPQ.h>
#include <faiss/gpu/GpuAutoTune.h>
#include <faiss/index_io.h>
#include <faiss/gpu/GpuCloner.h>
#include <chrono>
#include <memory>
#include <utility>

2
core/src/index/knowhere/knowhere/index/vector_index/IndexIVF.h
View File

@ -30,7 +30,7 @@ namespace knowhere {
using Graph = std::vector<std::vector<int64_t>>;
class IVF : public VectorIndex, protected FaissBaseIndex {
class IVF : public VectorIndex, public FaissBaseIndex {
public:
IVF() : FaissBaseIndex(nullptr) {
}

10
core/src/index/knowhere/knowhere/index/vector_index/IndexIVFSQ.cpp
View File

@ -15,7 +15,8 @@
// specific language governing permissions and limitations
// under the License.
#include <faiss/gpu/GpuAutoTune.h>
#include <faiss/gpu/GpuCloner.h>
#include <faiss/index_factory.h>
#include <memory>
#include "knowhere/adapter/VectorAdapter.h"
@ -56,14 +57,7 @@ IVFSQ::CopyCpuToGpu(const int64_t& device_id, const Config& config) {
if (auto res = FaissGpuResourceMgr::GetInstance().GetRes(device_id)) {
ResScope rs(res, device_id, false);
#ifdef CUSTOMIZATION
faiss::gpu::GpuClonerOptions option;
option.allInGpu = true;
auto gpu_index = faiss::gpu::index_cpu_to_gpu(res->faiss_res.get(), device_id, index_.get(), &option);
#else
auto gpu_index = faiss::gpu::index_cpu_to_gpu(res->faiss_res.get(), device_id, index_.get());
#endif
std::shared_ptr<faiss::Index> device_index;
device_index.reset(gpu_index);

116
core/src/index/knowhere/knowhere/index/vector_index/IndexIVFSQHybrid.cpp
View File

@ -17,19 +17,25 @@
// under the License.
#include "knowhere/index/vector_index/IndexIVFSQHybrid.h"
#include <utility>
#include "faiss/AutoTune.h"
#include "faiss/gpu/GpuAutoTune.h"
#include "faiss/gpu/GpuIndexIVF.h"
#include "knowhere/adapter/VectorAdapter.h"
#include "knowhere/common/Exception.h"
#include <utility>
#include <faiss/gpu/GpuCloner.h>
#include <faiss/gpu/GpuIndexIVF.h>
#include <faiss/index_factory.h>
namespace knowhere {
#ifdef CUSTOMIZATION
// std::mutex g_mutex;
IndexModelPtr
IVFSQHybrid::Train(const DatasetPtr& dataset, const Config& config) {
// std::lock_guard<std::mutex> lk(g_mutex);
auto build_cfg = std::dynamic_pointer_cast<IVFSQCfg>(config);
if (build_cfg != nullptr) {
build_cfg->CheckValid(); // throw exception
@ -63,23 +69,25 @@ IVFSQHybrid::Train(const DatasetPtr& dataset, const Config& config) {
VectorIndexPtr
IVFSQHybrid::CopyGpuToCpu(const Config& config) {
std::lock_guard<std::mutex> lk(mutex_);
if (auto device_idx = std::dynamic_pointer_cast<faiss::IndexIVF>(index_)) {
faiss::Index* device_index = index_.get();
faiss::Index* host_index = faiss::gpu::index_gpu_to_cpu(device_index);
std::shared_ptr<faiss::Index> new_index;
new_index.reset(host_index);
return std::make_shared<IVFSQHybrid>(new_index);
} else {
// TODO(linxj): why? jinhai
if (gpu_mode == 0) {
return std::make_shared<IVFSQHybrid>(index_);
}
std::lock_guard<std::mutex> lk(mutex_);
faiss::Index* device_index = index_.get();
faiss::Index* host_index = faiss::gpu::index_gpu_to_cpu(device_index);
std::shared_ptr<faiss::Index> new_index;
new_index.reset(host_index);
return std::make_shared<IVFSQHybrid>(new_index);
}
VectorIndexPtr
IVFSQHybrid::CopyCpuToGpu(const int64_t& device_id, const Config& config) {
if (gpu_mode != 0) {
KNOWHERE_THROW_MSG("Not a GpuIndex Type");
}
if (auto res = FaissGpuResourceMgr::GetInstance().GetRes(device_id)) {
ResScope rs(res, device_id, false);
faiss::gpu::GpuClonerOptions option;
@ -105,16 +113,26 @@ IVFSQHybrid::LoadImpl(const BinarySet& index_binary) {
FaissBaseIndex::LoadImpl(index_binary); // load on cpu
auto* ivf_index = dynamic_cast<faiss::IndexIVF*>(index_.get());
ivf_index->backup_quantizer();
gpu_mode = 0;
}
void
IVFSQHybrid::search_impl(int64_t n, const float* data, int64_t k, float* distances, int64_t* labels,
const Config& cfg) {
// std::lock_guard<std::mutex> lk(g_mutex);
// static int64_t search_count;
// ++search_count;
if (gpu_mode == 2) {
GPUIVF::search_impl(n, data, k, distances, labels, cfg);
} else if (gpu_mode == 1) {
ResScope rs(res_, gpu_id_);
IVF::search_impl(n, data, k, distances, labels, cfg);
// index_->search(n, (float*)data, k, distances, labels);
} else if (gpu_mode == 1) { // hybrid
if (auto res = FaissGpuResourceMgr::GetInstance().GetRes(quantizer_gpu_id_)) {
ResScope rs(res, quantizer_gpu_id_, true);
IVF::search_impl(n, data, k, distances, labels, cfg);
} else {
KNOWHERE_THROW_MSG("Hybrid Search Error, can't get gpu: " + std::to_string(quantizer_gpu_id_) + "resource");
}
} else if (gpu_mode == 0) {
IVF::search_impl(n, data, k, distances, labels, cfg);
}
@ -122,16 +140,18 @@ IVFSQHybrid::search_impl(int64_t n, const float* data, int64_t k, float* distanc
QuantizerPtr
IVFSQHybrid::LoadQuantizer(const Config& conf) {
// std::lock_guard<std::mutex> lk(g_mutex);
auto quantizer_conf = std::dynamic_pointer_cast<QuantizerCfg>(conf);
if (quantizer_conf != nullptr) {
if (quantizer_conf->mode != 1) {
KNOWHERE_THROW_MSG("mode only support 1 in this func");
}
}
gpu_id_ = quantizer_conf->gpu_id;
auto gpu_id = quantizer_conf->gpu_id;
if (auto res = FaissGpuResourceMgr::GetInstance().GetRes(gpu_id_)) {
ResScope rs(res, gpu_id_, false);
if (auto res = FaissGpuResourceMgr::GetInstance().GetRes(gpu_id)) {
ResScope rs(res, gpu_id, false);
faiss::gpu::GpuClonerOptions option;
option.allInGpu = true;
@ -140,7 +160,7 @@ IVFSQHybrid::LoadQuantizer(const Config& conf) {
index_composition->quantizer = nullptr;
index_composition->mode = quantizer_conf->mode; // only 1
auto gpu_index = faiss::gpu::index_cpu_to_gpu(res->faiss_res.get(), gpu_id_, index_composition, &option);
auto gpu_index = faiss::gpu::index_cpu_to_gpu(res->faiss_res.get(), gpu_id, index_composition, &option);
delete gpu_index;
auto q = std::make_shared<FaissIVFQuantizer>();
@ -148,16 +168,19 @@ IVFSQHybrid::LoadQuantizer(const Config& conf) {
auto& q_ptr = index_composition->quantizer;
q->size = q_ptr->d * q_ptr->getNumVecs() * sizeof(float);
q->quantizer = q_ptr;
q->gpu_id = gpu_id;
res_ = res;
gpu_mode = 1;
return q;
} else {
KNOWHERE_THROW_MSG("CopyCpuToGpu Error, can't get gpu: " + std::to_string(gpu_id_) + "resource");
KNOWHERE_THROW_MSG("CopyCpuToGpu Error, can't get gpu: " + std::to_string(gpu_id) + "resource");
}
}
void
IVFSQHybrid::SetQuantizer(const QuantizerPtr& q) {
// std::lock_guard<std::mutex> lk(g_mutex);
auto ivf_quantizer = std::dynamic_pointer_cast<FaissIVFQuantizer>(q);
if (ivf_quantizer == nullptr) {
KNOWHERE_THROW_MSG("Quantizer type error");
@ -170,20 +193,27 @@ IVFSQHybrid::SetQuantizer(const QuantizerPtr& q) {
// delete ivf_index->quantizer;
ivf_index->quantizer = ivf_quantizer->quantizer;
}
quantizer_gpu_id_ = ivf_quantizer->gpu_id;
gpu_mode = 1;
}
void
IVFSQHybrid::UnsetQuantizer() {
// std::lock_guard<std::mutex> lk(g_mutex);
auto* ivf_index = dynamic_cast<faiss::IndexIVF*>(index_.get());
if (ivf_index == nullptr) {
KNOWHERE_THROW_MSG("Index type error");
}
ivf_index->quantizer = nullptr;
quantizer_gpu_id_ = -1;
}
VectorIndexPtr
IVFSQHybrid::LoadData(const knowhere::QuantizerPtr& q, const Config& conf) {
// std::lock_guard<std::mutex> lk(g_mutex);
auto quantizer_conf = std::dynamic_pointer_cast<QuantizerCfg>(conf);
if (quantizer_conf != nullptr) {
if (quantizer_conf->mode != 2) {
@ -192,13 +222,11 @@ IVFSQHybrid::LoadData(const knowhere::QuantizerPtr& q, const Config& conf) {
} else {
KNOWHERE_THROW_MSG("conf error");
}
// if (quantizer_conf->gpu_id != gpu_id_) {
// KNOWHERE_THROW_MSG("quantizer and data must on the same gpu card");
// }
gpu_id_ = quantizer_conf->gpu_id;
if (auto res = FaissGpuResourceMgr::GetInstance().GetRes(gpu_id_)) {
ResScope rs(res, gpu_id_, false);
auto gpu_id = quantizer_conf->gpu_id;
if (auto res = FaissGpuResourceMgr::GetInstance().GetRes(gpu_id)) {
ResScope rs(res, gpu_id, false);
faiss::gpu::GpuClonerOptions option;
option.allInGpu = true;
@ -211,18 +239,20 @@ IVFSQHybrid::LoadData(const knowhere::QuantizerPtr& q, const Config& conf) {
index_composition->quantizer = ivf_quantizer->quantizer;
index_composition->mode = quantizer_conf->mode; // only 2
auto gpu_index = faiss::gpu::index_cpu_to_gpu(res->faiss_res.get(), gpu_id_, index_composition, &option);
auto gpu_index = faiss::gpu::index_cpu_to_gpu(res->faiss_res.get(), gpu_id, index_composition, &option);
std::shared_ptr<faiss::Index> new_idx;
new_idx.reset(gpu_index);
auto sq_idx = std::make_shared<IVFSQHybrid>(new_idx, gpu_id_, res);
auto sq_idx = std::make_shared<IVFSQHybrid>(new_idx, gpu_id, res);
return sq_idx;
} else {
KNOWHERE_THROW_MSG("CopyCpuToGpu Error, can't get gpu: " + std::to_string(gpu_id_) + "resource");
KNOWHERE_THROW_MSG("CopyCpuToGpu Error, can't get gpu: " + std::to_string(gpu_id) + "resource");
}
}
std::pair<VectorIndexPtr, QuantizerPtr>
IVFSQHybrid::CopyCpuToGpuWithQuantizer(const int64_t& device_id, const Config& config) {
// std::lock_guard<std::mutex> lk(g_mutex);
if (auto res = FaissGpuResourceMgr::GetInstance().GetRes(device_id)) {
ResScope rs(res, device_id, false);
faiss::gpu::GpuClonerOptions option;
@ -242,12 +272,29 @@ IVFSQHybrid::CopyCpuToGpuWithQuantizer(const int64_t& device_id, const Config& c
auto q = std::make_shared<FaissIVFQuantizer>();
q->quantizer = index_composition.quantizer;
q->size = index_composition.quantizer->d * index_composition.quantizer->getNumVecs() * sizeof(float);
q->gpu_id = device_id;
return std::make_pair(new_idx, q);
} else {
KNOWHERE_THROW_MSG("CopyCpuToGpu Error, can't get gpu: " + std::to_string(gpu_id_) + "resource");
}
}
void
IVFSQHybrid::set_index_model(IndexModelPtr model) {
std::lock_guard<std::mutex> lk(mutex_);
auto host_index = std::static_pointer_cast<IVFIndexModel>(model);
if (auto gpures = FaissGpuResourceMgr::GetInstance().GetRes(gpu_id_)) {
ResScope rs(gpures, gpu_id_, false);
auto device_index = faiss::gpu::index_cpu_to_gpu(gpures->faiss_res.get(), gpu_id_, host_index->index_.get());
index_.reset(device_index);
res_ = gpures;
gpu_mode = 2;
} else {
KNOWHERE_THROW_MSG("load index model error, can't get gpu_resource");
}
}
FaissIVFQuantizer::~FaissIVFQuantizer() {
if (quantizer != nullptr) {
delete quantizer;
@ -307,5 +354,10 @@ IVFSQHybrid::LoadImpl(const BinarySet& index_binary) {
GPUIVF::LoadImpl(index_binary);
}
void
IVFSQHybrid::set_index_model(IndexModelPtr model) {
GPUIVF::set_index_model(model);
}
#endif
} // namespace knowhere

7
core/src/index/knowhere/knowhere/index/vector_index/IndexIVFSQHybrid.h
View File

@ -17,7 +17,9 @@
#pragma once
#include <faiss/gpu/GpuIndexFlat.h>
#include <faiss/index_io.h>
#include <memory>
#include <utility>
@ -29,6 +31,7 @@ namespace knowhere {
#ifdef CUSTOMIZATION
struct FaissIVFQuantizer : public Quantizer {
faiss::gpu::GpuIndexFlat* quantizer = nullptr;
int64_t gpu_id;
~FaissIVFQuantizer() override;
};
@ -52,6 +55,9 @@ class IVFSQHybrid : public GPUIVFSQ {
}
public:
void
set_index_model(IndexModelPtr model) override;
QuantizerPtr
LoadQuantizer(const Config& conf);
@ -85,6 +91,7 @@ class IVFSQHybrid : public GPUIVFSQ {
protected:
int64_t gpu_mode = 0; // 0,1,2
int64_t quantizer_gpu_id_ = -1;
};
} // namespace knowhere

1
core/src/index/knowhere/knowhere/index/vector_index/VectorIndex.h
View File

@ -48,6 +48,7 @@ class VectorIndex : public Index {
virtual void
Seal() = 0;
// TODO(linxj): Deprecated
virtual VectorIndexPtr
Clone() = 0;

2
core/src/index/knowhere/knowhere/index/vector_index/helpers/FaissIO.h
View File

@ -17,7 +17,7 @@
#pragma once
#include <faiss/AuxIndexStructures.h>
#include <faiss/impl/io.h>
namespace knowhere {

2
core/src/index/thirdparty/versions.txt vendored
View File

@ -3,4 +3,4 @@ BOOST_VERSION=1.70.0
GTEST_VERSION=1.8.1
LAPACK_VERSION=v3.8.0
OPENBLAS_VERSION=v0.3.6
FAISS_VERSION=branch-0.2.1
FAISS_VERSION=branch-0.3.0

3
core/src/index/unittest/CMakeLists.txt
View File

@ -20,7 +20,7 @@ set(basic_libs
)
set(util_srcs
${MILVUS_ENGINE_SRC}/utils/easylogging++.cc
${MILVUS_ENGINE_SRC}/external/easyloggingpp/easylogging++.cc
${INDEX_SOURCE_DIR}/knowhere/knowhere/index/vector_index/helpers/FaissGpuResourceMgr.cpp
${INDEX_SOURCE_DIR}/knowhere/knowhere/index/vector_index/helpers/FaissIO.cpp
${INDEX_SOURCE_DIR}/knowhere/knowhere/index/vector_index/helpers/IndexParameter.cpp
@ -86,5 +86,6 @@ install(TARGETS test_gpuresource DESTINATION unittest)
install(TARGETS test_customized_index DESTINATION unittest)
#add_subdirectory(faiss_ori)
#add_subdirectory(faiss_benchmark)
add_subdirectory(test_nsg)

2
core/src/index/unittest/Helper.h
View File

@ -26,7 +26,7 @@
#include "knowhere/index/vector_index/IndexIVFSQ.h"
#include "knowhere/index/vector_index/IndexIVFSQHybrid.h"
constexpr int DEVICEID = 0;
int DEVICEID = 0;
constexpr int64_t DIM = 128;
constexpr int64_t NB = 10000;
constexpr int64_t NQ = 10;

24
core/src/index/unittest/faiss_benchmark/CMakeLists.txt Normal file
View File

@ -0,0 +1,24 @@
include_directories(${INDEX_SOURCE_DIR}/thirdparty)
include_directories(${INDEX_SOURCE_DIR}/include)
include_directories(/usr/local/cuda/include)
include_directories(/usr/local/hdf5/include)
link_directories(/usr/local/cuda/lib64)
link_directories(/usr/local/hdf5/lib)
set(unittest_libs
gtest gmock gtest_main gmock_main)
set(depend_libs
faiss openblas lapack hdf5
arrow ${ARROW_PREFIX}/lib/libjemalloc_pic.a
)
set(basic_libs
cudart cublas
gomp gfortran pthread
)
add_executable(test_faiss_benchmark faiss_benchmark_test.cpp)
target_link_libraries(test_faiss_benchmark ${depend_libs} ${unittest_libs} ${basic_libs})
install(TARGETS test_faiss_benchmark DESTINATION unittest)

25
core/src/index/unittest/faiss_benchmark/README.md Normal file
View File

@ -0,0 +1,25 @@
### To run this FAISS benchmark, please follow these steps:
#### Step 1:
Download the HDF5 source from:
https://support.hdfgroup.org/ftp/HDF5/releases/
and build/install to "/usr/local/hdf5".
#### Step 2:
Download HDF5 data files from:
https://github.com/erikbern/ann-benchmarks
#### Step 3:
Update 'milvus/core/src/index/unittest/CMakeLists.txt',
uncomment "#add_subdirectory(faiss_benchmark)".
#### Step 4:
Build Milvus with unittest enabled: "./build.sh -t Release -u",
binary 'test_faiss_benchmark' will be generated.
#### Step 5:
Put HDF5 data files into the same directory with binary 'test_faiss_benchmark'.
#### Step 6:
Run test binary 'test_faiss_benchmark'.

614
core/src/index/unittest/faiss_benchmark/faiss_benchmark_test.cpp Normal file
View File

@ -0,0 +1,614 @@
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.
#include <gtest/gtest.h>
#include <cassert>
#include <cmath>
#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <faiss/AutoTune.h>
#include <faiss/Index.h>
#include <faiss/IndexIVF.h>
#include <faiss/gpu/GpuAutoTune.h>
#include <faiss/gpu/GpuIndexFlat.h>
#include <faiss/gpu/GpuIndexIVFSQHybrid.h>
#include <faiss/gpu/StandardGpuResources.h>
#include <faiss/index_io.h>
#include <faiss/utils.h>
#include <hdf5.h>
#include <sys/stat.h>
#include <sys/time.h>
#include <sys/types.h>
#include <unistd.h>
#include <vector>
/*****************************************************
* To run this test, please download the HDF5 from
* https://support.hdfgroup.org/ftp/HDF5/releases/
* and install it to /usr/local/hdf5 .
*****************************************************/
double
elapsed() {
struct timeval tv;
gettimeofday(&tv, nullptr);
return tv.tv_sec + tv.tv_usec * 1e-6;
}
void*
hdf5_read(const char* file_name, const char* dataset_name, H5T_class_t dataset_class, size_t& d_out, size_t& n_out) {
hid_t file, dataset, datatype, dataspace, memspace;
H5T_class_t t_class; /* data type class */
H5T_order_t order; /* data order */
size_t size; /* size of the data element stored in file */
hsize_t dimsm[3]; /* memory space dimensions */
hsize_t dims_out[2]; /* dataset dimensions */
hsize_t count[2]; /* size of the hyperslab in the file */
hsize_t offset[2]; /* hyperslab offset in the file */
hsize_t count_out[3]; /* size of the hyperslab in memory */
hsize_t offset_out[3]; /* hyperslab offset in memory */
int rank;
void* data_out; /* output buffer */
/* Open the file and the dataset. */
file = H5Fopen(file_name, H5F_ACC_RDONLY, H5P_DEFAULT);
dataset = H5Dopen2(file, dataset_name, H5P_DEFAULT);
/*
* Get datatype and dataspace handles and then query
* dataset class, order, size, rank and dimensions.
*/
datatype = H5Dget_type(dataset); /* datatype handle */
t_class = H5Tget_class(datatype);
assert(t_class == dataset_class || !"Illegal dataset class type");
order = H5Tget_order(datatype);
switch (order) {
case H5T_ORDER_LE:
printf("Little endian order \n");
break;
case H5T_ORDER_BE:
printf("Big endian order \n");
break;
default:
printf("Illegal endian order \n");
break;
}
size = H5Tget_size(datatype);
printf("Data size is %d \n", (int)size);
dataspace = H5Dget_space(dataset); /* dataspace handle */
rank = H5Sget_simple_extent_ndims(dataspace);
H5Sget_simple_extent_dims(dataspace, dims_out, NULL);
n_out = dims_out[0];
d_out = dims_out[1];
printf("rank %d, dimensions %lu x %lu \n", rank, n_out, d_out);
/* Define hyperslab in the dataset. */
offset[0] = offset[1] = 0;
count[0] = dims_out[0];
count[1] = dims_out[1];
H5Sselect_hyperslab(dataspace, H5S_SELECT_SET, offset, NULL, count, NULL);
/* Define the memory dataspace. */
dimsm[0] = dims_out[0];
dimsm[1] = dims_out[1];
dimsm[2] = 1;
memspace = H5Screate_simple(3, dimsm, NULL);
/* Define memory hyperslab. */
offset_out[0] = offset_out[1] = offset_out[2] = 0;
count_out[0] = dims_out[0];
count_out[1] = dims_out[1];
count_out[2] = 1;
H5Sselect_hyperslab(memspace, H5S_SELECT_SET, offset_out, NULL, count_out, NULL);
/* Read data from hyperslab in the file into the hyperslab in memory and display. */
switch (t_class) {
case H5T_INTEGER:
data_out = new int[dims_out[0] * dims_out[1]];
H5Dread(dataset, H5T_NATIVE_INT, memspace, dataspace, H5P_DEFAULT, data_out);
break;
case H5T_FLOAT:
data_out = new float[dims_out[0] * dims_out[1]];
H5Dread(dataset, H5T_NATIVE_FLOAT, memspace, dataspace, H5P_DEFAULT, data_out);
break;
default:
printf("Illegal dataset class type\n");
break;
}
/* Close/release resources. */
H5Tclose(datatype);
H5Dclose(dataset);
H5Sclose(dataspace);
H5Sclose(memspace);
H5Fclose(file);
return data_out;
}
std::string
get_index_file_name(const std::string& ann_test_name, const std::string& index_key, int32_t data_loops) {
size_t pos = index_key.find_first_of(',', 0);
std::string file_name = ann_test_name;
file_name = file_name + "_" + index_key.substr(0, pos) + "_" + index_key.substr(pos + 1);
file_name = file_name + "_" + std::to_string(data_loops) + ".index";
return file_name;
}
bool
parse_ann_test_name(const std::string& ann_test_name, size_t& dim, faiss::MetricType& metric_type) {
size_t pos1, pos2;
if (ann_test_name.empty())
return false;
pos1 = ann_test_name.find_first_of('-', 0);
if (pos1 == std::string::npos)
return false;
pos2 = ann_test_name.find_first_of('-', pos1 + 1);
if (pos2 == std::string::npos)
return false;
dim = std::stoi(ann_test_name.substr(pos1 + 1, pos2 - pos1 - 1));
std::string metric_str = ann_test_name.substr(pos2 + 1);
if (metric_str == "angular") {
metric_type = faiss::METRIC_INNER_PRODUCT;
} else if (metric_str == "euclidean") {
metric_type = faiss::METRIC_L2;
} else {
return false;
}
return true;
}
int32_t
GetResultHitCount(const faiss::Index::idx_t* ground_index, const faiss::Index::idx_t* index, size_t ground_k, size_t k,
size_t nq, int32_t index_add_loops) {
assert(ground_k <= k);
int hit = 0;
for (int i = 0; i < nq; i++) {
// count the num of results exist in ground truth result set
// each result replicates INDEX_ADD_LOOPS times
for (int j_c = 0; j_c < ground_k; j_c++) {
int r_c = index[i * k + j_c];
int j_g = 0;
for (; j_g < ground_k / index_add_loops; j_g++) {
if (ground_index[i * ground_k + j_g] == r_c) {
hit++;
continue;
}
}
}
}
return hit;
}
void
test_ann_hdf5(const std::string& ann_test_name, const std::string& index_key, int32_t index_add_loops,
const std::vector<size_t>& nprobes, int32_t search_loops) {
double t0 = elapsed();
const std::string ann_file_name = ann_test_name + ".hdf5";
faiss::MetricType metric_type;
size_t dim;
if (!parse_ann_test_name(ann_test_name, dim, metric_type)) {
printf("Invalid ann test name: %s\n", ann_test_name.c_str());
return;
}
faiss::Index* index;
size_t d;
std::string index_file_name = get_index_file_name(ann_test_name, index_key, index_add_loops);
try {
index = faiss::read_index(index_file_name.c_str());
d = dim;
} catch (...) {
printf("Cannot read index file: %s\n", index_file_name.c_str());
printf("[%.3f s] Loading train set\n", elapsed() - t0);
size_t nb;
float* xb = (float*)hdf5_read(ann_file_name.c_str(), "train", H5T_FLOAT, d, nb);
assert(d == dim || !"dataset does not have correct dimension");
printf("[%.3f s] Preparing index \"%s\" d=%ld\n", elapsed() - t0, index_key.c_str(), d);
index = faiss::index_factory(d, index_key.c_str(), metric_type);
printf("[%.3f s] Training on %ld vectors\n", elapsed() - t0, nb);
index->train(nb, xb);
printf("[%.3f s] Loading database\n", elapsed() - t0);
// add index multiple times to get ~1G data set
for (int i = 0; i < index_add_loops; i++) {
printf("[%.3f s] Indexing database, size %ld*%ld\n", elapsed() - t0, nb, d);
index->add(nb, xb);
}
faiss::write_index(index, index_file_name.c_str());
delete[] xb;
}
size_t nq;
float* xq;
{
printf("[%.3f s] Loading queries\n", elapsed() - t0);
size_t d2;
xq = (float*)hdf5_read(ann_file_name.c_str(), "test", H5T_FLOAT, d2, nq);
assert(d == d2 || !"query does not have same dimension as train set");
}
size_t k; // nb of results per query in the GT
faiss::Index::idx_t* gt; // nq * k matrix of ground-truth nearest-neighbors
{
printf("[%.3f s] Loading ground truth for %ld queries\n", elapsed() - t0, nq);
// load ground-truth and convert int to long
size_t nq2;
int* gt_int = (int*)hdf5_read(ann_file_name.c_str(), "neighbors", H5T_INTEGER, k, nq2);
assert(nq2 == nq || !"incorrect nb of ground truth entries");
gt = new faiss::Index::idx_t[k * nq];
for (int i = 0; i < k * nq; i++) {
gt[i] = gt_int[i];
}
delete[] gt_int;
}
for (auto nprobe : nprobes) {
faiss::ParameterSpace params;
std::string nprobe_str = "nprobe=" + std::to_string(nprobe);
params.set_index_parameters(index, nprobe_str.c_str());
// output buffers
#if 1
const size_t NQ = 1000, K = 1000;
faiss::Index::idx_t* I = new faiss::Index::idx_t[NQ * K];
float* D = new float[NQ * K];
printf("\n%s | %s | nprobe=%lu\n", ann_test_name.c_str(), index_key.c_str(), nprobe);
printf("======================================================================================\n");
for (size_t t_nq = 10; t_nq <= NQ; t_nq *= 10) { // nq = {10, 100, 1000}
for (size_t t_k = 100; t_k <= K; t_k *= 10) { // k = {100, 1000}
faiss::indexIVF_stats.quantization_time = 0.0;
faiss::indexIVF_stats.search_time = 0.0;
double t_start = elapsed(), t_end;
for (int i = 0; i < search_loops; i++) {
index->search(t_nq, xq, t_k, D, I);
}
t_end = elapsed();
// k = 100 for ground truth
int32_t hit = GetResultHitCount(gt, I, k, t_k, t_nq, index_add_loops);
printf("nq = %4ld, k = %4ld, elapse = %.4fs (quant = %.4fs, search = %.4fs), R@ = %.4f\n", t_nq, t_k,
(t_end - t_start) / search_loops, faiss::indexIVF_stats.quantization_time / 1000 / search_loops,
faiss::indexIVF_stats.search_time / 1000 / search_loops,
(hit / float(t_nq * k / index_add_loops)));
}
}
printf("======================================================================================\n");
#else
printf("[%.3f s] Perform a search on %ld queries\n", elapsed() - t0, nq);
faiss::Index::idx_t* I = new faiss::Index::idx_t[nq * k];
float* D = new float[nq * k];
index->search(nq, xq, k, D, I);
printf("[%.3f s] Compute recalls\n", elapsed() - t0);
// evaluate result by hand.
int n_1 = 0, n_10 = 0, n_100 = 0;
for (int i = 0; i < nq; i++) {
int gt_nn = gt[i * k];
for (int j = 0; j < k; j++) {
if (I[i * k + j] == gt_nn) {
if (j < 1)
n_1++;
if (j < 10)
n_10++;
if (j < 100)
n_100++;
}
}
}
printf("R@1 = %.4f\n", n_1 / float(nq));
printf("R@10 = %.4f\n", n_10 / float(nq));
printf("R@100 = %.4f\n", n_100 / float(nq));
#endif
printf("[%.3f s] Search test done\n\n", elapsed() - t0);
delete[] I;
delete[] D;
}
delete[] xq;
delete[] gt;
delete index;
}
#ifdef CUSTOMIZATION
void
test_ivfsq8h(const std::string& ann_test_name, int32_t index_add_loops, const std::vector<size_t>& nprobes,
bool pure_gpu_mode, int32_t search_loops) {
double t0 = elapsed();
const std::string ann_file_name = ann_test_name + ".hdf5";
faiss::MetricType metric_type;
size_t dim;
if (!parse_ann_test_name(ann_test_name, dim, metric_type)) {
printf("Invalid ann test name: %s\n", ann_test_name.c_str());
return;
}
faiss::distance_compute_blas_threshold = 800;
faiss::gpu::StandardGpuResources res;
const std::string index_key = "IVF16384,SQ8Hybrid";
faiss::Index* cpu_index = nullptr;
size_t d;
std::string index_file_name = get_index_file_name(ann_test_name, index_key, index_add_loops);
try {
cpu_index = faiss::read_index(index_file_name.c_str());
d = dim;
} catch (...) {
printf("Cannot read index file: %s\n", index_file_name.c_str());
printf("[%.3f s] Loading train set\n", elapsed() - t0);
size_t nb;
float* xb = (float*)hdf5_read(ann_file_name.c_str(), "train", H5T_FLOAT, d, nb);
assert(d == dim || !"dataset does not have correct dimension");
printf("[%.3f s] Preparing index \"%s\" d=%ld\n", elapsed() - t0, index_key.c_str(), d);
faiss::Index* ori_index = faiss::index_factory(d, index_key.c_str(), metric_type);
auto device_index = faiss::gpu::index_cpu_to_gpu(&res, 0, ori_index);
printf("[%.3f s] Training on %ld vectors\n", elapsed() - t0, nb);
device_index->train(nb, xb);
printf("[%.3f s] Loading database\n", elapsed() - t0);
for (int i = 0; i < index_add_loops; i++) {
printf("[%.3f s] Indexing database, size %ld*%ld\n", elapsed() - t0, nb, d);
device_index->add(nb, xb);
}
cpu_index = faiss::gpu::index_gpu_to_cpu(device_index);
faiss::write_index(cpu_index, index_file_name.c_str());
delete[] xb;
}
faiss::IndexIVF* cpu_ivf_index = dynamic_cast<faiss::IndexIVF*>(cpu_index);
if (cpu_ivf_index != nullptr) {
cpu_ivf_index->to_readonly();
}
size_t nq;
float* xq;
{
printf("[%.3f s] Loading queries\n", elapsed() - t0);
size_t d2;
xq = (float*)hdf5_read(ann_file_name.c_str(), "test", H5T_FLOAT, d2, nq);
assert(d == d2 || !"query does not have same dimension as train set");
}
size_t k;
faiss::Index::idx_t* gt;
{
printf("[%.3f s] Loading ground truth for %ld queries\n", elapsed() - t0, nq);
size_t nq2;
int* gt_int = (int*)hdf5_read(ann_file_name.c_str(), "neighbors", H5T_INTEGER, k, nq2);
assert(nq2 == nq || !"incorrect nb of ground truth entries");
gt = new faiss::Index::idx_t[k * nq];
for (uint64_t i = 0; i < k * nq; ++i) {
gt[i] = gt_int[i];
}
delete[] gt_int;
}
faiss::gpu::GpuClonerOptions option;
option.allInGpu = true;
faiss::IndexComposition index_composition;
index_composition.index = cpu_index;
index_composition.quantizer = nullptr;
faiss::Index* index;
double copy_time;
if (!pure_gpu_mode) {
index_composition.mode = 1; // 0: all data, 1: copy quantizer, 2: copy data
index = faiss::gpu::index_cpu_to_gpu(&res, 0, &index_composition, &option);
delete index;
copy_time = elapsed();
index = faiss::gpu::index_cpu_to_gpu(&res, 0, &index_composition, &option);
delete index;
} else {
index_composition.mode = 2;
index = faiss::gpu::index_cpu_to_gpu(&res, 0, &index_composition, &option);
delete index;
copy_time = elapsed();
index = faiss::gpu::index_cpu_to_gpu(&res, 0, &index_composition, &option);
}
copy_time = elapsed() - copy_time;
printf("[%.3f s] Copy quantizer completed, cost %f s\n", elapsed() - t0, copy_time);
const size_t NQ = 1000, K = 1000;
if (!pure_gpu_mode) {
for (auto nprobe : nprobes) {
auto ivf_index = dynamic_cast<faiss::IndexIVF*>(cpu_index);
ivf_index->nprobe = nprobe;
auto is_gpu_flat_index = dynamic_cast<faiss::gpu::GpuIndexFlat*>(ivf_index->quantizer);
if (is_gpu_flat_index == nullptr) {
delete ivf_index->quantizer;
ivf_index->quantizer = index_composition.quantizer;
}
int64_t* I = new faiss::Index::idx_t[NQ * K];
float* D = new float[NQ * K];
printf("\n%s | %s-MIX | nprobe=%lu\n", ann_test_name.c_str(), index_key.c_str(), nprobe);
printf("======================================================================================\n");
for (size_t t_nq = 10; t_nq <= NQ; t_nq *= 10) { // nq = {10, 100, 1000}
for (size_t t_k = 100; t_k <= K; t_k *= 10) { // k = {100, 1000}
faiss::indexIVF_stats.quantization_time = 0.0;
faiss::indexIVF_stats.search_time = 0.0;
double t_start = elapsed(), t_end;
for (int32_t i = 0; i < search_loops; i++) {
cpu_index->search(t_nq, xq, t_k, D, I);
}
t_end = elapsed();
// k = 100 for ground truth
int32_t hit = GetResultHitCount(gt, I, k, t_k, t_nq, index_add_loops);
printf("nq = %4ld, k = %4ld, elapse = %.4fs (quant = %.4fs, search = %.4fs), R@ = %.4f\n", t_nq,
t_k, (t_end - t_start) / search_loops,
faiss::indexIVF_stats.quantization_time / 1000 / search_loops,
faiss::indexIVF_stats.search_time / 1000 / search_loops,
(hit / float(t_nq * k / index_add_loops)));
}
}
printf("======================================================================================\n");
printf("[%.3f s] Search test done\n\n", elapsed() - t0);
delete[] I;
delete[] D;
}
} else {
std::shared_ptr<faiss::Index> gpu_index_ivf_ptr = std::shared_ptr<faiss::Index>(index);
for (auto nprobe : nprobes) {
faiss::gpu::GpuIndexIVFSQHybrid* gpu_index_ivf_hybrid =
dynamic_cast<faiss::gpu::GpuIndexIVFSQHybrid*>(gpu_index_ivf_ptr.get());
gpu_index_ivf_hybrid->setNumProbes(nprobe);
int64_t* I = new faiss::Index::idx_t[NQ * K];
float* D = new float[NQ * K];
printf("\n%s | %s-GPU | nprobe=%lu\n", ann_test_name.c_str(), index_key.c_str(), nprobe);
printf("======================================================================================\n");
for (size_t t_nq = 10; t_nq <= NQ; t_nq *= 10) { // nq = {10, 100, 1000}
for (size_t t_k = 100; t_k <= K; t_k *= 10) { // k = {100, 1000}
faiss::indexIVF_stats.quantization_time = 0.0;
faiss::indexIVF_stats.search_time = 0.0;
double t_start = elapsed(), t_end;
for (int32_t i = 0; i < search_loops; i++) {
gpu_index_ivf_ptr->search(nq, xq, k, D, I);
}
t_end = elapsed();
// k = 100 for ground truth
int32_t hit = GetResultHitCount(gt, I, k, t_k, t_nq, index_add_loops);
printf("nq = %4ld, k = %4ld, elapse = %.4fs (quant = %.4fs, search = %.4fs), R@ = %.4f\n", t_nq,
t_k, (t_end - t_start) / search_loops,
faiss::indexIVF_stats.quantization_time / 1000 / search_loops,
faiss::indexIVF_stats.search_time / 1000 / search_loops,
(hit / float(t_nq * k / index_add_loops)));
}
}
printf("======================================================================================\n");
printf("[%.3f s] Search test done\n\n", elapsed() - t0);
delete[] I;
delete[] D;
}
}
delete[] xq;
delete[] gt;
delete cpu_index;
}
#endif
/************************************************************************************
* https://github.com/erikbern/ann-benchmarks
*
* Dataset Dimensions Train_size Test_size Neighbors Distance Download
* Fashion-
* MNIST 784 60,000 10,000 100 Euclidean HDF5 (217MB)
* GIST 960 1,000,000 1,000 100 Euclidean HDF5 (3.6GB)
* GloVe 100 1,183,514 10,000 100 Angular HDF5 (463MB)
* GloVe 200 1,183,514 10,000 100 Angular HDF5 (918MB)
* MNIST 784 60,000 10,000 100 Euclidean HDF5 (217MB)
* NYTimes 256 290,000 10,000 100 Angular HDF5 (301MB)
* SIFT 128 1,000,000 10,000 100 Euclidean HDF5 (501MB)
*************************************************************************************/
TEST(FAISSTEST, BENCHMARK) {
std::vector<size_t> param_nprobes = {8, 128};
const int32_t SEARCH_LOOPS = 5;
const int32_t SIFT_INSERT_LOOPS = 2; // insert twice to get ~1G data set
const int32_t GLOVE_INSERT_LOOPS = 1;
test_ann_hdf5("sift-128-euclidean", "IVF4096,Flat", SIFT_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
test_ann_hdf5("sift-128-euclidean", "IVF16384,SQ8", SIFT_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
#ifdef CUSTOMIZATION
test_ann_hdf5("sift-128-euclidean", "IVF16384,SQ8Hybrid", SIFT_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
test_ivfsq8h("sift-128-euclidean", SIFT_INSERT_LOOPS, param_nprobes, false, SEARCH_LOOPS);
test_ivfsq8h("sift-128-euclidean", SIFT_INSERT_LOOPS, param_nprobes, true, SEARCH_LOOPS);
#endif
test_ann_hdf5("glove-200-angular", "IVF4096,Flat", GLOVE_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
test_ann_hdf5("glove-200-angular", "IVF16384,SQ8", GLOVE_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
#ifdef CUSTOMIZATION
test_ann_hdf5("glove-200-angular", "IVF16384,SQ8Hybrid", GLOVE_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
test_ivfsq8h("glove-200-angular", GLOVE_INSERT_LOOPS, param_nprobes, false, SEARCH_LOOPS);
test_ivfsq8h("glove-200-angular", GLOVE_INSERT_LOOPS, param_nprobes, true, SEARCH_LOOPS);
#endif
}

119
core/src/index/unittest/test_customized_index.cpp
View File

@ -16,17 +16,23 @@
// under the License.
#include <gtest/gtest.h>
#include <thread>
#include "unittest/Helper.h"
#include "unittest/utils.h"
#include "knowhere/common/Timer.h"
class SingleIndexTest : public DataGen, public TestGpuIndexBase {
protected:
void
SetUp() override {
TestGpuIndexBase::SetUp();
Generate(DIM, NB, NQ);
k = K;
nb = 1000000;
nq = 1000;
dim = DIM;
Generate(dim, nb, nq);
k = 1000;
}
void
@ -119,4 +125,113 @@ TEST_F(SingleIndexTest, IVFSQHybrid) {
}
}
// TEST_F(SingleIndexTest, thread_safe) {
// assert(!xb.empty());
//
// index_type = "IVFSQHybrid";
// index_ = IndexFactory(index_type);
// auto base = ParamGenerator::GetInstance().Gen(ParameterType::ivfsq);
// auto conf = std::dynamic_pointer_cast<knowhere::IVFSQCfg>(base);
// conf->nlist = 16384;
// conf->k = k;
// conf->nprobe = 10;
// conf->d = dim;
// auto preprocessor = index_->BuildPreprocessor(base_dataset, conf);
// index_->set_preprocessor(preprocessor);
//
// auto model = index_->Train(base_dataset, conf);
// index_->set_index_model(model);
// index_->Add(base_dataset, conf);
// EXPECT_EQ(index_->Count(), nb);
// EXPECT_EQ(index_->Dimension(), dim);
//
// auto binaryset = index_->Serialize();
//
//
//
// auto cpu_idx = std::make_shared<knowhere::IVFSQHybrid>(DEVICEID);
// cpu_idx->Load(binaryset);
// auto pair = cpu_idx->CopyCpuToGpuWithQuantizer(DEVICEID, conf);
// auto quantizer = pair.second;
//
// auto quantizer_conf = std::make_shared<knowhere::QuantizerCfg>();
// quantizer_conf->mode = 2; // only copy data
// quantizer_conf->gpu_id = DEVICEID;
//
// auto CopyAllToGpu = [&](int64_t search_count, bool do_search = false) {
// for (int i = 0; i < search_count; ++i) {
// auto gpu_idx = cpu_idx->CopyCpuToGpu(DEVICEID, conf);
// if (do_search) {
// auto result = gpu_idx->Search(query_dataset, conf);
// AssertAnns(result, nq, conf->k);
// }
// }
// };
//
// auto hybrid_qt_idx = std::make_shared<knowhere::IVFSQHybrid>(DEVICEID);
// hybrid_qt_idx->Load(binaryset);
// auto SetQuantizerDoSearch = [&](int64_t search_count) {
// for (int i = 0; i < search_count; ++i) {
// hybrid_qt_idx->SetQuantizer(quantizer);
// auto result = hybrid_qt_idx->Search(query_dataset, conf);
// AssertAnns(result, nq, conf->k);
// // PrintResult(result, nq, k);
// hybrid_qt_idx->UnsetQuantizer();
// }
// };
//
// auto hybrid_data_idx = std::make_shared<knowhere::IVFSQHybrid>(DEVICEID);
// hybrid_data_idx->Load(binaryset);
// auto LoadDataDoSearch = [&](int64_t search_count, bool do_search = false) {
// for (int i = 0; i < search_count; ++i) {
// auto hybrid_idx = hybrid_data_idx->LoadData(quantizer, quantizer_conf);
// if (do_search) {
// auto result = hybrid_idx->Search(query_dataset, conf);
//// AssertAnns(result, nq, conf->k);
// }
// }
// };
//
// knowhere::TimeRecorder tc("");
// CopyAllToGpu(2000/2, false);
// tc.RecordSection("CopyAllToGpu witout search");
// CopyAllToGpu(400/2, true);
// tc.RecordSection("CopyAllToGpu with search");
// SetQuantizerDoSearch(6);
// tc.RecordSection("SetQuantizer with search");
// LoadDataDoSearch(2000/2, false);
// tc.RecordSection("LoadData without search");
// LoadDataDoSearch(400/2, true);
// tc.RecordSection("LoadData with search");
//
// {
// std::thread t1(CopyAllToGpu, 2000, false);
// std::thread t2(CopyAllToGpu, 400, true);
// t1.join();
// t2.join();
// }
//
// {
// std::thread t1(SetQuantizerDoSearch, 12);
// std::thread t2(CopyAllToGpu, 400, true);
// t1.join();
// t2.join();
// }
//
// {
// std::thread t1(SetQuantizerDoSearch, 12);
// std::thread t2(LoadDataDoSearch, 400, true);
// t1.join();
// t2.join();
// }
//
// {
// std::thread t1(LoadDataDoSearch, 2000, false);
// std::thread t2(LoadDataDoSearch, 400, true);
// t1.join();
// t2.join();
// }
//
//}
#endif

41
core/src/index/unittest/test_ivf.cpp
View File

@ -20,19 +20,12 @@
#include <iostream>
#include <thread>
#include <faiss/AutoTune.h>
#include <faiss/gpu/GpuAutoTune.h>
#include <faiss/gpu/GpuIndexIVFFlat.h>
#include "knowhere/common/Exception.h"
#include "knowhere/common/Timer.h"
#include "knowhere/index/vector_index/IndexGPUIVF.h"
#include "knowhere/index/vector_index/IndexGPUIVFPQ.h"
#include "knowhere/index/vector_index/IndexGPUIVFSQ.h"
#include "knowhere/index/vector_index/IndexIVF.h"
#include "knowhere/index/vector_index/IndexIVFPQ.h"
#include "knowhere/index/vector_index/IndexIVFSQ.h"
#include "knowhere/index/vector_index/IndexIVFSQHybrid.h"
#include "knowhere/index/vector_index/helpers/Cloner.h"
#include "unittest/Helper.h"
@ -51,6 +44,9 @@ class IVFTest : public DataGen, public TestWithParam<::std::tuple<std::string, P
ParameterType parameter_type;
std::tie(index_type, parameter_type) = GetParam();
// Init_with_default();
// nb = 1000000;
// nq = 1000;
// k = 1000;
Generate(DIM, NB, NQ);
index_ = IndexFactory(index_type);
conf = ParamGenerator::GetInstance().Gen(parameter_type);
@ -61,16 +57,6 @@ class IVFTest : public DataGen, public TestWithParam<::std::tuple<std::string, P
knowhere::FaissGpuResourceMgr::GetInstance().Free();
}
knowhere::VectorIndexPtr
ChooseTodo() {
std::vector<std::string> gpu_idx{"GPUIVFSQ"};
auto finder = std::find(gpu_idx.cbegin(), gpu_idx.cend(), index_type);
if (finder != gpu_idx.cend()) {
return knowhere::cloner::CopyCpuToGpu(index_, DEVICEID, knowhere::Config());
}
return index_;
}
protected:
std::string index_type;
knowhere::Config conf;
@ -100,8 +86,7 @@ TEST_P(IVFTest, ivf_basic) {
EXPECT_EQ(index_->Count(), nb);
EXPECT_EQ(index_->Dimension(), dim);
auto new_idx = ChooseTodo();
auto result = new_idx->Search(query_dataset, conf);
auto result = index_->Search(query_dataset, conf);
AssertAnns(result, nq, conf->k);
// PrintResult(result, nq, k);
}
@ -134,8 +119,7 @@ TEST_P(IVFTest, ivf_serialize) {
index_->set_index_model(model);
index_->Add(base_dataset, conf);
auto new_idx = ChooseTodo();
auto result = new_idx->Search(query_dataset, conf);
auto result = index_->Search(query_dataset, conf);
AssertAnns(result, nq, conf->k);
}
@ -159,8 +143,7 @@ TEST_P(IVFTest, ivf_serialize) {
index_->Load(binaryset);
EXPECT_EQ(index_->Count(), nb);
EXPECT_EQ(index_->Dimension(), dim);
auto new_idx = ChooseTodo();
auto result = new_idx->Search(query_dataset, conf);
auto result = index_->Search(query_dataset, conf);
AssertAnns(result, nq, conf->k);
}
}
@ -176,8 +159,7 @@ TEST_P(IVFTest, clone_test) {
index_->Add(base_dataset, conf);
EXPECT_EQ(index_->Count(), nb);
EXPECT_EQ(index_->Dimension(), dim);
auto new_idx = ChooseTodo();
auto result = new_idx->Search(query_dataset, conf);
auto result = index_->Search(query_dataset, conf);
AssertAnns(result, nq, conf->k);
// PrintResult(result, nq, k);
@ -210,12 +192,6 @@ TEST_P(IVFTest, clone_test) {
// }
// }
{
if (index_type == "IVFSQHybrid") {
return;
}
}
{
// copy from gpu to cpu
std::vector<std::string> support_idx_vec{"GPUIVF", "GPUIVFSQ", "IVFSQHybrid"};
@ -277,8 +253,7 @@ TEST_P(IVFTest, gpu_seal_test) {
index_->Add(base_dataset, conf);
EXPECT_EQ(index_->Count(), nb);
EXPECT_EQ(index_->Dimension(), dim);
auto new_idx = ChooseTodo();
auto result = new_idx->Search(query_dataset, conf);
auto result = index_->Search(query_dataset, conf);
AssertAnns(result, nq, conf->k);
auto cpu_idx = knowhere::cloner::CopyGpuToCpu(index_, knowhere::Config());

4
core/src/main.cpp
View File

@ -22,12 +22,12 @@
#include <cstring>
#include <string>
#include "../version.h"
#include "external/easyloggingpp/easylogging++.h"
#include "metrics/Metrics.h"
#include "server/Server.h"
#include "src/version.h"
#include "utils/CommonUtil.h"
#include "utils/SignalUtil.h"
#include "utils/easylogging++.h"
INITIALIZE_EASYLOGGINGPP

2
core/src/scheduler/Algorithm.cpp
View File

@ -54,7 +54,7 @@ ShortestPath(const ResourcePtr& src, const ResourcePtr& dest, const ResourceMgrP
auto cur_neighbours = cur_node->GetNeighbours();
for (auto& neighbour : cur_neighbours) {
auto neighbour_res = std::static_pointer_cast<Resource>(neighbour.neighbour_node.lock());
auto neighbour_res = std::static_pointer_cast<Resource>(neighbour.neighbour_node);
dis_matrix[name_id_map.at(res->name())][name_id_map.at(neighbour_res->name())] =
neighbour.connection.transport_cost();
}

25
core/src/scheduler/BuildMgr.h
View File

@ -34,27 +34,30 @@ namespace scheduler {
class BuildMgr {
public:
explicit BuildMgr(int64_t numoftasks) : numoftasks_(numoftasks) {
explicit BuildMgr(int64_t concurrent_limit) : available_(concurrent_limit) {
}
public:
void
Put() {
++numoftasks_;
std::lock_guard<std::mutex> lock(mutex_);
++available_;
}
void
take() {
--numoftasks_;
}
int64_t
numoftasks() {
return (int64_t)numoftasks_;
bool
Take() {
std::lock_guard<std::mutex> lock(mutex_);
if (available_ < 1) {
return false;
} else {
--available_;
return true;
}
}
private:
std::atomic_long numoftasks_;
std::int64_t available_;
std::mutex mutex_;
};
using BuildMgrPtr = std::shared_ptr<BuildMgr>;

119
core/src/scheduler/CircleQueue.h Normal file
View File

@ -0,0 +1,119 @@
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.
#pragma once
#include <atomic>
#include <condition_variable>
#include <deque>
#include <list>
#include <memory>
#include <mutex>
#include <queue>
#include <string>
#include <thread>
#include <unordered_map>
#include <utility>
#include <vector>
namespace milvus {
namespace scheduler {
template <typename T>
class CircleQueue {
using value_type = T;
using atomic_size_type = std::atomic_ullong;
using size_type = uint64_t;
using const_reference = const value_type&;
#define MEMORY_ORDER (std::memory_order::memory_order_seq_cst)
public:
explicit CircleQueue(size_type cap) : data_(cap, nullptr), capacity_(cap), front_() {
front_.store(cap - 1, MEMORY_ORDER);
}
CircleQueue() = delete;
CircleQueue(const CircleQueue& q) = delete;
CircleQueue(CircleQueue&& q) = delete;
public:
const_reference operator[](size_type n) {
return data_[n % capacity_];
}
size_type
front() {
return front_.load(MEMORY_ORDER);
}
size_type
rear() {
return rear_;
}
size_type
size() {
return size_;
}
size_type
capacity() {
return capacity_;
}
void
set_front(uint64_t last_finish) {
if (last_finish == rear_) {
throw;
}
front_.store(last_finish % capacity_, MEMORY_ORDER);
}
void
put(const value_type& x) {
if ((rear_) % capacity_ == front_.load(MEMORY_ORDER)) {
throw;
}
data_[rear_] = x;
rear_ = ++rear_ % capacity_;
if (size_ < capacity_) {
++size_;
}
}
void
put(value_type&& x) {
if ((rear_) % capacity_ == front_.load(MEMORY_ORDER)) {
throw;
}
data_[rear_] = std::move(x);
rear_ = ++rear_ % capacity_;
if (size_ < capacity_) {
++size_;
}
}
private:
std::vector<value_type> data_;
size_type capacity_;
atomic_size_type front_;
size_type rear_ = 0;
size_type size_ = 0;
#undef MEMORY_ORDER
};
} // namespace scheduler
} // namespace milvus

9
core/src/scheduler/JobMgr.cpp
View File

@ -49,6 +49,15 @@ JobMgr::Stop() {
}
}
json
JobMgr::Dump() const {
json ret{
{"running", running_},
{"event_queue_length", queue_.size()},
};
return ret;
}
void
JobMgr::Put(const JobPtr& job) {
{

6
core/src/scheduler/JobMgr.h
View File

@ -28,13 +28,14 @@
#include <vector>
#include "ResourceMgr.h"
#include "interface/interfaces.h"
#include "job/Job.h"
#include "task/Task.h"
namespace milvus {
namespace scheduler {
class JobMgr {
class JobMgr : public interface::dumpable {
public:
explicit JobMgr(ResourceMgrPtr res_mgr);
@ -44,6 +45,9 @@ class JobMgr {
void
Stop();
json
Dump() const override;
public:
void
Put(const JobPtr& job);

26
core/src/scheduler/ResourceMgr.cpp
View File

@ -170,16 +170,20 @@ ResourceMgr::GetNumGpuResource() const {
return num;
}
std::string
ResourceMgr::Dump() {
std::stringstream ss;
ss << "ResourceMgr contains " << resources_.size() << " resources." << std::endl;
json
ResourceMgr::Dump() const {
json resources{};
for (auto& res : resources_) {
ss << res->Dump();
resources.push_back(res->Dump());
}
return ss.str();
json ret{
{"number_of_resource", resources_.size()},
{"number_of_disk_resource", disk_resources_.size()},
{"number_of_cpu_resource", cpu_resources_.size()},
{"number_of_gpu_resource", gpu_resources_.size()},
{"resources", resources},
};
return ret;
}
std::string
@ -187,9 +191,9 @@ ResourceMgr::DumpTaskTables() {
std::stringstream ss;
ss << ">>>>>>>>>>>>>>>ResourceMgr::DumpTaskTable<<<<<<<<<<<<<<<" << std::endl;
for (auto& resource : resources_) {
ss << resource->Dump() << std::endl;
ss << resource->task_table().Dump();
ss << resource->Dump() << std::endl << std::endl;
ss << resource->name() << std::endl;
ss << resource->task_table().Dump().dump();
ss << resource->name() << std::endl << std::endl;
}
return ss.str();
}

8
core/src/scheduler/ResourceMgr.h
View File

@ -25,13 +25,14 @@
#include <utility>
#include <vector>
#include "interface/interfaces.h"
#include "resource/Resource.h"
#include "utils/Log.h"
namespace milvus {
namespace scheduler {
class ResourceMgr {
class ResourceMgr : public interface::dumpable {
public:
ResourceMgr() = default;
@ -74,7 +75,6 @@ class ResourceMgr {
return gpu_resources_;
}
// TODO(wxyu): why return shared pointer
inline std::vector<ResourcePtr>
GetAllResources() {
return resources_;
@ -103,8 +103,8 @@ class ResourceMgr {
public:
/******** Utility Functions ********/
std::string
Dump();
json
Dump() const override;
std::string
DumpTaskTables();

70
core/src/scheduler/SchedInst.cpp
View File

@ -82,79 +82,9 @@ load_simple_config() {
}
}
void
load_advance_config() {
// try {
// server::ConfigNode &config = server::Config::GetInstance().GetConfig(server::CONFIG_RESOURCE);
//
// if (config.GetChildren().empty()) throw "resource_config null exception";
//
// auto resources = config.GetChild(server::CONFIG_RESOURCES).GetChildren();
//
// if (resources.empty()) throw "Children of resource_config null exception";
//
// for (auto &resource : resources) {
// auto &resname = resource.first;
// auto &resconf = resource.second;
// auto type = resconf.GetValue(server::CONFIG_RESOURCE_TYPE);
//// auto memory = resconf.GetInt64Value(server::CONFIG_RESOURCE_MEMORY);
// auto device_id = resconf.GetInt64Value(server::CONFIG_RESOURCE_DEVICE_ID);
//// auto enable_loader = resconf.GetBoolValue(server::CONFIG_RESOURCE_ENABLE_LOADER);
// auto enable_loader = true;
// auto enable_executor = resconf.GetBoolValue(server::CONFIG_RESOURCE_ENABLE_EXECUTOR);
// auto pinned_memory = resconf.GetInt64Value(server::CONFIG_RESOURCE_PIN_MEMORY);
// auto temp_memory = resconf.GetInt64Value(server::CONFIG_RESOURCE_TEMP_MEMORY);
// auto resource_num = resconf.GetInt64Value(server::CONFIG_RESOURCE_NUM);
//
// auto res = ResMgrInst::GetInstance()->Add(ResourceFactory::Create(resname,
// type,
// device_id,
// enable_loader,
// enable_executor));
//
// if (res.lock()->type() == ResourceType::GPU) {
// auto pinned_memory = resconf.GetInt64Value(server::CONFIG_RESOURCE_PIN_MEMORY, 300);
// auto temp_memory = resconf.GetInt64Value(server::CONFIG_RESOURCE_TEMP_MEMORY, 300);
// auto resource_num = resconf.GetInt64Value(server::CONFIG_RESOURCE_NUM, 2);
// pinned_memory = 1024 * 1024 * pinned_memory;
// temp_memory = 1024 * 1024 * temp_memory;
// knowhere::FaissGpuResourceMgr::GetInstance().InitDevice(device_id,
// pinned_memory,
// temp_memory,
// resource_num);
// }
// }
//
// knowhere::FaissGpuResourceMgr::GetInstance().InitResource();
//
// auto connections = config.GetChild(server::CONFIG_RESOURCE_CONNECTIONS).GetChildren();
// if (connections.empty()) throw "connections config null exception";
// for (auto &conn : connections) {
// auto &connect_name = conn.first;
// auto &connect_conf = conn.second;
// auto connect_speed = connect_conf.GetInt64Value(server::CONFIG_SPEED_CONNECTIONS);
// auto connect_endpoint = connect_conf.GetValue(server::CONFIG_ENDPOINT_CONNECTIONS);
//
// std::string delimiter = "===";
// std::string left = connect_endpoint.substr(0, connect_endpoint.find(delimiter));
// std::string right = connect_endpoint.substr(connect_endpoint.find(delimiter) + 3,
// connect_endpoint.length());
//
// auto connection = Connection(connect_name, connect_speed);
// ResMgrInst::GetInstance()->Connect(left, right, connection);
// }
// } catch (const char *msg) {
// SERVER_LOG_ERROR << msg;
// // TODO(wxyu): throw exception instead
// exit(-1);
//// throw std::exception();
// }
}
void
StartSchedulerService() {
load_simple_config();
// load_advance_config();
ResMgrInst::GetInstance()->Start();
SchedInst::GetInstance()->Start();
JobMgrInst::GetInstance()->Start();

1
core/src/scheduler/SchedInst.h
View File

@ -94,6 +94,7 @@ class OptimizerInst {
std::lock_guard<std::mutex> lock(mutex_);
if (instance == nullptr) {
std::vector<PassPtr> pass_list;
pass_list.push_back(std::make_shared<LargeSQ8HPass>());
pass_list.push_back(std::make_shared<HybridPass>());
instance = std::make_shared<Optimizer>(pass_list);
}

74
core/src/scheduler/Scheduler.cpp
View File

@ -26,10 +26,8 @@
namespace milvus {
namespace scheduler {
Scheduler::Scheduler(ResourceMgrWPtr res_mgr) : running_(false), res_mgr_(std::move(res_mgr)) {
if (auto mgr = res_mgr_.lock()) {
mgr->RegisterSubscriber(std::bind(&Scheduler::PostEvent, this, std::placeholders::_1));
}
Scheduler::Scheduler(ResourceMgrPtr res_mgr) : running_(false), res_mgr_(std::move(res_mgr)) {
res_mgr_->RegisterSubscriber(std::bind(&Scheduler::PostEvent, this, std::placeholders::_1));
event_register_.insert(std::make_pair(static_cast<uint64_t>(EventType::START_UP),
std::bind(&Scheduler::OnStartUp, this, std::placeholders::_1)));
event_register_.insert(std::make_pair(static_cast<uint64_t>(EventType::LOAD_COMPLETED),
@ -40,6 +38,10 @@ Scheduler::Scheduler(ResourceMgrWPtr res_mgr) : running_(false), res_mgr_(std::m
std::bind(&Scheduler::OnFinishTask, this, std::placeholders::_1)));
}
Scheduler::~Scheduler() {
res_mgr_ = nullptr;
}
void
Scheduler::Start() {
running_ = true;
@ -66,9 +68,13 @@ Scheduler::PostEvent(const EventPtr& event) {
event_cv_.notify_one();
}
std::string
Scheduler::Dump() {
return std::string();
json
Scheduler::Dump() const {
json ret{
{"running", running_},
{"event_queue_length", event_queue_.size()},
};
return ret;
}
void
@ -96,51 +102,45 @@ Scheduler::Process(const EventPtr& event) {
void
Scheduler::OnLoadCompleted(const EventPtr& event) {
auto load_completed_event = std::static_pointer_cast<LoadCompletedEvent>(event);
if (auto resource = event->resource_.lock()) {
resource->WakeupExecutor();
auto task_table_type = load_completed_event->task_table_item_->task->label()->Type();
switch (task_table_type) {
case TaskLabelType::DEFAULT: {
Action::DefaultLabelTaskScheduler(res_mgr_, resource, load_completed_event);
break;
}
case TaskLabelType::SPECIFIED_RESOURCE: {
Action::SpecifiedResourceLabelTaskScheduler(res_mgr_, resource, load_completed_event);
break;
}
case TaskLabelType::BROADCAST: {
if (resource->HasExecutor() == false) {
load_completed_event->task_table_item_->Move();
}
Action::PushTaskToAllNeighbour(load_completed_event->task_table_item_->task, resource);
break;
}
default: { break; }
auto resource = event->resource_;
resource->WakeupExecutor();
auto task_table_type = load_completed_event->task_table_item_->task->label()->Type();
switch (task_table_type) {
case TaskLabelType::DEFAULT: {
Action::DefaultLabelTaskScheduler(res_mgr_, resource, load_completed_event);
break;
}
resource->WakeupLoader();
case TaskLabelType::SPECIFIED_RESOURCE: {
Action::SpecifiedResourceLabelTaskScheduler(res_mgr_, resource, load_completed_event);
break;
}
case TaskLabelType::BROADCAST: {
if (resource->HasExecutor() == false) {
load_completed_event->task_table_item_->Move();
}
Action::PushTaskToAllNeighbour(load_completed_event->task_table_item_->task, resource);
break;
}
default: { break; }
}
resource->WakeupLoader();
}
void
Scheduler::OnStartUp(const EventPtr& event) {
if (auto resource = event->resource_.lock()) {
resource->WakeupLoader();
}
event->resource_->WakeupLoader();
}
void
Scheduler::OnFinishTask(const EventPtr& event) {
if (auto resource = event->resource_.lock()) {
resource->WakeupLoader();
}
event->resource_->WakeupLoader();
}
void
Scheduler::OnTaskTableUpdated(const EventPtr& event) {
if (auto resource = event->resource_.lock()) {
resource->WakeupLoader();
}
event->resource_->WakeupLoader();
}
} // namespace scheduler

17
core/src/scheduler/Scheduler.h
View File

@ -25,16 +25,18 @@
#include <unordered_map>
#include "ResourceMgr.h"
#include "interface/interfaces.h"
#include "resource/Resource.h"
#include "utils/Log.h"
namespace milvus {
namespace scheduler {
// TODO(wxyu): refactor, not friendly to unittest, logical in framework code
class Scheduler {
class Scheduler : public interface::dumpable {
public:
explicit Scheduler(ResourceMgrWPtr res_mgr);
explicit Scheduler(ResourceMgrPtr res_mgr);
~Scheduler();
Scheduler(const Scheduler&) = delete;
Scheduler(Scheduler&&) = delete;
@ -57,11 +59,8 @@ class Scheduler {
void
PostEvent(const EventPtr& event);
/*
* Dump as string;
*/
std::string
Dump();
json
Dump() const override;
private:
/******** Events ********/
@ -121,7 +120,7 @@ class Scheduler {
std::unordered_map<uint64_t, std::function<void(EventPtr)>> event_register_;
ResourceMgrWPtr res_mgr_;
ResourceMgrPtr res_mgr_;
std::queue<EventPtr> event_queue_;
std::thread worker_thread_;
std::mutex event_mutex_;

105
core/src/scheduler/TaskTable.cpp
View File

@ -20,6 +20,7 @@
#include "event/TaskTableUpdatedEvent.h"
#include "scheduler/SchedInst.h"
#include "utils/Log.h"
#include "utils/TimeRecorder.h"
#include <ctime>
#include <sstream>
@ -53,7 +54,7 @@ ToString(TaskTableItemState state) {
}
json
TaskTimestamp::Dump() {
TaskTimestamp::Dump() const {
json ret{
{"start", start}, {"load", load}, {"loaded", loaded}, {"execute", execute},
{"executed", executed}, {"move", move}, {"moved", moved}, {"finish", finish},
@ -141,7 +142,7 @@ TaskTableItem::Moved() {
}
json
TaskTableItem::Dump() {
TaskTableItem::Dump() const {
json ret{
{"id", id},
{"task", (int64_t)task.get()},
@ -153,7 +154,42 @@ TaskTableItem::Dump() {
std::vector<uint64_t>
TaskTable::PickToLoad(uint64_t limit) {
std::lock_guard<std::mutex> lock(mutex_);
#if 1
TimeRecorder rc("");
std::vector<uint64_t> indexes;
bool cross = false;
uint64_t available_begin = table_.front() + 1;
for (uint64_t i = 0, loaded_count = 0, pick_count = 0; i < table_.size() && pick_count < limit; ++i) {
auto index = available_begin + i;
if (not table_[index])
break;
if (index % table_.capacity() == table_.rear())
break;
if (not cross && table_[index]->IsFinish()) {
table_.set_front(index);
} else if (table_[index]->state == TaskTableItemState::LOADED) {
cross = true;
++loaded_count;
if (loaded_count > 2)
return std::vector<uint64_t>();
} else if (table_[index]->state == TaskTableItemState::START) {
auto task = table_[index]->task;
// if task is a build index task, limit it
if (task->Type() == TaskType::BuildIndexTask && task->path().Current() == "cpu") {
if (not BuildMgrInst::GetInstance()->Take()) {
continue;
}
}
cross = true;
indexes.push_back(index);
++pick_count;
}
}
rc.ElapseFromBegin("PickToLoad ");
return indexes;
#else
size_t count = 0;
for (uint64_t j = last_finish_ + 1; j < table_.size(); ++j) {
if (not table_[j]) {
@ -197,34 +233,44 @@ TaskTable::PickToLoad(uint64_t limit) {
}
}
return indexes;
#endif
}
std::vector<uint64_t>
TaskTable::PickToExecute(uint64_t limit) {
std::lock_guard<std::mutex> lock(mutex_);
TimeRecorder rc("");
std::vector<uint64_t> indexes;
bool cross = false;
for (uint64_t i = last_finish_ + 1, count = 0; i < table_.size() && count < limit; ++i) {
if (not cross && table_[i]->IsFinish()) {
last_finish_ = i;
} else if (table_[i]->state == TaskTableItemState::LOADED) {
uint64_t available_begin = table_.front() + 1;
for (uint64_t i = 0, pick_count = 0; i < table_.size() && pick_count < limit; ++i) {
uint64_t index = available_begin + i;
if (not table_[index]) {
break;
}
if (index % table_.capacity() == table_.rear()) {
break;
}
if (not cross && table_[index]->IsFinish()) {
table_.set_front(index);
} else if (table_[index]->state == TaskTableItemState::LOADED) {
cross = true;
indexes.push_back(i);
++count;
indexes.push_back(index);
++pick_count;
}
}
rc.ElapseFromBegin("PickToExecute ");
return indexes;
}
void
TaskTable::Put(TaskPtr task) {
std::lock_guard<std::mutex> lock(mutex_);
auto item = std::make_shared<TaskTableItem>();
item->id = id_++;
item->task = std::move(task);
item->state = TaskTableItemState::START;
item->timestamp.start = get_current_timestamp();
table_.push_back(item);
table_.put(std::move(item));
if (subscriber_) {
subscriber_();
}
@ -232,42 +278,35 @@ TaskTable::Put(TaskPtr task) {
void
TaskTable::Put(std::vector<TaskPtr>& tasks) {
std::lock_guard<std::mutex> lock(mutex_);
for (auto& task : tasks) {
auto item = std::make_shared<TaskTableItem>();
item->id = id_++;
item->task = std::move(task);
item->state = TaskTableItemState::START;
item->timestamp.start = get_current_timestamp();
table_.push_back(item);
table_.put(std::move(item));
}
if (subscriber_) {
subscriber_();
}
}
TaskTableItemPtr
TaskTable::Get(uint64_t index) {
std::lock_guard<std::mutex> lock(mutex_);
return table_[index];
size_t
TaskTable::TaskToExecute() {
size_t count = 0;
auto begin = table_.front() + 1;
for (size_t i = 0; i < table_.size(); ++i) {
auto index = begin + i;
if (table_[index]->state == TaskTableItemState::LOADED) {
++count;
}
}
return count;
}
// void
// TaskTable::Clear() {
//// find first task is NOT (done or moved), erase from begin to it;
//// auto iterator = table_.begin();
//// while (iterator->state == TaskTableItemState::EXECUTED or
//// iterator->state == TaskTableItemState::MOVED)
//// iterator++;
//// table_.erase(table_.begin(), iterator);
//}
json
TaskTable::Dump() {
json ret;
for (auto& item : table_) {
ret.push_back(item->Dump());
}
TaskTable::Dump() const {
json ret{{"error.message", "not support yet."}};
return ret;
}

91
core/src/scheduler/TaskTable.h
View File

@ -25,6 +25,7 @@
#include <utility>
#include <vector>
#include "CircleQueue.h"
#include "event/Event.h"
#include "interface/interfaces.h"
#include "task/SearchTask.h"
@ -54,7 +55,7 @@ struct TaskTimestamp : public interface::dumpable {
uint64_t finish = 0;
json
Dump() override;
Dump() const override;
};
struct TaskTableItem : public interface::dumpable {
@ -92,18 +93,24 @@ struct TaskTableItem : public interface::dumpable {
Moved();
json
Dump() override;
Dump() const override;
};
using TaskTableItemPtr = std::shared_ptr<TaskTableItem>;
class TaskTable : public interface::dumpable {
public:
TaskTable() = default;
TaskTable() : table_(1ULL << 16ULL) {
}
TaskTable(const TaskTable&) = delete;
TaskTable(TaskTable&&) = delete;
public:
json
Dump() const override;
public:
inline void
RegisterSubscriber(std::function<void(void)> subscriber) {
subscriber_ = std::move(subscriber);
@ -122,59 +129,35 @@ class TaskTable : public interface::dumpable {
void
Put(std::vector<TaskPtr>& tasks);
/*
* Return task table item reference;
*/
TaskTableItemPtr
Get(uint64_t index);
size_t
TaskToExecute();
/*
* TODO(wxyu): BIG GC
* Remove sequence task which is DONE or MOVED from front;
* Called by ?
*/
// void
// Clear();
/*
* Return true if task table empty, otherwise false;
*/
inline bool
Empty() {
return table_.empty();
}
/*
* Return size of task table;
*/
inline size_t
Size() {
return table_.size();
}
public:
TaskTableItemPtr& operator[](uint64_t index) {
std::lock_guard<std::mutex> lock(mutex_);
return table_[index];
}
std::deque<TaskTableItemPtr>::iterator
begin() {
return table_.begin();
}
std::deque<TaskTableItemPtr>::iterator
end() {
return table_.end();
}
public:
std::vector<uint64_t>
PickToLoad(uint64_t limit);
std::vector<uint64_t>
PickToExecute(uint64_t limit);
public:
inline const TaskTableItemPtr& operator[](uint64_t index) {
return table_[index];
}
inline const TaskTableItemPtr&
at(uint64_t index) {
return table_[index];
}
inline size_t
capacity() {
return table_.capacity();
}
inline size_t
size() {
return table_.size();
}
public:
/******** Action ********/
@ -240,17 +223,9 @@ class TaskTable : public interface::dumpable {
return table_[index]->Moved();
}
public:
/*
* Dump;
*/
json
Dump() override;
private:
std::uint64_t id_ = 0;
mutable std::mutex mutex_;
std::deque<TaskTableItemPtr> table_;
CircleQueue<TaskTableItemPtr> table_;
std::function<void(void)> subscriber_ = nullptr;
// cache last finish avoid Pick task from begin always

5
core/src/scheduler/action/Action.h
View File

@ -37,10 +37,11 @@ class Action {
PushTaskToResource(const TaskPtr& task, const ResourcePtr& dest);
static void
DefaultLabelTaskScheduler(ResourceMgrWPtr res_mgr, ResourcePtr resource, std::shared_ptr<LoadCompletedEvent> event);
DefaultLabelTaskScheduler(const ResourceMgrPtr& res_mgr, ResourcePtr resource,
std::shared_ptr<LoadCompletedEvent> event);
static void
SpecifiedResourceLabelTaskScheduler(ResourceMgrWPtr res_mgr, ResourcePtr resource,
SpecifiedResourceLabelTaskScheduler(const ResourceMgrPtr& res_mgr, ResourcePtr resource,
std::shared_ptr<LoadCompletedEvent> event);
};

22
core/src/scheduler/action/PushTaskToNeighbour.cpp
View File

@ -30,7 +30,7 @@ std::vector<ResourcePtr>
get_neighbours(const ResourcePtr& self) {
std::vector<ResourcePtr> neighbours;
for (auto& neighbour_node : self->GetNeighbours()) {
auto node = neighbour_node.neighbour_node.lock();
auto node = neighbour_node.neighbour_node;
if (not node)
continue;
@ -46,7 +46,7 @@ std::vector<std::pair<ResourcePtr, Connection>>
get_neighbours_with_connetion(const ResourcePtr& self) {
std::vector<std::pair<ResourcePtr, Connection>> neighbours;
for (auto& neighbour_node : self->GetNeighbours()) {
auto node = neighbour_node.neighbour_node.lock();
auto node = neighbour_node.neighbour_node;
if (not node)
continue;
@ -102,7 +102,7 @@ Action::PushTaskToResource(const TaskPtr& task, const ResourcePtr& dest) {
}
void
Action::DefaultLabelTaskScheduler(ResourceMgrWPtr res_mgr, ResourcePtr resource,
Action::DefaultLabelTaskScheduler(const ResourceMgrPtr& res_mgr, ResourcePtr resource,
std::shared_ptr<LoadCompletedEvent> event) {
if (not resource->HasExecutor() && event->task_table_item_->Move()) {
auto task = event->task_table_item_->task;
@ -114,11 +114,11 @@ Action::DefaultLabelTaskScheduler(ResourceMgrWPtr res_mgr, ResourcePtr resource,
if (auto index_engine = search_task->index_engine_) {
auto location = index_engine->GetLocation();
for (auto i = 0; i < res_mgr.lock()->GetNumGpuResource(); ++i) {
for (auto i = 0; i < res_mgr->GetNumGpuResource(); ++i) {
auto index = milvus::cache::GpuCacheMgr::GetInstance(i)->GetIndex(location);
if (index != nullptr) {
moved = true;
auto dest_resource = res_mgr.lock()->GetResource(ResourceType::GPU, i);
auto dest_resource = res_mgr->GetResource(ResourceType::GPU, i);
PushTaskToResource(event->task_table_item_->task, dest_resource);
break;
}
@ -133,17 +133,17 @@ Action::DefaultLabelTaskScheduler(ResourceMgrWPtr res_mgr, ResourcePtr resource,
}
void
Action::SpecifiedResourceLabelTaskScheduler(ResourceMgrWPtr res_mgr, ResourcePtr resource,
Action::SpecifiedResourceLabelTaskScheduler(const ResourceMgrPtr& res_mgr, ResourcePtr resource,
std::shared_ptr<LoadCompletedEvent> event) {
auto task = event->task_table_item_->task;
if (resource->type() == ResourceType::DISK) {
// step 1: calculate shortest path per resource, from disk to compute resource
auto compute_resources = res_mgr.lock()->GetComputeResources();
auto compute_resources = res_mgr->GetComputeResources();
std::vector<std::vector<std::string>> paths;
std::vector<uint64_t> transport_costs;
for (auto& res : compute_resources) {
std::vector<std::string> path;
uint64_t transport_cost = ShortestPath(resource, res, res_mgr.lock(), path);
uint64_t transport_cost = ShortestPath(resource, res, res_mgr, path);
transport_costs.push_back(transport_cost);
paths.emplace_back(path);
}
@ -187,10 +187,10 @@ Action::SpecifiedResourceLabelTaskScheduler(ResourceMgrWPtr res_mgr, ResourcePtr
Status stat = config.GetResourceConfigIndexBuildDevice(build_index_gpu);
bool find_gpu_res = false;
if (res_mgr.lock()->GetResource(ResourceType::GPU, build_index_gpu) != nullptr) {
if (res_mgr->GetResource(ResourceType::GPU, build_index_gpu) != nullptr) {
for (uint64_t i = 0; i < compute_resources.size(); ++i) {
if (compute_resources[i]->name() ==
res_mgr.lock()->GetResource(ResourceType::GPU, build_index_gpu)->name()) {
res_mgr->GetResource(ResourceType::GPU, build_index_gpu)->name()) {
find_gpu_res = true;
Path task_path(paths[i], paths[i].size() - 1);
task->path() = task_path;
@ -208,7 +208,7 @@ Action::SpecifiedResourceLabelTaskScheduler(ResourceMgrWPtr res_mgr, ResourcePtr
resource->WakeupExecutor();
} else {
auto next_res_name = task->path().Next();
auto next_res = res_mgr.lock()->GetResource(next_res_name);
auto next_res = res_mgr->GetResource(next_res_name);
// if (event->task_table_item_->Move()) {
// next_res->task_table().Put(task);
// }

4
core/src/scheduler/event/Event.h
View File

@ -30,7 +30,7 @@ class Resource;
class Event {
public:
explicit Event(EventType type, std::weak_ptr<Resource> resource) : type_(type), resource_(std::move(resource)) {
explicit Event(EventType type, std::shared_ptr<Resource> resource) : type_(type), resource_(std::move(resource)) {
}
inline EventType
@ -46,7 +46,7 @@ class Event {
public:
EventType type_;
std::weak_ptr<Resource> resource_;
std::shared_ptr<Resource> resource_;
};
using EventPtr = std::shared_ptr<Event>;

2
core/src/scheduler/event/FinishTaskEvent.h
View File

@ -29,7 +29,7 @@ namespace scheduler {
class FinishTaskEvent : public Event {
public:
FinishTaskEvent(std::weak_ptr<Resource> resource, TaskTableItemPtr task_table_item)
FinishTaskEvent(std::shared_ptr<Resource> resource, TaskTableItemPtr task_table_item)
: Event(EventType::FINISH_TASK, std::move(resource)), task_table_item_(std::move(task_table_item)) {
}

2
core/src/scheduler/event/LoadCompletedEvent.h
View File

@ -29,7 +29,7 @@ namespace scheduler {
class LoadCompletedEvent : public Event {
public:
LoadCompletedEvent(std::weak_ptr<Resource> resource, TaskTableItemPtr task_table_item)
LoadCompletedEvent(std::shared_ptr<Resource> resource, TaskTableItemPtr task_table_item)
: Event(EventType::LOAD_COMPLETED, std::move(resource)), task_table_item_(std::move(task_table_item)) {
}

2
core/src/scheduler/event/StartUpEvent.h
View File

@ -28,7 +28,7 @@ namespace scheduler {
class StartUpEvent : public Event {
public:
explicit StartUpEvent(std::weak_ptr<Resource> resource) : Event(EventType::START_UP, std::move(resource)) {
explicit StartUpEvent(std::shared_ptr<Resource> resource) : Event(EventType::START_UP, std::move(resource)) {
}
inline std::string

2
core/src/scheduler/event/TaskTableUpdatedEvent.h
View File

@ -28,7 +28,7 @@ namespace scheduler {
class TaskTableUpdatedEvent : public Event {
public:
explicit TaskTableUpdatedEvent(std::weak_ptr<Resource> resource)
explicit TaskTableUpdatedEvent(std::shared_ptr<Resource> resource)
: Event(EventType::TASK_TABLE_UPDATED, std::move(resource)) {
}

2
core/src/scheduler/interface/interfaces.h
View File

@ -37,7 +37,7 @@ struct dumpable {
}
virtual json
Dump() = 0;
Dump() const = 0;
};
} // namespace interface

14
core/src/scheduler/job/BuildIndexJob.cpp
View File

@ -23,8 +23,8 @@
namespace milvus {
namespace scheduler {
BuildIndexJob::BuildIndexJob(JobId id, engine::meta::MetaPtr meta_ptr, engine::DBOptions options)
: Job(id, JobType::BUILD), meta_ptr_(std::move(meta_ptr)), options_(std::move(options)) {
BuildIndexJob::BuildIndexJob(engine::meta::MetaPtr meta_ptr, engine::DBOptions options)
: Job(JobType::BUILD), meta_ptr_(std::move(meta_ptr)), options_(std::move(options)) {
}
bool
@ -54,5 +54,15 @@ BuildIndexJob::BuildIndexDone(size_t to_index_id) {
SERVER_LOG_DEBUG << "BuildIndexJob " << id() << " finish index file: " << to_index_id;
}
json
BuildIndexJob::Dump() const {
json ret{
{"number_of_to_index_file", to_index_files_.size()},
};
auto base = Job::Dump();
ret.insert(base.begin(), base.end());
return ret;
}
} // namespace scheduler
} // namespace milvus

5
core/src/scheduler/job/BuildIndexJob.h
View File

@ -41,7 +41,7 @@ using Id2ToTableFileMap = std::unordered_map<size_t, TableFileSchema>;
class BuildIndexJob : public Job {
public:
explicit BuildIndexJob(JobId id, engine::meta::MetaPtr meta_ptr, engine::DBOptions options);
explicit BuildIndexJob(engine::meta::MetaPtr meta_ptr, engine::DBOptions options);
public:
bool
@ -53,6 +53,9 @@ class BuildIndexJob : public Job {
void
BuildIndexDone(size_t to_index_id);
json
Dump() const override;
public:
Status&
GetStatus() {

16
core/src/scheduler/job/DeleteJob.cpp
View File

@ -22,8 +22,8 @@
namespace milvus {
namespace scheduler {
DeleteJob::DeleteJob(JobId id, std::string table_id, engine::meta::MetaPtr meta_ptr, uint64_t num_resource)
: Job(id, JobType::DELETE),
DeleteJob::DeleteJob(std::string table_id, engine::meta::MetaPtr meta_ptr, uint64_t num_resource)
: Job(JobType::DELETE),
table_id_(std::move(table_id)),
meta_ptr_(std::move(meta_ptr)),
num_resource_(num_resource) {
@ -45,5 +45,17 @@ DeleteJob::ResourceDone() {
cv_.notify_one();
}
json
DeleteJob::Dump() const {
json ret{
{"table_id", table_id_},
{"number_of_resource", num_resource_},
{"number_of_done", done_resource},
};
auto base = Job::Dump();
ret.insert(base.begin(), base.end());
return ret;
}
} // namespace scheduler
} // namespace milvus

5
core/src/scheduler/job/DeleteJob.h
View File

@ -35,7 +35,7 @@ namespace scheduler {
class DeleteJob : public Job {
public:
DeleteJob(JobId id, std::string table_id, engine::meta::MetaPtr meta_ptr, uint64_t num_resource);
DeleteJob(std::string table_id, engine::meta::MetaPtr meta_ptr, uint64_t num_resource);
public:
void
@ -44,6 +44,9 @@ class DeleteJob : public Job {
void
ResourceDone();
json
Dump() const override;
public:
std::string
table_id() const {

43
core/src/scheduler/job/Job.cpp Normal file
View File

@ -0,0 +1,43 @@
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.
#include "scheduler/job/Job.h"
namespace milvus {
namespace scheduler {
namespace {
std::mutex unique_job_mutex;
uint64_t unique_job_id = 0;
} // namespace
Job::Job(JobType type) : type_(type) {
std::lock_guard<std::mutex> lock(unique_job_mutex);
id_ = unique_job_id++;
}
json
Job::Dump() const {
json ret{
{"id", id_},
{"type", type_},
};
return ret;
}
} // namespace scheduler
} // namespace milvus

12
core/src/scheduler/job/Job.h
View File

@ -27,6 +27,8 @@
#include <unordered_map>
#include <vector>
#include "scheduler/interface/interfaces.h"
namespace milvus {
namespace scheduler {
@ -39,7 +41,7 @@ enum class JobType {
using JobId = std::uint64_t;
class Job {
class Job : public interface::dumpable {
public:
inline JobId
id() const {
@ -51,12 +53,14 @@ class Job {
return type_;
}
json
Dump() const override;
protected:
Job(JobId id, JobType type) : id_(id), type_(type) {
}
explicit Job(JobType type);
private:
JobId id_;
JobId id_ = 0;
JobType type_;
};

16
core/src/scheduler/job/SearchJob.cpp
View File

@ -21,8 +21,8 @@
namespace milvus {
namespace scheduler {
SearchJob::SearchJob(milvus::scheduler::JobId id, uint64_t topk, uint64_t nq, uint64_t nprobe, const float* vectors)
: Job(id, JobType::SEARCH), topk_(topk), nq_(nq), nprobe_(nprobe), vectors_(vectors) {
SearchJob::SearchJob(uint64_t topk, uint64_t nq, uint64_t nprobe, const float* vectors)
: Job(JobType::SEARCH), topk_(topk), nq_(nq), nprobe_(nprobe), vectors_(vectors) {
}
bool
@ -63,5 +63,17 @@ SearchJob::GetStatus() {
return status_;
}
json
SearchJob::Dump() const {
json ret{
{"topk", topk_},
{"nq", nq_},
{"nprobe", nprobe_},
};
auto base = Job::Dump();
ret.insert(base.begin(), base.end());
return ret;
}
} // namespace scheduler
} // namespace milvus

5
core/src/scheduler/job/SearchJob.h
View File

@ -43,7 +43,7 @@ using ResultSet = std::vector<Id2DistVec>;
class SearchJob : public Job {
public:
SearchJob(JobId id, uint64_t topk, uint64_t nq, uint64_t nprobe, const float* vectors);
SearchJob(uint64_t topk, uint64_t nq, uint64_t nprobe, const float* vectors);
public:
bool
@ -61,6 +61,9 @@ class SearchJob : public Job {
Status&
GetStatus();
json
Dump() const override;
public:
uint64_t
topk() const {

84
core/src/scheduler/optimizer/LargeSQ8HPass.cpp
View File

@ -26,48 +26,48 @@
namespace milvus {
namespace scheduler {
// bool
// LargeSQ8HPass::Run(const TaskPtr& task) {
// if (task->Type() != TaskType::SearchTask) {
// return false;
// }
//
// auto search_task = std::static_pointer_cast<XSearchTask>(task);
// if (search_task->file_->engine_type_ != (int)engine::EngineType::FAISS_IVFSQ8H) {
// return false;
// }
//
// auto search_job = std::static_pointer_cast<SearchJob>(search_task->job_.lock());
//
// // TODO: future, Index::IVFSQ8H, if nq < threshold set cpu, else set gpu
// if (search_job->nq() < 100) {
// return false;
// }
//
// std::vector<uint64_t> gpus = scheduler::get_gpu_pool();
// std::vector<int64_t> all_free_mem;
// for (auto& gpu : gpus) {
// auto cache = cache::GpuCacheMgr::GetInstance(gpu);
// auto free_mem = cache->CacheCapacity() - cache->CacheUsage();
// all_free_mem.push_back(free_mem);
// }
//
// auto max_e = std::max_element(all_free_mem.begin(), all_free_mem.end());
// auto best_index = std::distance(all_free_mem.begin(), max_e);
// auto best_device_id = gpus[best_index];
//
// ResourcePtr res_ptr = ResMgrInst::GetInstance()->GetResource(ResourceType::GPU, best_device_id);
// if (not res_ptr) {
// SERVER_LOG_ERROR << "GpuResource " << best_device_id << " invalid.";
// // TODO: throw critical error and exit
// return false;
// }
//
// auto label = std::make_shared<SpecResLabel>(std::weak_ptr<Resource>(res_ptr));
// task->label() = label;
//
// return true;
// }
bool
LargeSQ8HPass::Run(const TaskPtr& task) {
if (task->Type() != TaskType::SearchTask) {
return false;
}
auto search_task = std::static_pointer_cast<XSearchTask>(task);
if (search_task->file_->engine_type_ != (int)engine::EngineType::FAISS_IVFSQ8H) {
return false;
}
auto search_job = std::static_pointer_cast<SearchJob>(search_task->job_.lock());
// TODO: future, Index::IVFSQ8H, if nq < threshold set cpu, else set gpu
if (search_job->nq() < 100) {
return false;
}
std::vector<uint64_t> gpus = scheduler::get_gpu_pool();
std::vector<int64_t> all_free_mem;
for (auto& gpu : gpus) {
auto cache = cache::GpuCacheMgr::GetInstance(gpu);
auto free_mem = cache->CacheCapacity() - cache->CacheUsage();
all_free_mem.push_back(free_mem);
}
auto max_e = std::max_element(all_free_mem.begin(), all_free_mem.end());
auto best_index = std::distance(all_free_mem.begin(), max_e);
auto best_device_id = gpus[best_index];
ResourcePtr res_ptr = ResMgrInst::GetInstance()->GetResource(ResourceType::GPU, best_device_id);
if (not res_ptr) {
SERVER_LOG_ERROR << "GpuResource " << best_device_id << " invalid.";
// TODO: throw critical error and exit
return false;
}
auto label = std::make_shared<SpecResLabel>(std::weak_ptr<Resource>(res_ptr));
task->label() = label;
return true;
}
} // namespace scheduler
} // namespace milvus

4
core/src/scheduler/optimizer/LargeSQ8HPass.h
View File

@ -37,8 +37,8 @@ class LargeSQ8HPass : public Pass {
LargeSQ8HPass() = default;
public:
// bool
// Run(const TaskPtr& task) override;
bool
Run(const TaskPtr& task) override;
};
using LargeSQ8HPassPtr = std::shared_ptr<LargeSQ8HPass>;

2
core/src/scheduler/resource/CpuResource.cpp
View File

@ -24,7 +24,7 @@ namespace scheduler {
std::ostream&
operator<<(std::ostream& out, const CpuResource& resource) {
out << resource.Dump();
out << resource.Dump().dump();
return out;
}

5
core/src/scheduler/resource/CpuResource.h
View File

@ -28,11 +28,6 @@ class CpuResource : public Resource {
public:
explicit CpuResource(std::string name, uint64_t device_id, bool enable_loader, bool enable_executor);
inline std::string
Dump() const override {
return "<CpuResource, name=" + name_ + ">";
}
friend std::ostream&
operator<<(std::ostream& out, const CpuResource& resource);

5
core/src/scheduler/resource/DiskResource.h
View File

@ -28,11 +28,6 @@ class DiskResource : public Resource {
public:
explicit DiskResource(std::string name, uint64_t device_id, bool enable_loader, bool enable_executor);
inline std::string
Dump() const override {
return "<DiskResource, name=" + name_ + ">";
}
friend std::ostream&
operator<<(std::ostream& out, const DiskResource& resource);

2
core/src/scheduler/resource/GpuResource.cpp
View File

@ -22,7 +22,7 @@ namespace scheduler {
std::ostream&
operator<<(std::ostream& out, const GpuResource& resource) {
out << resource.Dump();
out << resource.Dump().dump();
return out;
}

5
core/src/scheduler/resource/GpuResource.h
View File

@ -29,11 +29,6 @@ class GpuResource : public Resource {
public:
explicit GpuResource(std::string name, uint64_t device_id, bool enable_loader, bool enable_executor);
inline std::string
Dump() const override {
return "<GpuResource, name=" + name_ + ">";
}
friend std::ostream&
operator<<(std::ostream& out, const GpuResource& resource);

24
core/src/scheduler/resource/Node.cpp
View File

@ -38,23 +38,27 @@ Node::GetNeighbours() {
return ret;
}
std::string
Node::Dump() {
std::stringstream ss;
ss << "<Node, id=" << std::to_string(id_) << ">::neighbours:" << std::endl;
json
Node::Dump() const {
json neighbours;
for (auto& neighbour : neighbours_) {
ss << "\t<Neighbour, id=" << std::to_string(neighbour.first);
ss << ", connection: " << neighbour.second.connection.Dump() << ">" << std::endl;
json n;
n["id"] = neighbour.first;
n["connection"] = neighbour.second.connection.Dump();
neighbours.push_back(n);
}
return ss.str();
json ret{
{"id", id_},
{"neighbours", neighbours},
};
return ret;
}
void
Node::AddNeighbour(const NeighbourNodePtr& neighbour_node, Connection& connection) {
std::lock_guard<std::mutex> lk(mutex_);
if (auto s = neighbour_node.lock()) {
neighbours_.emplace(std::make_pair(s->id_, Neighbour(neighbour_node, connection)));
}
neighbours_.emplace(std::make_pair(neighbour_node->id_, Neighbour(neighbour_node, connection)));
// else do nothing, consider it..
}

16
core/src/scheduler/resource/Node.h
View File

@ -20,20 +20,26 @@
#include <map>
#include <memory>
#include <string>
#include <utility>
#include <vector>
#include "Connection.h"
#include "scheduler/TaskTable.h"
#include "scheduler/interface/interfaces.h"
namespace milvus {
namespace scheduler {
class Node;
using NeighbourNodePtr = std::weak_ptr<Node>;
using NeighbourNodePtr = std::shared_ptr<Node>;
struct Neighbour {
Neighbour(NeighbourNodePtr nei, Connection conn) : neighbour_node(nei), connection(conn) {
Neighbour(NeighbourNodePtr nei, Connection conn) : neighbour_node(std::move(nei)), connection(std::move(conn)) {
}
~Neighbour() {
neighbour_node = nullptr;
}
NeighbourNodePtr neighbour_node;
@ -41,7 +47,7 @@ struct Neighbour {
};
// TODO(lxj): return type void -> Status
class Node {
class Node : public interface::dumpable {
public:
Node();
@ -52,8 +58,8 @@ class Node {
GetNeighbours();
public:
std::string
Dump();
json
Dump() const override;
private:
std::mutex mutex_;

43
core/src/scheduler/resource/Resource.cpp
View File

@ -32,6 +32,22 @@ operator<<(std::ostream& out, const Resource& resource) {
return out;
}
std::string
ToString(ResourceType type) {
switch (type) {
case ResourceType::DISK: {
return "DISK";
}
case ResourceType::CPU: {
return "CPU";
}
case ResourceType::GPU: {
return "GPU";
}
default: { return "UNKNOWN"; }
}
}
Resource::Resource(std::string name, ResourceType type, uint64_t device_id, bool enable_loader, bool enable_executor)
: name_(std::move(name)),
type_(type),
@ -89,14 +105,25 @@ Resource::WakeupExecutor() {
exec_cv_.notify_one();
}
json
Resource::Dump() const {
json ret{
{"device_id", device_id_},
{"name", name_},
{"type", ToString(type_)},
{"task_average_cost", TaskAvgCost()},
{"task_total_cost", total_cost_},
{"total_tasks", total_task_},
{"running", running_},
{"enable_loader", enable_loader_},
{"enable_executor", enable_executor_},
};
return ret;
}
uint64_t
Resource::NumOfTaskToExec() {
uint64_t count = 0;
for (auto& task : task_table_) {
if (task->state == TaskTableItemState::LOADED)
++count;
}
return count;
return task_table_.TaskToExecute();
}
TaskTableItemPtr
@ -105,7 +132,7 @@ Resource::pick_task_load() {
for (auto index : indexes) {
// try to set one task loading, then return
if (task_table_.Load(index))
return task_table_.Get(index);
return task_table_.at(index);
// else try next
}
return nullptr;
@ -123,7 +150,7 @@ Resource::pick_task_execute() {
}
if (task_table_.Execute(index)) {
return task_table_.Get(index);
return task_table_.at(index);
}
// if (task_table_[index]->task->label()->Type() == TaskLabelType::SPECIFIED_RESOURCE) {
// if (task_table_.Get(index)->task->path().Current() == task_table_.Get(index)->task->path().Last()

9
core/src/scheduler/resource/Resource.h
View File

@ -77,10 +77,8 @@ class Resource : public Node, public std::enable_shared_from_this<Resource> {
subscriber_ = std::move(subscriber);
}
inline virtual std::string
Dump() const {
return "<Resource>";
}
json
Dump() const override;
public:
inline std::string
@ -121,6 +119,9 @@ class Resource : public Node, public std::enable_shared_from_this<Resource> {
// TODO(wxyu): need double ?
inline uint64_t
TaskAvgCost() const {
if (total_task_ == 0) {
return 0;
}
return total_cost_ / total_task_;
}

5
core/src/scheduler/resource/TestResource.h
View File

@ -29,11 +29,6 @@ class TestResource : public Resource {
public:
explicit TestResource(std::string name, uint64_t device_id, bool enable_loader, bool enable_executor);
inline std::string
Dump() const override {
return "<TestResource, name=" + name_ + ">";
}
friend std::ostream&
operator<<(std::ostream& out, const TestResource& resource);

3
core/src/sdk/CMakeLists.txt
View File

@ -30,9 +30,6 @@ add_library(milvus_sdk STATIC
target_link_libraries(milvus_sdk
${client_grpc_lib}
bzip2
lz4
snappy
zlib
)

2
core/src/sdk/grpc/ClientProxy.cpp
View File

@ -16,8 +16,8 @@
// under the License.
#include "sdk/grpc/ClientProxy.h"
#include "../../../version.h"
#include "grpc/gen-milvus/milvus.grpc.pb.h"
#include "src/version.h"
#include <memory>
#include <string>

157
core/src/server/Config.cpp
View File

@ -363,7 +363,9 @@ Config::PrintAll() {
Status
Config::CheckServerConfigAddress(const std::string& value) {
if (!ValidationUtil::ValidateIpAddress(value).ok()) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid server config address: " + value);
std::string msg =
"Invalid server IP address: " + value + ". Possible reason: server_config.address is invalid.";
return Status(SERVER_INVALID_ARGUMENT, msg);
}
return Status::OK();
}
@ -371,11 +373,14 @@ Config::CheckServerConfigAddress(const std::string& value) {
Status
Config::CheckServerConfigPort(const std::string& value) {
if (!ValidationUtil::ValidateStringIsNumber(value).ok()) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid server config port: " + value);
std::string msg = "Invalid server port: " + value + ". Possible reason: server_config.port is not a number.";
return Status(SERVER_INVALID_ARGUMENT, msg);
} else {
int32_t port = std::stoi(value);
if (!(port > 1024 && port < 65535)) {
return Status(SERVER_INVALID_ARGUMENT, "Server config port out of range (1024, 65535): " + value);
std::string msg = "Invalid server port: " + value +
". Possible reason: server_config.port is not in range [1025, 65534].";
return Status(SERVER_INVALID_ARGUMENT, msg);
}
}
return Status::OK();
@ -385,7 +390,8 @@ Status
Config::CheckServerConfigDeployMode(const std::string& value) {
if (value != "single" && value != "cluster_readonly" && value != "cluster_writable") {
return Status(SERVER_INVALID_ARGUMENT,
"Invalid server config mode [single, cluster_readonly, cluster_writable]: " + value);
"server_config.deploy_mode is not one of "
"single, cluster_readonly, and cluster_writable.");
}
return Status::OK();
}
@ -393,15 +399,15 @@ Config::CheckServerConfigDeployMode(const std::string& value) {
Status
Config::CheckServerConfigTimeZone(const std::string& value) {
if (value.length() <= 3) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid server config time_zone: " + value);
return Status(SERVER_INVALID_ARGUMENT, "Invalid server_config.time_zone: " + value);
} else {
if (value.substr(0, 3) != "UTC") {
return Status(SERVER_INVALID_ARGUMENT, "Invalid server config time_zone: " + value);
return Status(SERVER_INVALID_ARGUMENT, "Invalid server_config.time_zone: " + value);
} else {
try {
stoi(value.substr(3));
} catch (...) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid server config time_zone: " + value);
return Status(SERVER_INVALID_ARGUMENT, "Invalid server_config.time_zone: " + value);
}
}
}
@ -411,7 +417,7 @@ Config::CheckServerConfigTimeZone(const std::string& value) {
Status
Config::CheckDBConfigPrimaryPath(const std::string& value) {
if (value.empty()) {
return Status(SERVER_INVALID_ARGUMENT, "DB config primary_path empty");
return Status(SERVER_INVALID_ARGUMENT, "db_config.db_path is empty.");
}
return Status::OK();
}
@ -424,7 +430,10 @@ Config::CheckDBConfigSecondaryPath(const std::string& value) {
Status
Config::CheckDBConfigBackendUrl(const std::string& value) {
if (!ValidationUtil::ValidateDbURI(value).ok()) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid DB config backend_url: " + value);
std::string msg =
"Invalid backend url: " + value + ". Possible reason: db_config.db_backend_url is invalid. " +
"The correct format should be like sqlite://:@:/ or mysql://root:123456@127.0.0.1:3306/milvus.";
return Status(SERVER_INVALID_ARGUMENT, "invalid db_backend_url: " + value);
}
return Status::OK();
}
@ -432,7 +441,9 @@ Config::CheckDBConfigBackendUrl(const std::string& value) {
Status
Config::CheckDBConfigArchiveDiskThreshold(const std::string& value) {
if (!ValidationUtil::ValidateStringIsNumber(value).ok()) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid DB config archive_disk_threshold: " + value);
std::string msg = "Invalid archive disk threshold: " + value +
". Possible reason: db_config.archive_disk_threshold is invalid.";
return Status(SERVER_INVALID_ARGUMENT, msg);
}
return Status::OK();
}
@ -440,7 +451,9 @@ Config::CheckDBConfigArchiveDiskThreshold(const std::string& value) {
Status
Config::CheckDBConfigArchiveDaysThreshold(const std::string& value) {
if (!ValidationUtil::ValidateStringIsNumber(value).ok()) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid DB config archive_days_threshold: " + value);
std::string msg = "Invalid archive days threshold: " + value +
". Possible reason: db_config.archive_disk_threshold is invalid.";
return Status(SERVER_INVALID_ARGUMENT, msg);
}
return Status::OK();
}
@ -448,13 +461,23 @@ Config::CheckDBConfigArchiveDaysThreshold(const std::string& value) {
Status
Config::CheckDBConfigInsertBufferSize(const std::string& value) {
if (!ValidationUtil::ValidateStringIsNumber(value).ok()) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid DB config insert_buffer_size: " + value);
std::string msg = "Invalid insert buffer size: " + value +
". Possible reason: db_config.insert_buffer_size is not a positive integer.";
return Status(SERVER_INVALID_ARGUMENT, msg);
} else {
int64_t buffer_size = std::stoi(value) * GB;
if (buffer_size <= 0) {
std::string msg = "Invalid insert buffer size: " + value +
". Possible reason: db_config.insert_buffer_size is not a positive integer.";
return Status(SERVER_INVALID_ARGUMENT, msg);
}
uint64_t total_mem = 0, free_mem = 0;
CommonUtil::GetSystemMemInfo(total_mem, free_mem);
if (buffer_size >= total_mem) {
return Status(SERVER_INVALID_ARGUMENT, "DB config insert_buffer_size exceed system memory: " + value);
std::string msg = "Invalid insert buffer size: " + value +
". Possible reason: db_config.insert_buffer_size exceeds system memory.";
return Status(SERVER_INVALID_ARGUMENT, msg);
}
}
return Status::OK();
@ -463,7 +486,9 @@ Config::CheckDBConfigInsertBufferSize(const std::string& value) {
Status
Config::CheckMetricConfigEnableMonitor(const std::string& value) {
if (!ValidationUtil::ValidateStringIsBool(value).ok()) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid metric config auto_bootup: " + value);
std::string msg =
"Invalid metric config: " + value + ". Possible reason: metric_config.enable_monitor is not a boolean.";
return Status(SERVER_INVALID_ARGUMENT, msg);
}
return Status::OK();
}
@ -471,7 +496,9 @@ Config::CheckMetricConfigEnableMonitor(const std::string& value) {
Status
Config::CheckMetricConfigCollector(const std::string& value) {
if (value != "prometheus") {
return Status(SERVER_INVALID_ARGUMENT, "Invalid metric config collector: " + value);
std::string msg =
"Invalid metric collector: " + value + ". Possible reason: metric_config.collector is invalid.";
return Status(SERVER_INVALID_ARGUMENT, msg);
}
return Status::OK();
}
@ -479,6 +506,8 @@ Config::CheckMetricConfigCollector(const std::string& value) {
Status
Config::CheckMetricConfigPrometheusPort(const std::string& value) {
if (!ValidationUtil::ValidateStringIsNumber(value).ok()) {
std::string msg = "Invalid metric port: " + value +
". Possible reason: metric_config.prometheus_config.port is not in range [1025, 65534].";
return Status(SERVER_INVALID_ARGUMENT, "Invalid metric config prometheus_port: " + value);
}
return Status::OK();
@ -487,15 +516,25 @@ Config::CheckMetricConfigPrometheusPort(const std::string& value) {
Status
Config::CheckCacheConfigCpuCacheCapacity(const std::string& value) {
if (!ValidationUtil::ValidateStringIsNumber(value).ok()) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid cache config cpu_cache_capacity: " + value);
std::string msg = "Invalid cpu cache capacity: " + value +
". Possible reason: cache_config.cpu_cache_capacity is not a positive integer.";
return Status(SERVER_INVALID_ARGUMENT, msg);
} else {
uint64_t cpu_cache_capacity = std::stoi(value) * GB;
int64_t cpu_cache_capacity = std::stoi(value) * GB;
if (cpu_cache_capacity <= 0) {
std::string msg = "Invalid cpu cache capacity: " + value +
". Possible reason: cache_config.cpu_cache_capacity is not a positive integer.";
return Status(SERVER_INVALID_ARGUMENT, msg);
}
uint64_t total_mem = 0, free_mem = 0;
CommonUtil::GetSystemMemInfo(total_mem, free_mem);
if (cpu_cache_capacity >= total_mem) {
return Status(SERVER_INVALID_ARGUMENT, "Cache config cpu_cache_capacity exceed system memory: " + value);
} else if (cpu_cache_capacity > static_cast<double>(total_mem * 0.9)) {
std::cerr << "Warning: cpu_cache_capacity value is too big" << std::endl;
if (static_cast<uint64_t>(cpu_cache_capacity) >= total_mem) {
std::string msg = "Invalid cpu cache capacity: " + value +
". Possible reason: cache_config.cpu_cache_capacity exceeds system memory.";
return Status(SERVER_INVALID_ARGUMENT, msg);
} else if (static_cast<double>(cpu_cache_capacity) > static_cast<double>(total_mem * 0.9)) {
std::cerr << "WARNING: cpu cache capacity value is too big" << std::endl;
}
int32_t buffer_value;
@ -506,7 +545,10 @@ Config::CheckCacheConfigCpuCacheCapacity(const std::string& value) {
int64_t insert_buffer_size = buffer_value * GB;
if (insert_buffer_size + cpu_cache_capacity >= total_mem) {
return Status(SERVER_INVALID_ARGUMENT, "Sum of cpu_cache_capacity and buffer_size exceed system memory");
std::string msg = "Invalid cpu cache capacity: " + value +
". Possible reason: sum of cache_config.cpu_cache_capacity and "
"db_config.insert_buffer_size exceeds system memory.";
return Status(SERVER_INVALID_ARGUMENT, msg);
}
}
return Status::OK();
@ -515,11 +557,15 @@ Config::CheckCacheConfigCpuCacheCapacity(const std::string& value) {
Status
Config::CheckCacheConfigCpuCacheThreshold(const std::string& value) {
if (!ValidationUtil::ValidateStringIsFloat(value).ok()) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid cache config cpu_cache_threshold: " + value);
std::string msg = "Invalid cpu cache threshold: " + value +
". Possible reason: cache_config.cpu_cache_threshold is not in range (0.0, 1.0].";
return Status(SERVER_INVALID_ARGUMENT, msg);
} else {
float cpu_cache_threshold = std::stof(value);
if (cpu_cache_threshold <= 0.0 || cpu_cache_threshold >= 1.0) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid cache config cpu_cache_threshold: " + value);
std::string msg = "Invalid cpu cache threshold: " + value +
". Possible reason: cache_config.cpu_cache_threshold is not in range (0.0, 1.0].";
return Status(SERVER_INVALID_ARGUMENT, msg);
}
}
return Status::OK();
@ -528,7 +574,9 @@ Config::CheckCacheConfigCpuCacheThreshold(const std::string& value) {
Status
Config::CheckCacheConfigGpuCacheCapacity(const std::string& value) {
if (!ValidationUtil::ValidateStringIsNumber(value).ok()) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid cache config gpu_cache_capacity: " + value);
std::string msg = "Invalid gpu cache capacity: " + value +
". Possible reason: cache_config.gpu_cache_capacity is not a positive integer.";
return Status(SERVER_INVALID_ARGUMENT, msg);
} else {
uint64_t gpu_cache_capacity = std::stoi(value) * GB;
int gpu_index;
@ -539,13 +587,14 @@ Config::CheckCacheConfigGpuCacheCapacity(const std::string& value) {
size_t gpu_memory;
if (!ValidationUtil::GetGpuMemory(gpu_index, gpu_memory).ok()) {
return Status(SERVER_UNEXPECTED_ERROR,
"Fail to get GPU memory for GPU device: " + std::to_string(gpu_index));
std::string msg = "Fail to get GPU memory for GPU device: " + std::to_string(gpu_index);
return Status(SERVER_UNEXPECTED_ERROR, msg);
} else if (gpu_cache_capacity >= gpu_memory) {
return Status(SERVER_INVALID_ARGUMENT,
"Cache config gpu_cache_capacity exceed GPU memory: " + std::to_string(gpu_memory));
std::string msg = "Invalid gpu cache capacity: " + value +
". Possible reason: cache_config.gpu_cache_capacity exceeds GPU memory.";
return Status(SERVER_INVALID_ARGUMENT, msg);
} else if (gpu_cache_capacity > (double)gpu_memory * 0.9) {
std::cerr << "Warning: gpu_cache_capacity value is too big" << std::endl;
std::cerr << "Warning: gpu cache capacity value is too big" << std::endl;
}
}
return Status::OK();
@ -554,11 +603,15 @@ Config::CheckCacheConfigGpuCacheCapacity(const std::string& value) {
Status
Config::CheckCacheConfigGpuCacheThreshold(const std::string& value) {
if (!ValidationUtil::ValidateStringIsFloat(value).ok()) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid cache config gpu_cache_threshold: " + value);
std::string msg = "Invalid gpu cache threshold: " + value +
". Possible reason: cache_config.gpu_cache_threshold is not in range (0.0, 1.0].";
return Status(SERVER_INVALID_ARGUMENT, msg);
} else {
float gpu_cache_threshold = std::stof(value);
if (gpu_cache_threshold <= 0.0 || gpu_cache_threshold >= 1.0) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid cache config gpu_cache_threshold: " + value);
std::string msg = "Invalid gpu cache threshold: " + value +
". Possible reason: cache_config.gpu_cache_threshold is not in range (0.0, 1.0].";
return Status(SERVER_INVALID_ARGUMENT, msg);
}
}
return Status::OK();
@ -567,7 +620,9 @@ Config::CheckCacheConfigGpuCacheThreshold(const std::string& value) {
Status
Config::CheckCacheConfigCacheInsertData(const std::string& value) {
if (!ValidationUtil::ValidateStringIsBool(value).ok()) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid cache config cache_insert_data: " + value);
std::string msg = "Invalid cache insert option: " + value +
". Possible reason: cache_config.cache_insert_data is not a boolean.";
return Status(SERVER_INVALID_ARGUMENT, msg);
}
return Status::OK();
}
@ -575,7 +630,9 @@ Config::CheckCacheConfigCacheInsertData(const std::string& value) {
Status
Config::CheckEngineConfigUseBlasThreshold(const std::string& value) {
if (!ValidationUtil::ValidateStringIsNumber(value).ok()) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid engine config use_blas_threshold: " + value);
std::string msg = "Invalid blas threshold: " + value +
". Possible reason: engine_config.use_blas_threshold is not a positive integer.";
return Status(SERVER_INVALID_ARGUMENT, msg);
}
return Status::OK();
}
@ -583,14 +640,18 @@ Config::CheckEngineConfigUseBlasThreshold(const std::string& value) {
Status
Config::CheckEngineConfigOmpThreadNum(const std::string& value) {
if (!ValidationUtil::ValidateStringIsNumber(value).ok()) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid engine config omp_thread_num: " + value);
std::string msg = "Invalid omp thread number: " + value +
". Possible reason: engine_config.omp_thread_num is not a positive integer.";
return Status(SERVER_INVALID_ARGUMENT, msg);
}
int32_t omp_thread = std::stoi(value);
uint32_t sys_thread_cnt = 8;
CommonUtil::GetSystemAvailableThreads(sys_thread_cnt);
if (omp_thread > static_cast<int32_t>(sys_thread_cnt)) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid engine config omp_thread_num: " + value);
std::string msg = "Invalid omp thread number: " + value +
". Possible reason: engine_config.omp_thread_num exceeds system cpu cores.";
return Status(SERVER_INVALID_ARGUMENT, msg);
}
return Status::OK();
}
@ -598,7 +659,8 @@ Config::CheckEngineConfigOmpThreadNum(const std::string& value) {
Status
Config::CheckResourceConfigMode(const std::string& value) {
if (value != "simple") {
return Status(SERVER_INVALID_ARGUMENT, "Invalid resource config mode: " + value);
std::string msg = "Invalid resource mode: " + value + ". Possible reason: resource_config.mode is invalid.";
return Status(SERVER_INVALID_ARGUMENT, msg);
}
return Status::OK();
}
@ -608,12 +670,16 @@ CheckGpuDevice(const std::string& value) {
const std::regex pat("gpu(\\d+)");
std::cmatch m;
if (!std::regex_match(value.c_str(), m, pat)) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid gpu device: " + value);
std::string msg = "Invalid gpu device: " + value +
". Possible reason: resource_config.search_resources does not match your hardware.";
return Status(SERVER_INVALID_ARGUMENT, msg);
}
int32_t gpu_index = std::stoi(value.substr(3));
if (!ValidationUtil::ValidateGpuIndex(gpu_index).ok()) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid gpu device: " + value);
std::string msg = "Invalid gpu device: " + value +
". Possible reason: resource_config.search_resources does not match your hardware.";
return Status(SERVER_INVALID_ARGUMENT, msg);
}
return Status::OK();
}
@ -621,12 +687,17 @@ CheckGpuDevice(const std::string& value) {
Status
Config::CheckResourceConfigSearchResources(const std::vector<std::string>& value) {
if (value.empty()) {
return Status(SERVER_INVALID_ARGUMENT, "Empty resource config search_resources");
std::string msg =
"Invalid search resource. "
"Possible reason: resource_config.search_resources is empty.";
return Status(SERVER_INVALID_ARGUMENT, msg);
}
for (auto& gpu_device : value) {
if (!CheckGpuDevice(gpu_device).ok()) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid resource config search_resources: " + gpu_device);
std::string msg = "Invalid search resource: " + gpu_device +
". Possible reason: resource_config.search_resources does not match your hardware.";
return Status(SERVER_INVALID_ARGUMENT, msg);
}
}
return Status::OK();
@ -635,7 +706,9 @@ Config::CheckResourceConfigSearchResources(const std::vector<std::string>& value
Status
Config::CheckResourceConfigIndexBuildDevice(const std::string& value) {
if (!CheckGpuDevice(value).ok()) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid resource config index_build_device: " + value);
std::string msg = "Invalid index build device: " + value +
". Possible reason: resource_config.index_build_device does not match your hardware.";
return Status(SERVER_INVALID_ARGUMENT, msg);
}
return Status::OK();
}

35
core/src/server/DBWrapper.cpp
View File

@ -15,7 +15,7 @@
// specific language governing permissions and limitations
// under the License.
#include <faiss/utils.h>
#include <faiss/utils/distances.h>
#include <omp.h>
#include <cmath>
#include <string>
@ -40,12 +40,14 @@ DBWrapper::StartService() {
engine::DBOptions opt;
s = config.GetDBConfigBackendUrl(opt.meta_.backend_uri_);
if (!s.ok()) {
std::cerr << s.ToString() << std::endl;
return s;
}
std::string path;
s = config.GetDBConfigPrimaryPath(path);
if (!s.ok()) {
std::cerr << s.ToString() << std::endl;
return s;
}
@ -54,6 +56,7 @@ DBWrapper::StartService() {
std::string db_slave_path;
s = config.GetDBConfigSecondaryPath(db_slave_path);
if (!s.ok()) {
std::cerr << s.ToString() << std::endl;
return s;
}
@ -62,12 +65,14 @@ DBWrapper::StartService() {
// cache config
s = config.GetCacheConfigCacheInsertData(opt.insert_cache_immediately_);
if (!s.ok()) {
std::cerr << s.ToString() << std::endl;
return s;
}
std::string mode;
s = config.GetServerConfigDeployMode(mode);
if (!s.ok()) {
std::cerr << s.ToString() << std::endl;
return s;
}
@ -78,8 +83,8 @@ DBWrapper::StartService() {
} else if (mode == "cluster_writable") {
opt.mode_ = engine::DBOptions::MODE::CLUSTER_WRITABLE;
} else {
std::cerr << "ERROR: mode specified in server_config must be ['single', 'cluster_readonly', 'cluster_writable']"
<< std::endl;
std::cerr << "Error: server_config.deploy_mode in server_config.yaml is not one of "
<< "single, cluster_readonly, and cluster_writable." << std::endl;
kill(0, SIGUSR1);
}
@ -87,6 +92,7 @@ DBWrapper::StartService() {
int32_t omp_thread;
s = config.GetEngineConfigOmpThreadNum(omp_thread);
if (!s.ok()) {
std::cerr << s.ToString() << std::endl;
return s;
}
@ -105,6 +111,7 @@ DBWrapper::StartService() {
int32_t use_blas_threshold;
s = config.GetEngineConfigUseBlasThreshold(use_blas_threshold);
if (!s.ok()) {
std::cerr << s.ToString() << std::endl;
return s;
}
@ -115,6 +122,7 @@ DBWrapper::StartService() {
int32_t disk, days;
s = config.GetDBConfigArchiveDiskThreshold(disk);
if (!s.ok()) {
std::cerr << s.ToString() << std::endl;
return s;
}
@ -124,6 +132,7 @@ DBWrapper::StartService() {
s = config.GetDBConfigArchiveDaysThreshold(days);
if (!s.ok()) {
std::cerr << s.ToString() << std::endl;
return s;
}
@ -133,16 +142,20 @@ DBWrapper::StartService() {
opt.meta_.archive_conf_.SetCriterias(criterial);
// create db root folder
Status status = CommonUtil::CreateDirectory(opt.meta_.path_);
if (!status.ok()) {
std::cerr << "ERROR! Failed to create database root path: " << opt.meta_.path_ << std::endl;
s = CommonUtil::CreateDirectory(opt.meta_.path_);
if (!s.ok()) {
std::cerr << "Error: Failed to create database primary path: " << path
<< ". Possible reason: db_config.primary_path is wrong in server_config.yaml or not available."
<< std::endl;
kill(0, SIGUSR1);
}
for (auto& path : opt.meta_.slave_paths_) {
status = CommonUtil::CreateDirectory(path);
if (!status.ok()) {
std::cerr << "ERROR! Failed to create database slave path: " << path << std::endl;
s = CommonUtil::CreateDirectory(path);
if (!s.ok()) {
std::cerr << "Error: Failed to create database secondary path: " << path
<< ". Possible reason: db_config.secondary_path is wrong in server_config.yaml or not available."
<< std::endl;
kill(0, SIGUSR1);
}
}
@ -151,7 +164,8 @@ DBWrapper::StartService() {
try {
db_ = engine::DBFactory::Build(opt);
} catch (std::exception& ex) {
std::cerr << "ERROR! Failed to open database: " << ex.what() << std::endl;
std::cerr << "Error: failed to open database: " << ex.what()
<< ". Possible reason: the meta system does not work." << std::endl;
kill(0, SIGUSR1);
}
@ -161,6 +175,7 @@ DBWrapper::StartService() {
std::string preload_tables;
s = config.GetDBConfigPreloadTable(preload_tables);
if (!s.ok()) {
std::cerr << s.ToString() << std::endl;
return s;
}

4
core/src/server/Server.cpp
View File

@ -25,6 +25,7 @@
#include "server/DBWrapper.h"
#include "server/Server.h"
#include "server/grpc_impl/GrpcServer.h"
#include "src/version.h"
#include "utils/Log.h"
#include "utils/LogUtil.h"
#include "utils/SignalUtil.h"
@ -180,6 +181,9 @@ Server::Start() {
InitLog(log_config_file_);
// print version information
SERVER_LOG_INFO << "Milvus " << BUILD_TYPE << " version: v" << MILVUS_VERSION << ", built at " << BUILD_TIME;
server::Metrics::GetInstance().Init();
server::SystemInfo::GetInstance().Init();

2
core/src/server/grpc_impl/GrpcRequestTask.cpp
View File

@ -23,12 +23,12 @@
#include <vector>
//#include <gperftools/profiler.h>
#include "../../../version.h"
#include "GrpcServer.h"
#include "db/Utils.h"
#include "scheduler/SchedInst.h"
#include "server/DBWrapper.h"
#include "server/Server.h"
#include "src/version.h"
#include "utils/CommonUtil.h"
#include "utils/Log.h"
#include "utils/TimeRecorder.h"

2
core/src/utils/Log.h
View File

@ -17,7 +17,7 @@
#pragma once
#include "utils/easylogging++.h"
#include "external/easyloggingpp/easylogging++.h"
namespace milvus {

2
core/src/utils/LogUtil.h
View File

@ -17,8 +17,8 @@
#pragma once
#include "external/easyloggingpp/easylogging++.h"
#include "utils/Status.h"
#include "utils/easylogging++.h"
#include <sstream>
#include <string>

0
core/version.h.macro → core/src/version.h.macro
View File

1
core/src/wrapper/VecIndex.h
View File

@ -71,6 +71,7 @@ class VecIndex : public cache::DataObj {
virtual VecIndexPtr
CopyToCpu(const Config& cfg = Config()) = 0;
// TODO(linxj): Deprecated
virtual VecIndexPtr
Clone() = 0;

7
core/thirdparty/versions.txt vendored
View File

@ -1,18 +1,13 @@
BOOST_VERSION=1.70.0
BZIP2_VERSION=1.0.6
EASYLOGGINGPP_VERSION=v9.96.7
GTEST_VERSION=1.8.1
LZ4_VERSION=v1.9.1
MYSQLPP_VERSION=3.2.4
PROMETHEUS_VERSION=v0.7.0
SNAPPY_VERSION=1.1.7
SQLITE_VERSION=3280000
SQLITE_ORM_VERSION=master
YAMLCPP_VERSION=0.6.2
ZLIB_VERSION=v1.2.11
ZSTD_VERSION=v1.4.0
LIBUNWIND_VERSION=1.3.1
GPERFTOOLS_VERSION=2.7
GRPC_VERSION=master
ZLIB_VERSION=v1.2.11
# vim: set filetype=sh:

3
core/ubuntu_build_deps.sh
View File

@ -1,5 +1,6 @@
#!/bin/bash
sudo apt-get install -y gfortran libmysqlclient-dev mysql-client libcurl4-openssl-dev libboost-system-dev libboost-filesystem-dev libboost-serialization-dev
sudo apt-get install -y gfortran libmysqlclient-dev mysql-client libcurl4-openssl-dev libboost-system-dev \
libboost-filesystem-dev libboost-serialization-dev libboost-regex-dev
sudo ln -s /usr/lib/x86_64-linux-gnu/libmysqlclient.so /usr/lib/x86_64-linux-gnu/libmysqlclient_r.so

Some files were not shown because too many files have changed in this diff Show More