milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge from 0.7.1 to master (#1840) 

* update mishards images to 0.7.1 (fix #1799) (#1802)

* update mishards images to 0.7.1 (fix #1799)

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* [skip ci] Skip CI

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* [skip ci] Update CHANGELOG

Signed-off-by: JinHai-CN <hai.jin@zilliz.com>

* fix lsn meta bug (#1813)

* fix lsn meta bug

Signed-off-by: groot <yihua.mo@zilliz.com>

* fix mysql lsn meta bug

Signed-off-by: groot <yihua.mo@zilliz.com>

* Update CHANGELOG.md (#1814)

Signed-off-by: jinhai <hai.jin@zilliz.com>

* Fix compile (#1815)

* Update CHANGELOG.md

Signed-off-by: jinhai <hai.jin@zilliz.com>

* Update

Signed-off-by: jinhai <hai.jin@zilliz.com>

* Fix compile (#1816)

* Update CHANGELOG.md

Signed-off-by: jinhai <hai.jin@zilliz.com>

* Update

Signed-off-by: jinhai <hai.jin@zilliz.com>

* Update

Signed-off-by: jinhai <hai.jin@zilliz.com>

* Fix compile (#1817)

* Update CHANGELOG.md

Signed-off-by: jinhai <hai.jin@zilliz.com>

* Update

Signed-off-by: jinhai <hai.jin@zilliz.com>

* Update

Signed-off-by: jinhai <hai.jin@zilliz.com>

* Increase Resources

Signed-off-by: jinhai <hai.jin@zilliz.com>

* Decrease the compiling thread

Signed-off-by: jinhai <hai.jin@zilliz.com>

* #1818 Duplicate data generated after restart milvus server (#1819)

* #1818 Duplicate data generated after restart milvus server

Signed-off-by: groot <yihua.mo@zilliz.com>

* set compile thread number

Signed-off-by: groot <yihua.mo@zilliz.com>

* [skip-ci]Remove auto flush interval from config (#1837)

Signed-off-by: jinhai <hai.jin@zilliz.com>

* Fix lint

Signed-off-by: JinHai-CN <hai.jin@zilliz.com>

* Fix compile error

Signed-off-by: JinHai-CN <hai.jin@zilliz.com>

Co-authored-by: BossZou <40255591+BossZou@users.noreply.github.com>
Co-authored-by: groot <yhmo@zeronedata.com>
pull/1869/head
Jin Hai 2020-04-02 14:20:33 +08:00 committed by GitHub
parent 74a1657752
commit 3e03850169
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
16 changed files with 144 additions and 59 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
CHANGELOG.md
View File

@ -12,6 +12,7 @@ Please mark all change in change log and use the issue from GitHub
- \#1825 Add annoy index type in C++ sdk
## Improvement
- \#1784 Add Substructure and Superstructure in http module
## Task
@ -34,7 +35,8 @@ Please mark all change in change log and use the issue from GitHub
- \#1735 Fix search out of memory with ivf_flat
- \#1747 Expected error status if search with partition_tag not existed
- \#1756 Fix memory exhausted during searching
- \#1781 Fix search hang with SQ8H
- \#1781 Fix search hang with SQ8H
- \#1818 Duplicate data generated after restart milvus server
## Feature
- \#261 Integrate ANNOY into Milvus
@ -57,7 +59,7 @@ Please mark all change in change log and use the issue from GitHub
- \#1698 Upgrade mishards to v0.7.0
- \#1719 Improve Milvus log
- \#1754 Optimize behavior to get file ids from metadata in mishards
- \#1784 Add Substructure and Superstructure in http module
- \#1799 Update docker images to 0.7.1 in mishards
## Task
@ -280,7 +282,7 @@ Please mark all change in change log and use the issue from GitHub
## Improvement
- \#204 improve grpc performance in search
- \#207 Add more unittest for config set/get
- \#208 optimize unittest to support run single test more easily
- \#208 Optimize unittest to support run single test more easily
- \#284 Change C++ SDK to shared library
- \#260 C++ SDK README
@ -677,7 +679,7 @@ Please mark all change in change log and use the issue from GitHub
- MS-6 Implement SDK interface part 1
- MS-16 Implement metrics without prometheus
- MS-21 Implement SDK interface part 2
- MS-26 cmake. Add thirdparty packages
- MS-26 CMake. Add thirdparty packages
- MS-31 cmake: add prometheus
- MS-33 cmake: add -j4 to make third party packages build faster
- MS-27 support gpu config and disable license build config in cmake

4
ci/jenkins/step/build.groovy
View File

@ -3,9 +3,9 @@ dir ("ci/scripts") {
def checkResult = sh(script: "./check_ccache.sh -l ${params.JFROG_ARTFACTORY_URL}/ccache", returnStatus: true)
if ("${BINARY_VERSION}" == "gpu") {
sh "/bin/bash --login -c \". ./before-install.sh && ./build.sh -t ${params.BUILD_TYPE} -i ${env.MILVUS_INSTALL_PREFIX} --with_fiu --coverage -l -g -u\""
sh "/bin/bash --login -c \". ./before-install.sh && ./build.sh -t ${params.BUILD_TYPE} -j4 -i ${env.MILVUS_INSTALL_PREFIX} --with_fiu --coverage -l -g -u\""
} else {
sh "/bin/bash --login -c \". ./before-install.sh && ./build.sh -t ${params.BUILD_TYPE} -i ${env.MILVUS_INSTALL_PREFIX} --with_fiu --coverage -l -u\""
sh "/bin/bash --login -c \". ./before-install.sh && ./build.sh -t ${params.BUILD_TYPE} -j4 -i ${env.MILVUS_INSTALL_PREFIX} --with_fiu --coverage -l -u\""
}
sh "./update_ccache.sh -l ${params.JFROG_ARTFACTORY_URL}/ccache -u ${USERNAME} -p ${PASSWORD}"
}

5
core/conf/demo/server_config.yaml
View File

@ -47,14 +47,9 @@ server_config:
# | '*' means preload all existing tables (single-quote or | | |
# | double-quote required). | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# auto_flush_interval | The interval, in seconds, at which Milvus automatically | Integer | 1 (s) |
# | flushes data to disk. | | |
# | 0 means disable the regular flush. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
db_config:
backend_url: sqlite://:@:/
preload_table:
auto_flush_interval: 1
#----------------------+------------------------------------------------------------+------------+-----------------+
# Storage Config | Description | Type | Default |

5
core/conf/server_cpu_config.template
View File

@ -47,14 +47,9 @@ server_config:
# | '*' means preload all existing tables (single-quote or | | |
# | double-quote required). | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# auto_flush_interval | The interval, in seconds, at which Milvus automatically | Integer | 1 (s) |
# | flushes data to disk. | | |
# | 0 means disable the regular flush. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
db_config:
backend_url: sqlite://:@:/
preload_table:
auto_flush_interval: 1
#----------------------+------------------------------------------------------------+------------+-----------------+
# Storage Config | Description | Type | Default |

5
core/conf/server_gpu_config.template
View File

@ -47,14 +47,9 @@ server_config:
# | '*' means preload all existing tables (single-quote or | | |
# | double-quote required). | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# auto_flush_interval | The interval, in seconds, at which Milvus automatically | Integer | 1 (s) |
# | flushes data to disk. | | |
# | 0 means disable the regular flush. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
db_config:
backend_url: sqlite://:@:/
preload_table:
auto_flush_interval: 1
#----------------------+------------------------------------------------------------+------------+-----------------+
# Storage Config | Description | Type | Default |

117
core/src/db/meta/MySQLMetaImpl.cpp
View File

@ -136,6 +136,11 @@ class MetaSchema {
MetaFields fields_;
};
// Environment schema
static const MetaSchema ENVIRONMENT_SCHEMA(META_ENVIRONMENT, {
MetaField("global_lsn", "BIGINT", "NOT NULL"),
});
// Tables schema
static const MetaSchema TABLES_SCHEMA(META_TABLES, {
MetaField("id", "BIGINT", "PRIMARY KEY AUTO_INCREMENT"),
@ -241,6 +246,11 @@ MySQLMetaImpl::ValidateMetaSchema() {
return schema.IsEqual(exist_fields);
};
// verify Environment
if (!validate_func(ENVIRONMENT_SCHEMA)) {
throw Exception(DB_INCOMPATIB_META, "Meta Environment schema is created by Milvus old version");
}
// verify Tables
if (!validate_func(TABLES_SCHEMA)) {
throw Exception(DB_INCOMPATIB_META, "Meta Tables schema is created by Milvus old version");
@ -349,6 +359,19 @@ MySQLMetaImpl::Initialize() {
throw Exception(DB_META_TRANSACTION_FAILED, msg);
}
// step 9: create meta table Environment
InitializeQuery << "CREATE TABLE IF NOT EXISTS " << ENVIRONMENT_SCHEMA.name() << " ("
<< ENVIRONMENT_SCHEMA.ToString() + ");";
ENGINE_LOG_DEBUG << "MySQLMetaImpl::Initialize: " << InitializeQuery.str();
initialize_query_exec = InitializeQuery.exec();
if (!initialize_query_exec) {
std::string msg = "Failed to create meta table 'Environment' in MySQL";
ENGINE_LOG_ERROR << msg;
throw Exception(DB_META_TRANSACTION_FAILED, msg);
}
return Status::OK();
}
@ -988,6 +1011,32 @@ MySQLMetaImpl::UpdateTableFlushLSN(const std::string& collection_id, uint64_t fl
Status
MySQLMetaImpl::GetTableFlushLSN(const std::string& collection_id, uint64_t& flush_lsn) {
try {
server::MetricCollector metric;
mysqlpp::StoreQueryResult res;
{
mysqlpp::ScopedConnection connectionPtr(*mysql_connection_pool_, safe_grab_);
if (connectionPtr == nullptr) {
return Status(DB_ERROR, "Failed to connect to meta server(mysql)");
}
mysqlpp::Query statement = connectionPtr->query();
statement << "SELECT flush_lsn FROM " << META_TABLES << " WHERE collection_id = " << mysqlpp::quote
<< collection_id << ";";
ENGINE_LOG_DEBUG << "MySQLMetaImpl::GetTableFlushLSN: " << statement.str();
res = statement.store();
} // Scoped Connection
if (!res.empty()) {
flush_lsn = res[0]["flush_lsn"];
}
} catch (std::exception& e) {
return HandleException("GENERAL ERROR WHEN GET TABLE FLUSH_LSN", e.what());
}
return Status::OK();
}
@ -2541,11 +2590,79 @@ MySQLMetaImpl::DiscardFiles(int64_t to_discard_size) {
Status
MySQLMetaImpl::SetGlobalLastLSN(uint64_t lsn) {
try {
server::MetricCollector metric;
{
mysqlpp::ScopedConnection connectionPtr(*mysql_connection_pool_, safe_grab_);
if (connectionPtr == nullptr) {
return Status(DB_ERROR, "Failed to connect to meta server(mysql)");
}
bool first_create = false;
{
mysqlpp::StoreQueryResult res;
mysqlpp::Query statement = connectionPtr->query();
statement << "SELECT global_lsn FROM " << META_ENVIRONMENT << ";";
res = statement.store();
if (res.num_rows() == 0) {
first_create = true;
}
}
if (first_create) { // first time to get global lsn
mysqlpp::Query statement = connectionPtr->query();
statement << "INSERT INTO " << META_ENVIRONMENT << " VALUES(" << lsn << ");";
ENGINE_LOG_DEBUG << "MySQLMetaImpl::SetGlobalLastLSN: " << statement.str();
if (!statement.exec()) {
return HandleException("QUERY ERROR WHEN SET GLOBAL LSN", statement.error());
}
} else {
mysqlpp::Query statement = connectionPtr->query();
statement << "UPDATE " << META_ENVIRONMENT << " SET global_lsn = " << lsn << ";";
ENGINE_LOG_DEBUG << "MySQLMetaImpl::SetGlobalLastLSN: " << statement.str();
if (!statement.exec()) {
return HandleException("QUERY ERROR WHEN SET GLOBAL LSN", statement.error());
}
}
} // Scoped Connection
ENGINE_LOG_DEBUG << "Successfully update global_lsn = " << lsn;
} catch (std::exception& e) {
return HandleException("QUERY ERROR WHEN SET GLOBAL LSN", e.what());
}
return Status::OK();
}
Status
MySQLMetaImpl::GetGlobalLastLSN(uint64_t& lsn) {
try {
server::MetricCollector metric;
mysqlpp::StoreQueryResult res;
{
mysqlpp::ScopedConnection connectionPtr(*mysql_connection_pool_, safe_grab_);
if (connectionPtr == nullptr) {
return Status(DB_ERROR, "Failed to connect to meta server(mysql)");
}
mysqlpp::Query statement = connectionPtr->query();
statement << "SELECT global_lsn FROM " << META_ENVIRONMENT << ";";
ENGINE_LOG_DEBUG << "MySQLMetaImpl::GetGlobalLastLSN: " << statement.str();
res = statement.store();
} // Scoped Connection
if (!res.empty()) {
lsn = res[0]["global_lsn"];
}
} catch (std::exception& e) {
return HandleException("GENERAL ERROR WHEN GET GLOBAL LSN", e.what());
}
return Status::OK();
}

12
core/src/db/meta/SqliteMetaImpl.cpp
View File

@ -520,7 +520,7 @@ SqliteMetaImpl::UpdateTableFlushLSN(const std::string& collection_id, uint64_t f
ConnectorPtr->update_all(set(c(&CollectionSchema::flush_lsn_) = flush_lsn),
where(c(&CollectionSchema::collection_id_) == collection_id));
ENGINE_LOG_DEBUG << "Successfully update collection flush_lsn, collection id = " << collection_id;
ENGINE_LOG_DEBUG << "Successfully update collection flush_lsn, collection id = " << collection_id << " flush_lsn = " << flush_lsn;;
} catch (std::exception& e) {
std::string msg = "Encounter exception when update collection lsn: collection_id = " + collection_id;
return HandleException(msg, e.what());
@ -718,10 +718,11 @@ SqliteMetaImpl::UpdateTableIndex(const std::string& collection_id, const TableIn
std::lock_guard<std::mutex> meta_lock(meta_mutex_);
auto tables = ConnectorPtr->select(
columns(&CollectionSchema::id_, &CollectionSchema::state_, &CollectionSchema::dimension_, &CollectionSchema::created_on_,
&CollectionSchema::flag_, &CollectionSchema::index_file_size_, &CollectionSchema::owner_table_,
&CollectionSchema::partition_tag_, &CollectionSchema::version_),
where(c(&CollectionSchema::collection_id_) == collection_id and c(&CollectionSchema::state_) != (int)CollectionSchema::TO_DELETE));
columns(&CollectionSchema::id_, &CollectionSchema::state_, &CollectionSchema::dimension_, &CollectionSchema::created_on_,
&CollectionSchema::flag_, &CollectionSchema::index_file_size_, &CollectionSchema::owner_table_,
&CollectionSchema::partition_tag_, &CollectionSchema::version_, &CollectionSchema::flush_lsn_),
where(c(&CollectionSchema::collection_id_) == collection_id and c(&CollectionSchema::state_) != (int)CollectionSchema::TO_DELETE));
if (tables.size() > 0) {
meta::CollectionSchema table_schema;
@ -735,6 +736,7 @@ SqliteMetaImpl::UpdateTableIndex(const std::string& collection_id, const TableIn
table_schema.owner_table_ = std::get<6>(tables[0]);
table_schema.partition_tag_ = std::get<7>(tables[0]);
table_schema.version_ = std::get<8>(tables[0]);
table_schema.flush_lsn_ = std::get<9>(tables[0]);
table_schema.engine_type_ = index.engine_type_;
table_schema.index_params_ = index.extra_params_.dump();
table_schema.metric_type_ = index.metric_type_;

10
core/src/main.cpp
View File

@ -26,10 +26,10 @@ void
print_help(const std::string& app_name) {
std::cout << std::endl << "Usage: " << app_name << " [OPTIONS]" << std::endl << std::endl;
std::cout << " Options:" << std::endl;
std::cout << " -h --help Print this help" << std::endl;
std::cout << " -c --conf_file filename Read configuration from the file" << std::endl;
std::cout << " -d --daemon Daemonize this application" << std::endl;
std::cout << " -p --pid_file filename PID file used by daemonized app" << std::endl;
std::cout << " -h --help Print this help." << std::endl;
std::cout << " -c --conf_file filename Read configuration from the file." << std::endl;
std::cout << " -d --daemon Daemonize this application." << std::endl;
std::cout << " -p --pid_file filename PID file used by daemonized app." << std::endl;
std::cout << std::endl;
}
@ -41,7 +41,7 @@ print_banner() {
std::cout << " / /|_/ // // /_| |/ / /_/ /\\ \\ " << std::endl;
std::cout << " /_/ /_/___/____/___/\\____/___/ " << std::endl;
std::cout << std::endl;
std::cout << "Welcome to Milvus!" << std::endl;
std::cout << "Welcome to use Milvus!" << std::endl;
std::cout << "Milvus " << BUILD_TYPE << " version: v" << MILVUS_VERSION << ", built at " << BUILD_TIME << ", with "
#ifdef WITH_MKL
<< "MKL"

4
shards/all_in_one/all_in_one.yml
View File

@ -3,7 +3,7 @@ services:
milvus_wr:
runtime: nvidia
restart: always
image: milvusdb/milvus:0.7.0-gpu-d031120-40c7e8
image: milvusdb/milvus:0.7.1-gpu-d032920-3cdba5
ports:
- "0.0.0.0:19540:19530"
volumes:
@ -13,7 +13,7 @@ services:
milvus_ro:
runtime: nvidia
restart: always
image: milvusdb/milvus:0.7.0-gpu-d031120-40c7e8
image: milvusdb/milvus:0.7.1-gpu-d032920-3cdba5
ports:
- "0.0.0.0:19541:19530"
volumes:

5
shards/all_in_one/ro_server.yml
View File

@ -47,14 +47,9 @@ server_config:
# | '*' means preload all existing tables (single-quote or | | |
# | double-quote required). | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# auto_flush_interval | The interval, in seconds, at which Milvus automatically | Integer | 1 (s) |
# | flushes data to disk. | | |
# | 0 means disable the regular flush. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
db_config:
backend_url: sqlite://:@:/
preload_table:
auto_flush_interval: 1
#----------------------+------------------------------------------------------------+------------+-----------------+
# Storage Config | Description | Type | Default |

5
shards/all_in_one/wr_server.yml
View File

@ -47,14 +47,9 @@ server_config:
# | '*' means preload all existing tables (single-quote or | | |
# | double-quote required). | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# auto_flush_interval | The interval, in seconds, at which Milvus automatically | Integer | 1 (s) |
# | flushes data to disk. | | |
# | 0 means disable the regular flush. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
db_config:
backend_url: sqlite://:@:/
preload_table:
auto_flush_interval: 1
#----------------------+------------------------------------------------------------+------------+-----------------+
# Storage Config | Description | Type | Default |

4
shards/all_in_one_with_mysql/all_in_one.yml
View File

@ -18,7 +18,7 @@ services:
milvus_wr:
runtime: nvidia
restart: always
image: milvusdb/milvus:0.7.0-gpu-d031120-40c7e8
image: milvusdb/milvus:0.7.1-gpu-d032920-3cdba5
volumes:
- /tmp/milvus/db:/var/lib/milvus/db
- ./wr_server.yml:/var/lib/milvus/conf/server_config.yaml
@ -29,7 +29,7 @@ services:
milvus_ro:
runtime: nvidia
restart: always
image: milvusdb/milvus:0.7.0-gpu-d031120-40c7e8
image: milvusdb/milvus:0.7.1-gpu-d032920-3cdba5
volumes:
- /tmp/milvus/db:/var/lib/milvus/db
- ./ro_server.yml:/var/lib/milvus/conf/server_config.yaml

5
shards/all_in_one_with_mysql/ro_server.yml
View File

@ -47,14 +47,9 @@ server_config:
# | '*' means preload all existing tables (single-quote or | | |
# | double-quote required). | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# auto_flush_interval | The interval, in seconds, at which Milvus automatically | Integer | 1 (s) |
# | flushes data to disk. | | |
# | 0 means disable the regular flush. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
db_config:
backend_url: mysql://root:milvusroot@milvus-mysql:3306/milvus
preload_table:
auto_flush_interval: 1
#----------------------+------------------------------------------------------------+------------+-----------------+
# Storage Config | Description | Type | Default |

5
shards/all_in_one_with_mysql/wr_server.yml
View File

@ -47,14 +47,9 @@ server_config:
# | '*' means preload all existing tables (single-quote or | | |
# | double-quote required). | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# auto_flush_interval | The interval, in seconds, at which Milvus automatically | Integer | 1 (s) |
# | flushes data to disk. | | |
# | 0 means disable the regular flush. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
db_config:
backend_url: mysql://root:milvusroot@milvus-mysql:3306/milvus
preload_table:
auto_flush_interval: 1
#----------------------+------------------------------------------------------------+------------+-----------------+
# Storage Config | Description | Type | Default |

3
shards/mishards/service_handler.py
View File

@ -151,7 +151,6 @@ class ServiceHandler(milvus_pb2_grpc.MilvusServiceServicer):
logger.error("Search fail {}".format(ret.status))
end = time.time()
all_topk_results.append(ret)
with self.tracer.start_span('do_search', child_of=p_span) as span:
@ -189,7 +188,7 @@ class ServiceHandler(milvus_pb2_grpc.MilvusServiceServicer):
reason=_status.message)
_status, _table_schema = unpacks
# if _status.error_code == 0:
# if _status.error_code != 0:
# logging.warning('[CreateTable] table schema error occurred: {}'.format(_status))
# return _status

4
shards/requirements.txt
View File

@ -14,8 +14,8 @@ py==1.8.0
pyasn1==0.4.7
pyasn1-modules==0.2.6
pylint==2.3.1
#pymilvus==0.2.8
pymilvus-test==0.3.3
pymilvus==0.2.9
#pymilvus-test==0.3.3
pyparsing==2.4.0
pytest==4.6.3
pytest-level==0.1.1