Merge branch 'master' into fix_tls_block

docs/en-US/source/5.developer-manual/5.ecosystem-tools/4.log.md

@@ -11,11 +11,11 @@ TuGraph keeps two types of logs: server logs and audit logs. Server logs record
 ### 2.1.Server Log Configuration Items
 The output directory of server logs can be specified through the `log_dir` configuration item. The level of log can be specified through the `verbose` configuration item.
 
-The `log_dir` configuration item is empty by default. If `log_dir` is empty, then all logs will be write to the console, the maximum size of a single log file is 256MB.
+The `log_dir` configuration item is empty by default. If `log_dir` configuration item is empty, then all logs will be write to the console(nothing will be written to the console if the `log_dir` configuration item is empty under daemon mode); if `log_dir` configuration item is configured specifically, log files will bew generated under that path. The maximum size of a single log file is 256MB.
 
 The `verbose` configuration item controls the level of log, and is divided into three levels of `0, 1, 2`. Ther verbosity of log record grows as the number grows. The default level is `1`. When the level is set to `2`, the server will print logs in `DEBUG` level and above. When the level is set to `1`, the server will print logs in `INFO` level and above. When the level is set to `0`, the server will print log in `ERROR` level and above.
 
-### 2.3.Example of Server Log Output Macro Usage
+### 2.2.Example of Server Log Output Macro Usage
 
 If the developer wants to add logs to the code during development, they can refer to the following example:
 
@@ -36,8 +36,39 @@ void LogExample() {
 
 You can also refer to the log macro usage in test/test_lgraph_log.cpp.
 
+### 2.3.Procedure log
+
+Developers can use the log function to output debug information that they need to the log for looking up and assisting development during the development of procedures. Debug information will be output to the same log file as the server log (if `log_dir` configuration item is not specified, it will also be output to the console)
+
+#### 2.3.1.Cpp procedure
+Please use the log macro provided in 2.2 to output debug information, and avoid using output methods such as cout or printf. For specific usage, please refer to the following sample code (see `procedures/demo/log_demo.cpp` for details)
+
+```
+#include <stdlib.h>
+#include "lgraph/lgraph.h"
+#include "tools/lgraph_log.h"  // add log dependency
+using namespace lgraph_api;
+
+void LogExample() {
+    LOG_DEBUG() << "This is a debug level log message.";
+    LOG_INFO() << "This is a info level log message.";
+    LOG_WARN() << "This is a warning level log message.";
+    LOG_ERROR() << "This is a error level log message.";
+}
+
+extern "C" bool Process(GraphDB& db, const std::string& request, std::string& response) {
+    response = "TuGraph log demo";
+    LogExample();
+    return true;
+}
+```
+After inserting the above sample code into the database as a procedure and running it, you can see the corresponding log entries in the log file.
+
+#### 2.3.1.python procedure
+Please use Python's built-in print to output debug information. The debug information will be merged into a WARN-level log entry and output to the log file after the procedure is executed.
+
 ## 3.Audit log
 
 Audit logs record each request and response, as well as the user who sent the request and when the request received. Audit logging can only be turned on or off. The results can be queried using the TuGraph visualization tool and the REST API.
 
-To enable the Audit Log, you need to set the `enable_audit_log` parameter to `true` in the configuration file. For the configuration file and parameter descriptions, see:[Tugraph Running/Service configuration](../2.running/2.tugraph-running.md/#4service-configuration)
+To enable the Audit Log, you need to set the `enable_audit_log` parameter to `true` in the configuration file. For the configuration file and parameter descriptions, see:[Tugraph Running/Service configuration](../2.running/2.tugraph-running.md/#4service-configuration)

docs/zh-CN/source/5.developer-manual/5.ecosystem-tools/4.log.md

@@ -12,11 +12,11 @@ TuGraph 保留两种类型的日志：服务器日志和审计日志。服务器
 
 服务器日志的输出位置可以通过`log_dir`配置指定。服务器日志详细程度可通过`verbose`配置项指定。
 
-`log_dir`配置项默认为空。若`log_dir`为空，则所有日志会输出到控制台；若手动指定`log_dir`，则日志文件会生成在对应的路径下面，单个日志文件最大大小为256MB。
+`log_dir`配置项默认为空。若`log_dir`配置项为空，则所有日志会输出到控制台(daemon模式下若log_dir配置项为空则不会向console输出任何日志)；若手动指定`log_dir`配置项，则日志文件会生成在对应的路径下面。单个日志文件最大大小为256MB。
 
 `verbose`配置项控制日志的详细程度，从粗到细分为`0, 1, 2`三个等级，默认等级为`1`。等级为`2`时，日志记录最详细，服务器将打印`DEBUG`及以上等级的全部日志信息；等级为`1`时，服务器将仅打印`INFO`等级及以上的主要事件的日志；等级为`0`时，服务器将仅打印`ERROR`等级及以上的错误日志。
 
-### 2.3.服务器日志输出宏使用示例
+### 2.2.服务器日志输出宏使用示例
 
 如果开发者在开发过程中希望在代码中添加日志，可以参考如下示例
 
@@ -36,8 +36,39 @@ void LogExample() {
 ```
 更多用法可以参考test/test_lgraph_log.cpp中的日志宏的使用方法
 
+### 2.3.存储过程日志
+
+用户在存储过程的编写过程中可以使用日志功能将所需的调试信息输出到日志中进行查看，辅助开发。调试信息会输出到与服务器日志相同的日志文件中(如未指定`log_dir`则同样输出至console)
+
+#### 2.3.1.cpp存储过程
+请使用2.2中提供的log宏输出调试信息，避免使用cout或者printf等输出方式。具体使用方式可参考如下示例代码（详见`procedures/demo/log_demo.cpp`）
+
+```
+#include <stdlib.h>
+#include "lgraph/lgraph.h"
+#include "tools/lgraph_log.h"  // add log dependency
+using namespace lgraph_api;
+
+void LogExample() {
+    LOG_DEBUG() << "This is a debug level log message.";
+    LOG_INFO() << "This is a info level log message.";
+    LOG_WARN() << "This is a warning level log message.";
+    LOG_ERROR() << "This is a error level log message.";
+}
+
+extern "C" bool Process(GraphDB& db, const std::string& request, std::string& response) {
+    response = "TuGraph log demo";
+    LogExample();
+    return true;
+}
+```
+将以上示例代码作为存储过程插入数据库并运行后，可以在日志文件中看到相应的日志条目。
+
+#### 2.3.1.python存储过程
+请使用python自带的print输出调试信息，调试信息会在存储过程运行结束后合并为一条WARN等级的日志条目输出至日志文件中。
+
 ## 3.审计日志
 
 审核日志记录每个请求和响应，以及发送请求的用户以及收到请求的时间。审核日志只能是打开或关闭状态。可以使用 TuGraph 可视化工具和 REST API 查询结果。
 
-开启审计日志需要在配置文件中将`enable_audit_log`参数设置为`true`。配置文件和配置参数说明详见：[数据库运行/服务配置](../2.running/2.tugraph-running.md/#4服务配置)。
+开启审计日志需要在配置文件中将`enable_audit_log`参数设置为`true`。配置文件和配置参数说明详见：[数据库运行/服务配置](../2.running/2.tugraph-running.md/#4服务配置)。

include/tools/lgraph_log.h

@@ -27,6 +27,7 @@
 #include <boost/log/utility/manipulators/add_value.hpp>
 #include <boost/phoenix/bind/bind_function.hpp>
 #include <boost/core/null_deleter.hpp>
+#include "tools/json.hpp"
 
 namespace lgraph_log {
 
@@ -38,11 +39,41 @@ namespace expr = boost::log::expressions;
 namespace keywords = boost::log::keywords;
 
 using boost::shared_ptr;
+using json = nlohmann::json;
 
 typedef sinks::synchronous_sink< sinks::text_file_backend > file_sink;
 typedef sinks::synchronous_sink< sinks::text_ostream_backend > stream_sink;
 typedef sinks::synchronous_sink< sinks::text_ostream_backend > ut_sink;
 
+// Define log macro
+#define LGRAPH_LOG(LEVEL) BOOST_LOG_SEV(::lgraph_log::debug_logger::get(), \
+  ::lgraph_log::severity_level::LEVEL) \
+  << ::lgraph_log::logging::add_value("Line", __LINE__) \
+  << ::lgraph_log::logging::add_value("File", __FILE__) \
+
+#define LOG_DEBUG() LGRAPH_LOG(DEBUG)
+#define LOG_INFO() LGRAPH_LOG(INFO)
+#define LOG_WARN() LGRAPH_LOG(WARNING)
+#define LOG_ERROR() LGRAPH_LOG(ERROR)
+
+#define LOG_FATAL() lgraph_log::FatalLogger(__FILE__, __LINE__)
+
+#define FMA_UT_LOG(LEVEL) BOOST_LOG_SEV(::lgraph_log::debug_logger::get(), \
+  LEVEL) \
+  << ::lgraph_log::logging::add_value("Line", __LINE__) \
+  << ::lgraph_log::logging::add_value("File", __FILE__) \
+
+#define AUDIT_LOG() BOOST_LOG(::lgraph_log::audit_logger::get())
+
+#define AUDIT_PREFIX "audit_"
+
+// LogType includes the following type: debug, audit
+// debug type for general log
+// audit type for audit log
+#define DEBUG_TYPE "debug"
+#define AUDIT_TYPE "audit"
+BOOST_LOG_ATTRIBUTE_KEYWORD(log_type_attr, "LogType", std::string)
+
 enum severity_level {
     TRACE,
     DEBUG,
@@ -128,7 +159,8 @@ class LoggerManager {
                 keywords::enable_final_rotation = false,
                 keywords::auto_flush = true,
                 keywords::rotation_size = rotation_size_));
-            file_sink_->set_filter(expr::attr< severity_level >("Severity") >= level_);
+            file_sink_->set_filter(expr::attr< severity_level >("Severity") >= level_ &&
+                log_type_attr == DEBUG_TYPE);
             file_sink_->set_formatter(&formatter);
 
             logging::core::get()->add_sink(file_sink_);
@@ -143,7 +175,8 @@ class LoggerManager {
                     boost::shared_ptr< std::ostream >(console_stream_));
             }
             stream_sink_->locked_backend()->auto_flush(true);
-            stream_sink_->set_filter(expr::attr< severity_level >("Severity") >= level_);
+            stream_sink_->set_filter(expr::attr< severity_level >("Severity") >= level_ &&
+                log_type_attr == DEBUG_TYPE);
             stream_sink_->set_formatter(&formatter);
 
             logging::core::get()->add_sink(stream_sink_);
@@ -166,9 +199,11 @@ class LoggerManager {
     void SetLevel(severity_level level) {
       level_ = level;
       if (!log_dir_.empty()) {
-            file_sink_->set_filter(expr::attr< severity_level >("Severity") >= level_);
+            file_sink_->set_filter(expr::attr< severity_level >("Severity") >= level_ &&
+                log_type_attr == DEBUG_TYPE);
       } else {
-        stream_sink_->set_filter(expr::attr< severity_level >("Severity") >= level_);
+        stream_sink_->set_filter(expr::attr< severity_level >("Severity") >= level_ &&
+                log_type_attr == DEBUG_TYPE);
       }
     }
 
@@ -237,6 +272,9 @@ class LoggerManager {
 
 BOOST_LOG_INLINE_GLOBAL_LOGGER_INIT(debug_logger, src::severity_logger_mt< severity_level >) {
     src::severity_logger_mt< severity_level > lg;
+    attrs::constant< std::string > debug_type(DEBUG_TYPE);
+    lg.add_attribute("LogType", debug_type);
+
     // Init empty console log first if not inited
     if (!LoggerManager::GetInstance().IsInited()) {
       boost::shared_ptr< stream_sink > empty_sink =
@@ -249,16 +287,62 @@ BOOST_LOG_INLINE_GLOBAL_LOGGER_INIT(debug_logger, src::severity_logger_mt< sever
     return lg;
 };
 
-// Define log macro
-#define LGRAPH_LOG(LEVEL) BOOST_LOG_SEV(::lgraph_log::debug_logger::get(), \
-  ::lgraph_log::severity_level::LEVEL) \
-  << ::lgraph_log::logging::add_value("Line", __LINE__) \
-  << ::lgraph_log::logging::add_value("File", __FILE__) \
+BOOST_LOG_INLINE_GLOBAL_LOGGER_INIT(audit_logger, src::logger_mt) {
+  src::logger_mt lg;
+  attrs::constant< std::string > audit_type(AUDIT_TYPE);
+  lg.add_attribute("LogType", audit_type);
+  return lg;
+};
 
-#define LOG_DEBUG() LGRAPH_LOG(DEBUG)
-#define LOG_INFO() LGRAPH_LOG(INFO)
-#define LOG_WARN() LGRAPH_LOG(WARNING)
-#define LOG_ERROR() LGRAPH_LOG(ERROR)
+class AuditLogger {
+ private:
+    std::string log_dir_;
+    size_t rotation_size_ = 256*1024*1024;
+    boost::shared_ptr< file_sink > audit_sink_;
+    bool global_inited_ = false;
+
+ public:
+    void Init(std::string log_dir = "logs/", size_t rotation_size = 256*1024*1024) {
+        // Set up log directory
+        log_dir_ = log_dir;
+        if (log_dir_.back() != '/') {
+            log_dir_ += '/';
+        }
+        rotation_size_ = rotation_size;
+
+        // Set up sink for audit log
+        audit_sink_ = boost::shared_ptr< file_sink > (new file_sink(
+            keywords::file_name = log_dir_ + AUDIT_PREFIX + "%Y%m%d_%H%M%S.log",
+            keywords::open_mode = std::ios_base::out | std::ios_base::app,
+            keywords::enable_final_rotation = false,
+            keywords::auto_flush = true,
+            keywords::rotation_size = rotation_size_));
+        audit_sink_->locked_backend()->set_file_collector(sinks::file::make_collector(
+            keywords::target = log_dir_));
+        audit_sink_->locked_backend()->scan_for_files();
+        audit_sink_->set_filter(log_type_attr == AUDIT_TYPE);
+
+        // Add sinks to log core
+        logging::core::get()->add_sink(audit_sink_);
+
+        global_inited_ = true;
+    }
+
+    // write a json record to log file.
+    void WriteLog(json log_msg) {
+        AUDIT_LOG() << log_msg.dump();
+    }
+
+    // remove sink from log core
+    void Close() {
+        logging::core::get()->remove_sink(audit_sink_);
+    }
+
+    static AuditLogger& GetInstance() {
+        static AuditLogger instance;
+        return instance;
+    }
+};
 
 class FatalLogger {
  public:
@@ -282,11 +366,5 @@ class FatalLogger {
     std::string file_;
     int line_;
 };
-#define LOG_FATAL() lgraph_log::FatalLogger(__FILE__, __LINE__)
-
-#define FMA_UT_LOG(LEVEL) BOOST_LOG_SEV(::lgraph_log::debug_logger::get(), \
-  LEVEL) \
-  << ::lgraph_log::logging::add_value("Line", __LINE__) \
-  << ::lgraph_log::logging::add_value("File", __FILE__) \
 
 }  // namespace lgraph_log

procedures/demo/log_demo.cpp

@@ -0,0 +1,31 @@
+/**
+ * Copyright 2022 AntGroup CO., Ltd.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ */
+
+#include <stdlib.h>
+#include "lgraph/lgraph.h"
+#include "tools/lgraph_log.h"  // add log dependency
+using namespace lgraph_api;
+
+void LogExample() {
+    LOG_DEBUG() << "This is a debug level log message.";
+    LOG_INFO() << "This is a info level log message.";
+    LOG_WARN() << "This is a warning level log message.";
+    LOG_ERROR() << "This is a error level log message.";
+}
+
+extern "C" bool Process(GraphDB& db, const std::string& request, std::string& response) {
+    response = "TuGraph log demo";
+    LogExample();
+    return true;
+}