forked from opengaussexamples/examples
first-commit:1.0
This commit is contained in:
Tang Wuguo
parent
d9162144bb
commit
8bc17711c1
|
|
@ -0,0 +1,100 @@
|
|||
# openGauss用户操作模型
|
||||
*Tang Wuguo, tangwg@csu.edu.cn*
|
||||
|
||||
本demo使用openGauss中的[AI模块](https://docs.opengauss.org/zh/docs/5.0.0/docs/AIFeatureGuide/%E5%8E%9F%E7%94%9FDB4AI%E5%BC%95%E6%93%8E.html)以及数据库产生的审计日志，对数据库用户的操作进行建模并进行用户画像。
|
||||
|
||||
在openGauss中提供了完善的日志系统，其中的审计日志中就记录了用户每次的操作，日志中每一条记录表示: 时间t用户A对客体B进行了操作C。首先对原始的审计日志进行筛选过滤去除噪声记录，然后使用特征工程设计了基于统计学上的特征。最后让AI模型使用前面的特征自动学习数据更深层的特征。最后模型具有识别不同类型用户的能力。
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## 1.文件说明🚀
|
||||
- **code1-SimulateAuditLog**
|
||||
由于没有找到公开的openGauss审计日志，所以在这里我使用python脚本模拟了一个教务管理系统场景下的审计日志
|
||||
- `\code\1.1_simulateOperation.ipynb`
|
||||
模拟了教务管理系统中4种用户(学生端、教师端、教务处、管理员)的操作
|
||||
- `\code\1.2_analyzeLog.ipynb`
|
||||
对产生的审计日志进行分析其频数分布，包括：操作的主体即用户，操作客体，操作类型，涉及到的数据库
|
||||
- `\data`
|
||||
- `xxx.csv`: 包括模拟操作需要的源数据(学生、教师信息、课程表等)
|
||||
- `hist_xxx.jpg`: 分析日志保存的分布图
|
||||
- `audit_log_202307101107.csv`: 经过模拟操作得到的审计日志
|
||||
- `log_01.csv`: 初步筛选后得到的日志
|
||||
- **code2-UserPortraitModel**
|
||||
用户操作建模&用户画像
|
||||
- `2.1FeatureEngineering.sql`
|
||||
收集&筛选日志，构建特征
|
||||
- `2.2UserPortraitModel.sql`
|
||||
训练、评估模型, 用户画像, 识别危险用户
|
||||
- **data**
|
||||
- `audit_log.csv`: 审计日志
|
||||
- `log_01.csv`： 过滤后的日志
|
||||
- `log_train.csv`: 训练集
|
||||
- `log_test.csv`: 测试集
|
||||
- `log_test2.csv`: 测试集2, 用来测试识别危险用户
|
||||
- **doc**
|
||||
- `设计文档.md `
|
||||
- `审计日志调研.md`
|
||||
- `AI模块调研.md`
|
||||
|
||||
|
||||
## 2.如何在自己的数据库上进行用户画像✨
|
||||
前面所构建的用户操作模型是在模拟的教务管理系统的日志上进行开发的，模型肯定无法适用各种用户场景，但是方式是通用的。所以在这个部分将介绍如何修改SQL脚本来在你的数据库上，适用自己的审计日志构建用户画像。
|
||||
需要修改的代码都在**code2-UserPortraitModel**目录下，步骤如下：
|
||||
|
||||
### 配置审计日志
|
||||
虽然openGauss中默认打开了审计日志，但是记录日志需要额外的存储所以并不会对所有用户的操作都会记录，所以需要对审计项目进行配置，例如下面的代码打开了对表修改和表查询的审计项：
|
||||
```
|
||||
# 检查audit_dml_state配置值
|
||||
cat /var/lib/opengauss/data/postgresql.conf | grep audit_dml_state
|
||||
|
||||
# 修改配置
|
||||
gs_guc set -D /var/lib/opengauss/data -c "audit_dml_state=1"
|
||||
gs_guc set -D /var/lib/opengauss/data -c "audit_dml_state_select=1"
|
||||
|
||||
# 重启数据库使参数生效
|
||||
gs_ctl restart -D /var/lib/opengauss/data
|
||||
```
|
||||
|
||||
### 修改FeatureEngineering.sql
|
||||
**1. 收集日志**
|
||||
这里需要筛选出你需要的日志数据，需要修改的地方有日志的起止时间、感兴趣的用户
|
||||
|
||||
**2. 提取特征**
|
||||
- 代码中统计了用户每天对各种类型的操作次数作为特征，每种操作的统计都存在一个视图中，操作的类型分为：**登录、系统级、数据库级、表级**。
|
||||
|
||||
- 对于前面3类的特征应该保留，重点需要修改对**表级别的操作特征提取**，考虑自己的系统中各个**用户的操作特点**来设计表级别的操作特征。
|
||||
|
||||
- 例如在教务系统中，对学生端和教师端分别对学生表和教师表查询得多，教师端还还会对成绩表进行增改操作，教务处则是维护学生信息、教师信息。所以我在设计特征的时候，统计了这些表的查询次数、修改次数。因为这些统计信息可以将各类型的用户区别开来。
|
||||
|
||||
- 最后，在这一步设计的特征可以尽可能的多些，因为在后面训练模型时还可以选择哪些特征要输入到模型中
|
||||
|
||||
**3. 汇总特征**
|
||||
这个部分将上一步所有的特征视图汇总到一张表中，这个表基本就是我们用来训练模型的数据集了
|
||||
|
||||
### 修改UserPortraitModel.sql
|
||||
**1. 整理数据**
|
||||
为了能让数据适用openGauss中需要的模式需要对数据进行一些修改：给每条数据标号、将特征值转换成双精度浮点型
|
||||
|
||||
**2. 创建模型**
|
||||
这个部分用到的是使用logistic_regression作为基础分类器的多分类模型
|
||||
这里应该修改的有：
|
||||
- 输入模型的特征，FEATURES关键字
|
||||
- 模型的[超参数](https://docs.opengauss.org/zh/docs/5.0.0/docs/AIFeatureGuide/%E5%8E%9F%E7%94%9FDB4AI%E5%BC%95%E6%93%8E.html#:~:text=GD%3A-,logistic_regression,-%E3%80%81linear_regression%E3%80%81svm_classification%E3%80%81pca)，WITH关键字部分
|
||||
|
||||
**3. 模型预测**
|
||||
使用上一步训练好的模型进行预测，注意这里用于预测的特征应该与前面用于训练的特征要一致
|
||||
|
||||
**4. 模型评估**
|
||||
计算模型分类的准确率作为评估模型的指标，我们如果准确率不高，应该回到第二步构建模型的时候调整参数，或是输入模型的特征数
|
||||
|
||||
**5. 模型应用**
|
||||
- 用户画像
|
||||
设置用户画像标签
|
||||
设置赋予标签的阈值，例如在代码中设置的是下四分位值作为阈值
|
||||
- 识别危险用户
|
||||
和模型预测这部分类似，需要修改模型接收的特征
|
||||
|
||||
|
||||
|
||||
|
|
@ -0,0 +1,323 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import psycopg2 as pg\n",
|
||||
"import random"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# 获取DB连接对象\n",
|
||||
"4个用户 \\\n",
|
||||
" pm: create 用户 数据库 \\\n",
|
||||
" t_coder: 老师端, select 教师表, insert\\update 成绩表 \\\n",
|
||||
" s_coder: 学生端, select 学生表 课程表 成绩表 \\ \n",
|
||||
" affair: 教务处, insert 教师表 学生表 课程表 \\"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def get_conn(user='pm'):\n",
|
||||
" conn = pg.connect(database=\"db_school\", \n",
|
||||
" user=user, \n",
|
||||
" password=\"123456twG\", \n",
|
||||
" host=\"127.0.0.1\", port=\"7654\")\n",
|
||||
" return conn"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# 从CSV文件导入\n",
|
||||
"Day1, affair用户, 将student.tsv, teacher.tsv, course.tsv导入到数据库中"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def insert_tsv(conn, table_name, tsv_path, sep='\\t'):\n",
|
||||
" print(\"Opened database successfully\")\n",
|
||||
" cur = conn.cursor()\n",
|
||||
"\n",
|
||||
" copy_query = f\"COPY public.{table_name} FROM STDIN WITH (FORMAT CSV, HEADER TRUE, DELIMITER E'\\\\t', ENCODING 'UTF-8')\"\n",
|
||||
" with open(tsv_path, 'r', encoding='utf-8') as file:\n",
|
||||
" cur.copy_expert(copy_query, file)\n",
|
||||
" print(f\"Copy {tsv_path} ok\")\n",
|
||||
" # 提交事务\n",
|
||||
" conn.commit()\n",
|
||||
" # 关闭数据库连接\n",
|
||||
" cur.close()\n",
|
||||
" conn.close()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"conn = get_conn(user='affair')\n",
|
||||
"insert_tsv(conn, \"teacher\", '../data/teacher.tsv')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Opened database successfully\n",
|
||||
"Copy ../data/student.tsv ok\n",
|
||||
"Opened database successfully\n",
|
||||
"Copy ../data/course.tsv ok\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"conn2 = get_conn(user='affair')\n",
|
||||
"conn3 = get_conn(user='affair')\n",
|
||||
"insert_tsv(conn2, \"student\", '../data/student.tsv')\n",
|
||||
"insert_tsv(conn3, \"course\", '../data/course.tsv')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# 向score添加数据\n",
|
||||
"Day2, t_coder用户，向score表插入数据"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def insert_score(conn):\n",
|
||||
" student_id = range(20230001, 20230041)\n",
|
||||
" course_id = range(10, 15)\n",
|
||||
" cur = conn.cursor()\n",
|
||||
" \n",
|
||||
" insert_value = []\n",
|
||||
" for s_id in student_id:\n",
|
||||
" for c_id in course_id:\n",
|
||||
" random_score = random.randint(75, 95)\n",
|
||||
" insert_value.append((s_id, c_id, random_score))\n",
|
||||
"\n",
|
||||
" insert_sql = \"INSERT INTO public.score (student_id, course_id, score) VALUES (%s, %s, %s)\"\n",
|
||||
" cur.executemany(insert_sql, insert_value)\n",
|
||||
" conn.commit()\n",
|
||||
" cur.close()\n",
|
||||
" conn.close()\n",
|
||||
" \n",
|
||||
"\n",
|
||||
" \n",
|
||||
"# 修改成绩\n",
|
||||
"def update_score(conn, repete_times=10):\n",
|
||||
" cur = conn.cursor() \n",
|
||||
" \n",
|
||||
" update_value = []\n",
|
||||
" for i in range(repete_times):\n",
|
||||
" s_id = random.randint(20230001, 20230040)\n",
|
||||
" c_id = random.randint(10, 14)\n",
|
||||
" det_score = random.randint(-5, 0)\n",
|
||||
" update_value.append((det_score, s_id, c_id))\n",
|
||||
" \n",
|
||||
" update_sql = \"UPDATE public.score SET score = score+(%s) WHERE student_id=(%s) AND course_id=(%s) \"\n",
|
||||
" cur.executemany(update_sql, update_value)\n",
|
||||
" conn.commit()\n",
|
||||
" cur.close()\n",
|
||||
" conn.close()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"conn = get_conn(user='t_coder')\n",
|
||||
"insert_score(conn)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"conn_update_score = get_conn(user='t_coder')\n",
|
||||
"update_score(conn_update_score)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# 查询teacher,student,score\n",
|
||||
"Day1-5: t_coder查询teacher表，s_coder查询student表、course表 \\\n",
|
||||
"Day2: teacher查询score, 学生查询score"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# 对表{table_name}指定{query_id}查询{repete_times}次\n",
|
||||
"# 查询student teacher course都是调用这个函数\n",
|
||||
"def select(conn, table_name, repete_times, query_id, id_range):\n",
|
||||
" cur = conn.cursor() \n",
|
||||
" query_value = [] \n",
|
||||
" for t in range(repete_times):\n",
|
||||
" query_value.append((random.randint(id_range[0], id_range[1]),))\n",
|
||||
" query_sql = f\"SELECT * FROM public.{table_name} WHERE {query_id}=(%s)\"\n",
|
||||
" \n",
|
||||
" cur.executemany(query_sql, query_value)\n",
|
||||
" conn.commit()\n",
|
||||
" cur.close()\n",
|
||||
" conn.close()\n",
|
||||
" \n",
|
||||
"\n",
|
||||
"# 学生查询课表\n",
|
||||
"def select_course(conn, repete_times):\n",
|
||||
" cur = conn.cursor() \n",
|
||||
" query_sql = \"SELECT course_id, cname, name, point FROM public.course c, public.teacher t where c.teacher_id = t.teacher_id\" \n",
|
||||
" \n",
|
||||
" for i in range(repete_times):\n",
|
||||
" cur.execute(query_sql)\n",
|
||||
" result = cur.fetchall()\n",
|
||||
" conn.commit()\n",
|
||||
" cur.close()\n",
|
||||
" conn.close()\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# 学生查询成绩\n",
|
||||
"def select_score(conn):\n",
|
||||
" cur = conn.cursor() \n",
|
||||
" query_sql = \"\"\"SELECT s.student_id, s.\"name\", c.cname, s2.score \n",
|
||||
" FROM public.score s2, public.course c, public.student s \n",
|
||||
" WHERE s2.course_id = c.course_id AND s2.student_id = s.student_id AND s.student_id=(%s)\"\"\" \n",
|
||||
" query_value = [(i,) for i in range(20230001, 20230041)]\n",
|
||||
" \n",
|
||||
" cur.executemany(query_sql, query_value) \n",
|
||||
" conn.commit()\n",
|
||||
" cur.close()\n",
|
||||
" conn.close() \n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# 老师查询指定课程的所有成绩\n",
|
||||
"def select_score_teacher(conn):\n",
|
||||
" cur = conn.cursor() \n",
|
||||
" query_value = [(i,) for i in range(10, 15)]\n",
|
||||
" query_sql = \"SELECT * FROM public.score WHERE course_id=(%s) ORDER BY score desc;\"\n",
|
||||
" \n",
|
||||
" cur.executemany(query_sql, query_value)\n",
|
||||
" conn.commit()\n",
|
||||
" cur.close()\n",
|
||||
" conn.close()\n",
|
||||
" "
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"执行查询学生信息、老师信息、课程信息脚本"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"conn_s = get_conn(user='s_coder')\n",
|
||||
"conn_s2 = get_conn(user='s_coder')\n",
|
||||
"conn_t = get_conn(user='t_coder')\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"s_repete = random.randint(70,100)\n",
|
||||
"s2_repete = s_repete + random.randint(5,10)\n",
|
||||
"t_repete = random.randint(10,30)\n",
|
||||
"\n",
|
||||
"# s_coder查询学生信息\n",
|
||||
"select(conn_s, \"student\", s_repete, \"student_id\", (20230001, 20230040))\n",
|
||||
"# s_coder查询课程信息\n",
|
||||
"select_course(conn_s2, s2_repete)\n",
|
||||
"# t_coder查询教师信息\n",
|
||||
"select(conn_t, \"teacher\", t_repete, \"teacher_id\", (101, 105))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"查询成绩"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"conn_score = get_conn(user='s_coder')\n",
|
||||
"conn_score_t = get_conn(user='t_coder')\n",
|
||||
"\n",
|
||||
"# s_coder查询成绩\n",
|
||||
"select_score(conn_score)\n",
|
||||
"# t_coder按照课程号查询成绩\n",
|
||||
"select_score_teacher(conn_score_t)"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "opengauss",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.16"
|
||||
},
|
||||
"orig_nbformat": 4
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,6 @@
|
|||
course_id teacher_id cname point
|
||||
10 101 C语言程序设计 2
|
||||
11 102 数据结构 3
|
||||
12 103 数据库 3
|
||||
13 104 操作系统 3
|
||||
14 105 计算机网络 2
|
||||
|
Binary file not shown.
|
After Width: | Height: | Size: 116 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 386 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 140 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 201 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 128 KiB |
File diff suppressed because it is too large
Load Diff
Binary file not shown.
|
After Width: | Height: | Size: 397 KiB |
|
|
@ -0,0 +1,41 @@
|
|||
student_id name class
|
||||
20230001 张伟 2301
|
||||
20230002 王芳 2301
|
||||
20230003 李娜 2301
|
||||
20230004 张磊 2301
|
||||
20230005 王伟 2301
|
||||
20230006 李秀英 2301
|
||||
20230007 王敏 2301
|
||||
20230008 李强 2301
|
||||
20230009 张静 2301
|
||||
20230010 王勇 2301
|
||||
20230011 李霞 2301
|
||||
20230012 张涛 2301
|
||||
20230013 王静 2301
|
||||
20230014 李军 2301
|
||||
20230015 张敏 2301
|
||||
20230016 王磊 2301
|
||||
20230017 李丽 2301
|
||||
20230018 张洋 2301
|
||||
20230019 王艳 2301
|
||||
20230020 李明 2301
|
||||
20230021 张梅 2302
|
||||
20230022 王强 2302
|
||||
20230023 李红 2302
|
||||
20230024 张宇 2302
|
||||
20230025 王欣 2302
|
||||
20230026 李杰 2302
|
||||
20230027 张婷 2302
|
||||
20230028 王辉 2302
|
||||
20230029 李静 2302
|
||||
20230030 张刚 2302
|
||||
20230031 王娜 2302
|
||||
20230032 李峰 2302
|
||||
20230033 张萍 2302
|
||||
20230034 王超 2302
|
||||
20230035 李莉 2302
|
||||
20230036 张鹏 2302
|
||||
20230037 王桂芳 2302
|
||||
20230038 李飞 2302
|
||||
20230039 张霞 2302
|
||||
20230040 王建国 2302
|
||||
|
|
|
@ -0,0 +1,6 @@
|
|||
teacher_id name
|
||||
101 张老师
|
||||
102 王老师
|
||||
103 李老师
|
||||
104 赵老师
|
||||
105 刘老师
|
||||
|
|
|
@ -0,0 +1,105 @@
|
|||
-- 在本脚本中将实现从openGauss中收集审计日志，并从审计日志中提取用户操作特征
|
||||
-- Tang Wuguo, 2023-7-24, tangwg@csu.edu.cn
|
||||
|
||||
|
||||
-- 1. 收集日志
|
||||
CREATE SCHEMA pm;
|
||||
|
||||
SELECT * INTO pm.audit_log
|
||||
FROM pg_query_audit('2023-07-05 02:01:53','2023-07-09 11:00:00');
|
||||
|
||||
CREATE OR REPLACE VIEW pm.log_01 as
|
||||
select to_char(al."time", 'YYYY-MM-DD') as date,
|
||||
al."type", al."result" , al."userid" ,
|
||||
al."username" , al."database", al."client_conninfo" ,
|
||||
al."object_name", al."detail_info"
|
||||
FROM pm.audit_log al
|
||||
where username is not null and username not in ('opengauss','[unknown]');
|
||||
|
||||
|
||||
-- 2. 提取各类特征
|
||||
-- Count user login times per day
|
||||
CREATE OR REPLACE VIEW pm.query_login as select "date", l.username, coalesce(count(*),0) as "login_count"
|
||||
from pm.log_01 l
|
||||
where l."type" in ('login_success')
|
||||
group by "date", l.username
|
||||
order by "date", "login_count" desc;
|
||||
|
||||
-- Count sys opetate
|
||||
CREATE OR REPLACE VIEW pm.quey_sys as select l."date", l.username, coalesce(count(*),0) as "sys_count"
|
||||
from pm.log_01 l
|
||||
where l."type" in ('set_parameter', 'internal_event', 'grant_role', 'ddl_user')
|
||||
group by l."date", l.username
|
||||
order by l."date", "sys_count" desc;
|
||||
|
||||
-- Count user db operate times per day
|
||||
CREATE OR REPLACE VIEW pm.query_db as select l."date", l.username, coalesce(count(*),0) as "db_count"
|
||||
from pm.log_01 l
|
||||
where l."type" in ('ddl_database', 'ddl_schema', 'ddl_table')
|
||||
group by l."date", l.username
|
||||
order by l."date", "db_count" desc;
|
||||
|
||||
-- Tabel insert
|
||||
CREATE OR REPLACE VIEW pm.query_insert_all as select l."date", l.username, coalesce(count(*),0) as "table_count"
|
||||
from pm.log_01 l
|
||||
where l."type" in ('dml_action', 'copy_from')
|
||||
group by l."date", l.username
|
||||
order by l."date", "table_count" desc;
|
||||
-- Score insert
|
||||
CREATE OR REPLACE VIEW pm.query_insert_score as select l."date", l.username, coalesce(count(*),0) as "table_count"
|
||||
from pm.log_01 l
|
||||
where l."type" in ('dml_action', 'copy_from') and object_name in ('score')
|
||||
group by l."date", l.username
|
||||
order by l."date", "table_count" desc;
|
||||
-- Student insert
|
||||
CREATE OR REPLACE VIEW pm.query_insert_student as select l."date", l.username, coalesce(count(*),0) as "table_count"
|
||||
from pm.log_01 l
|
||||
where l."type" in ('dml_action', 'copy_from') and object_name in ('student')
|
||||
group by l."date", l.username
|
||||
order by l."date", "table_count" desc;
|
||||
-- Teacher insert
|
||||
CREATE OR REPLACE VIEW pm.query_insert_teacher as select l."date", l.username, coalesce(count(*),0) as "table_count"
|
||||
from pm.log_01 l
|
||||
where l."type" in ('dml_action', 'copy_from') and object_name in ('teacher')
|
||||
group by l."date", l.username
|
||||
order by l."date", "table_count" desc;
|
||||
|
||||
-- Table select
|
||||
CREATE OR REPLACE VIEW pm.query_sel_info as select l."date", l.username, coalesce(count(*),0) as "tab_sel_uinfo_count"
|
||||
from pm.log_01 l
|
||||
where l."type" in ('dml_action_select') and l.object_name in ('student', 'teacher', 'course')
|
||||
group by l."date", l.username
|
||||
order by l."date", "tab_sel_uinfo_count" desc;
|
||||
-- Score select
|
||||
CREATE OR REPLACE VIEW pm.query_sel_score as select l."date", l.username, coalesce(count(*),0) as "tab_sel_score_count"
|
||||
from pm.log_01 l
|
||||
where l."type" in ('dml_action_select') and l.object_name in ('score')
|
||||
group by l."date", l.username
|
||||
order by l."date", "tab_sel_score_count" desc;
|
||||
|
||||
|
||||
|
||||
-- 3. 汇总所有特征
|
||||
-- Merge All Sub-view
|
||||
CREATE OR REPLACE VIEW pm.log_refined as
|
||||
select t1.date, t1.username,
|
||||
COALESCE(t1.login_count, 0) as login,
|
||||
COALESCE(t2.sys_count, 0) as sys,
|
||||
COALESCE(t3.db_count, 0) as db,
|
||||
COALESCE(t4.table_count, 0) as insert_all,
|
||||
COALESCE(t5.table_count, 0) as insert_score,
|
||||
COALESCE(t6.table_count, 0) as insert_student,
|
||||
COALESCE(t7.table_count, 0) as insert_teacher,
|
||||
COALESCE(t8.tab_sel_uinfo_count, 0) as sel_info,
|
||||
COALESCE(t9.tab_sel_score_count, 0) as sel_score
|
||||
FROM pm.query_login t1
|
||||
LEFT JOIN pm.query_sys t2 ON t1."date" = t2."date" AND t1.username = t2.username
|
||||
LEFT JOIN pm.query_db t3 ON t1."date" = t3."date" AND t1.username = t3.username
|
||||
LEFT JOIN pm.query_insert_all t4 ON t1."date" = t4."date" AND t1.username = t4.username
|
||||
LEFT JOIN pm.query_insert_score t5 ON t1."date" = t5."date" AND t1.username = t5.username
|
||||
LEFT JOIN pm.query_insert_student t6 ON t1."date" = t6."date" AND t1.username = t6.username
|
||||
LEFT JOIN pm.query_insert_teacher t7 ON t1."date" = t7."date" AND t1.username = t7.username
|
||||
LEFT JOIN pm.query_sel_info t8 ON t1."date" = t8."date" AND t1.username = t8.username
|
||||
LEFT JOIN pm.query_sel_score t9 ON t1."date" = t9."date" AND t1.username = t9.username
|
||||
ORDER BY t1.date, t1.username;
|
||||
|
||||
|
|
@ -0,0 +1,107 @@
|
|||
-- 在这里我们将使用openGauss中的DB4AI模块对用户的画像
|
||||
-- 需要注意只有企业版的openGauss才具有AI特性，如果是轻量版请安装最新的企业版软件
|
||||
|
||||
|
||||
-- 1. 整理数据
|
||||
-- 为了适应模型的输入要求，需要对原始数据进行进一步修改
|
||||
CREATE OR REPLACE VIEW pm.log_train as
|
||||
SELECT row_number() OVER () AS id, "date", username,
|
||||
dense_rank() OVER (ORDER BY username) AS name_id,
|
||||
login::double precision, sys::double precision, db::double precision,
|
||||
insert_all::double precision, insert_score::double precision, insert_student::double precision, insert_teacher::double precision,
|
||||
sel_info::double precision, sel_score::double precision
|
||||
FROM log_refined
|
||||
ORDER BY log_refined."date", dense_rank() OVER (ORDER BY log_refined.username);
|
||||
|
||||
|
||||
-- 2. 创建模型
|
||||
-- 模型m0，没有使用insert_student, insert_teacher特征
|
||||
CREATE MODEL log_m0
|
||||
USING multiclass
|
||||
FEATURES login, sys, db, insert_all, insert_score, sel_info, sel_score
|
||||
TARGET name_id
|
||||
FROM log_train
|
||||
WITH classifier="logistic_regression";
|
||||
-- 模型m1
|
||||
CREATE MODEL log_m1
|
||||
USING multiclass
|
||||
FEATURES login, sys, db, insert_score, insert_student, insert_teacher, sel_info, sel_score
|
||||
TARGET name_id
|
||||
FROM log_train
|
||||
WITH classifier="logistic_regression", batch_size=5;
|
||||
|
||||
-- 查看数据库中所有模型
|
||||
select modelname, createtime, processedtuples,iterations,modeltype, outputtype
|
||||
from gs_model_warehouse limit 20;
|
||||
|
||||
-- 检查模型log_m1的详细参数信息
|
||||
SELECT gs_explain_model('log_m1');
|
||||
|
||||
|
||||
-- 3. 模型预测
|
||||
SELECT id,
|
||||
PREDICT BY log_m1
|
||||
(FEATURES login, sys, db, insert_score, insert_student, insert_teacher, sel_info, sel_score)
|
||||
as "PREDICT",
|
||||
name_id as "LABEL"
|
||||
INTO temp_pred_train
|
||||
FROM log_train;
|
||||
|
||||
SELECT id,
|
||||
PREDICT BY log_m1
|
||||
(FEATURES login, sys, db, insert_score, insert_student, insert_teacher, sel_info, sel_score)
|
||||
as "PREDICT",
|
||||
name_id as "LABEL"
|
||||
INTO temp_pred_test
|
||||
FROM log_test;
|
||||
|
||||
|
||||
-- 4. 模型评估
|
||||
-- 计算分类结果的准确率
|
||||
SELECT
|
||||
COUNT(*) AS total_count,
|
||||
SUM(CASE WHEN "PREDICT"= "LABEL" THEN 1 ELSE 0 END) AS correct_count,
|
||||
CASE
|
||||
WHEN COUNT(*) = 0 THEN 0.0
|
||||
ELSE (SUM(CASE WHEN "PREDICT" = "LABEL" THEN 1 ELSE 0 END)::FLOAT / COUNT(*)) * 100.0
|
||||
END AS accuracy
|
||||
FROM temp_pred_train;
|
||||
|
||||
|
||||
-- 5. 模型的应用
|
||||
-- 5.1 用户画像
|
||||
-- 选择各特征下四分位值作为比较的阈值
|
||||
CREATE VIEW pm.log_25 AS
|
||||
SELECT
|
||||
CASE WHEN PERCENTILE_CONT(0.25) WITHIN GROUP (ORDER BY login) = 0 THEN 1 ELSE PERCENTILE_CONT(0.25) WITHIN GROUP (ORDER BY login) END AS login_25,
|
||||
CASE WHEN PERCENTILE_CONT(0.25) WITHIN GROUP (ORDER BY sys) = 0 THEN 1 ELSE PERCENTILE_CONT(0.25) WITHIN GROUP (ORDER BY sys) END AS sys_25,
|
||||
CASE WHEN PERCENTILE_CONT(0.25) WITHIN GROUP (ORDER BY db) = 0 THEN 1 ELSE PERCENTILE_CONT(0.25) WITHIN GROUP (ORDER BY db) END AS db_25,
|
||||
CASE WHEN PERCENTILE_CONT(0.25) WITHIN GROUP (ORDER BY insert_all) = 0 THEN 1 ELSE PERCENTILE_CONT(0.25) WITHIN GROUP (ORDER BY insert_all) END AS insert_all_25,
|
||||
CASE WHEN PERCENTILE_CONT(0.25) WITHIN GROUP (ORDER BY insert_score) = 0 THEN 1 ELSE PERCENTILE_CONT(0.25) WITHIN GROUP (ORDER BY insert_score) END AS insert_score_25,
|
||||
CASE WHEN PERCENTILE_CONT(0.25) WITHIN GROUP (ORDER BY sel_info) = 0 THEN 1 ELSE PERCENTILE_CONT(0.25) WITHIN GROUP (ORDER BY sel_info) END AS sel_info_25,
|
||||
CASE WHEN PERCENTILE_CONT(0.25) WITHIN GROUP (ORDER BY sel_score) = 0 THEN 1 ELSE PERCENTILE_CONT(0.25) WITHIN GROUP (ORDER BY sel_score) END AS sel_score_25
|
||||
FROM
|
||||
pm.log_refined;
|
||||
|
||||
-- 比较各个样本，得用户标签
|
||||
SELECT
|
||||
date,
|
||||
username,
|
||||
array_remove(ARRAY[
|
||||
CASE WHEN login > login_25 THEN '活跃' ELSE NULL END,
|
||||
CASE WHEN sys > sys_25 THEN '系统维护员' ELSE NULL END,
|
||||
CASE WHEN db > db_25 THEN '数据库维护员' ELSE NULL END,
|
||||
CASE WHEN insert_all > insert_all_25 THEN '维护数据表' ELSE NULL END,
|
||||
CASE WHEN insert_score > insert_score_25 THEN '维护成绩表' ELSE NULL END,
|
||||
CASE WHEN sel_info > sel_info_25 THEN '查询多' ELSE NULL END,
|
||||
CASE WHEN sel_score > sel_score_25 THEN '关注成绩' ELSE NULL END
|
||||
], NULL) AS labels
|
||||
FROM
|
||||
log_refined, (SELECT * FROM log_25 LIMIT 1) AS quartiles;
|
||||
|
||||
-- 5.2 识别危险用户
|
||||
SELECT id, username, date,
|
||||
PREDICT BY log_m1 (FEATURES login, sys, db, insert_score, insert_student, insert_teacher, sel_info, sel_score)
|
||||
as "PREDICT",
|
||||
name_id as "LABEL"
|
||||
FROM log_test2 where "PREDICT"!="LABEL";
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,10 @@
|
|||
id,name_id,date,username,login,sys,db,insert_all,insert_score,sel_info,sel_score,insert_student,insert_teacher
|
||||
1,2,2023-07-10,pm,7,36,0,0,0,0,0,0,0
|
||||
2,3,2023-07-10,s_coder,2,0,0,0,0,199,0,0,0
|
||||
3,4,2023-07-10,t_coder,1,0,0,0,0,26,0,0,0
|
||||
4,2,2023-07-11,pm,9,16,0,0,0,0,0,0,0
|
||||
5,3,2023-07-11,s_coder,2,0,0,0,0,211,0,0,0
|
||||
6,4,2023-07-11,t_coder,1,0,0,0,0,18,0,0,0
|
||||
7,2,2023-07-12,pm,6,29,0,0,0,0,0,0,0
|
||||
8,3,2023-07-12,s_coder,2,0,0,0,0,191,0,0,0
|
||||
9,4,2023-07-12,t_coder,1,0,0,0,0,32,0,0,0
|
||||
|
|
|
@ -0,0 +1,7 @@
|
|||
id,name_id,date,username,login,sys,db,insert_all,insert_score,insert_student,insert_teacher,sel_info,sel_score
|
||||
1,2,2023-07-13,pm,7,36,0,0,0,0,0,0,0
|
||||
2,3,2023-07-13,s_coder,2,0,0,0,0,0,0,199,0
|
||||
3,4,2023-07-13,t_coder,1,0,0,0,0,0,0,26,0
|
||||
4,2,2023-07-14,pm,9,16,0,3,0,0,0,0,0
|
||||
5,3,2023-07-14,s_coder,2,0,0,0,2,0,0,211,0
|
||||
6,4,2023-07-14,t_coder,1,0,0,0,0,0,0,18,0
|
||||
|
|
|
@ -0,0 +1,21 @@
|
|||
id,date,username,name_id,login,sys,db,insert_all,insert_score,insert_student,insert_teacher,sel_info,sel_score
|
||||
1,2023-07-05,affair,1,3,0,0,3,0,1,1,5,0
|
||||
2,2023-07-05,pm,2,8,34,8,0,0,0,0,1,0
|
||||
3,2023-07-05,s_coder,3,2,0,0,0,0,0,0,152,0
|
||||
4,2023-07-05,t_coder,4,1,0,0,0,0,0,0,18,0
|
||||
5,2023-07-06,pm,2,7,30,0,0,0,0,0,0,4
|
||||
6,2023-07-06,s_coder,3,3,0,0,0,0,0,0,145,40
|
||||
7,2023-07-06,t_coder,4,4,0,0,210,210,0,0,413,5
|
||||
8,2023-07-07,pm,2,6,29,0,0,0,0,0,0,0
|
||||
9,2023-07-07,s_coder,3,2,0,0,0,0,0,0,203,0
|
||||
10,2023-07-07,t_coder,4,1,0,0,0,0,0,0,28,0
|
||||
11,2023-07-08,pm,2,10,43,0,0,0,0,0,0,0
|
||||
12,2023-07-08,s_coder,3,2,0,0,0,0,0,0,195,0
|
||||
13,2023-07-08,t_coder,4,1,0,0,0,0,0,0,21,0
|
||||
14,2023-07-09,pm,2,5,22,0,0,0,0,0,0,0
|
||||
15,2023-07-09,s_coder,3,2,0,0,0,0,0,0,186,0
|
||||
16,2023-07-09,t_coder,4,1,0,0,0,0,0,0,24,0
|
||||
17,2023-07-06,affair,1,1,0,0,1,0,1,0,0,0
|
||||
18,2023-07-07,affair,1,1,0,0,2,0,1,1,2,0
|
||||
19,2023-07-08,affair,1,2,0,0,2,0,0,2,0,0
|
||||
20,2023-07-09,affair,1,3,0,0,3,0,0,1,2,0
|
||||
|
|
|
@ -0,0 +1,160 @@
|
|||
# openGauss中的AI模块调研
|
||||
*Tang Wuguo, tangwg@csu.edu.cn*
|
||||
|
||||
这篇博客中对openGauss中的AI模块进行介绍，并且以iris鸢尾花数据集为例介绍AI模块的使用.
|
||||
|
||||
openGauss将AI与数据库结合，其中的AI特性大致可分为AI4DB和DB4AI两个部分:
|
||||
|
||||
- **AI4DB**就是指用人工智能技术优化数据库的性能，从而获得更好地执行表现；也可以通过人工智能的手段实现自治、免运维等。主要包括自调优、自诊断、自安全、自运维、自愈等子领域；
|
||||
- **DB4AI**就是指打通数据库到人工智能应用的端到端流程，通过数据库来驱动AI任务，统一人工智能技术栈，达到开箱即用、高性能、节约成本等目的。例如通过SQL-like语句实现推荐系统、图像检索、时序预测等功能，充分发挥数据库的高并行、列存储等优势，既可以避免数据和碎片化存储的代价，又可以避免因信息泄漏造成的安全风险；
|
||||
|
||||
## DB4AI概要
|
||||
在我们的用户操作模型的构建中，主要使用DB4AI模块，借助其中的类似SQL语句的形式，直接在openGauss数据库中对数据进行建模、训练及预测。
|
||||
|
||||
|
||||
|
||||
|
||||
## 鸢尾花分类模型实践
|
||||
下面我们已鸢尾花数据集为例，使用openGauss中的DB4AI模块搭建两种模型：二分类模型，多分类模型。同时还利用内置的函数`_gs_explain_model_`查看模型的详细信息，`_PREDICT BY_`关键字来进行模型的推理。
|
||||
### Iris数据集
|
||||
iris数据集是ML中最经典的数据集之一，其中共包括150个样本，对于每个样本有花萼长度、花萼宽度、花瓣长度、花瓣宽度4个特征，我们需要依据这4个特征区分3种花型：山鸢尾、变色鸢尾还是维吉尼亚鸢尾。
|
||||
|
||||
### 划分数据集
|
||||
我们对iris数据集随机打乱，选择80%的数据作为训练集，剩下的作为测试集。
|
||||
训练集用于模型的训练，模型训练好后可以分别在计算训练集和测试集上的准确率。
|
||||
```sql
|
||||
CREATE VIEW iris_random AS
|
||||
SELECT *,
|
||||
ROW_NUMBER() OVER () AS row_num,
|
||||
COUNT(*) OVER () AS total_rows
|
||||
FROM iris_1
|
||||
ORDER BY RANDOM();
|
||||
|
||||
CREATE VIEW iris_train AS
|
||||
SELECT *
|
||||
FROM iris_random
|
||||
WHERE row_num <= total_rows * 0.8;
|
||||
|
||||
CREATE VIEW iris_test AS
|
||||
SELECT *
|
||||
FROM iris_random
|
||||
WHERE row_num > total_rows * 0.8;
|
||||
|
||||
```
|
||||
|
||||
### 二分类模型
|
||||
假定我们现在的目标是要区分是否为山鸢尾，yes or no，这是一个二分类问题，这里我们选择使用逻辑回归来解决。
|
||||
|
||||
**CREATE MODEL**
|
||||
```sql
|
||||
CREATE MODEL iris_m1 USING logistic_regression
|
||||
FEATURES sepal_length, sepal_width,petal_length,petal_width
|
||||
TARGET target_id < 2
|
||||
FROM iris_train
|
||||
WITH batch_size=20;
|
||||
```
|
||||
执行上面的脚本后正常会输出：`MODEL CREATED. PROCESSED 1`
|
||||
表示模型构建好了，下面测试模型的预测效果
|
||||
通过查询`gs_model_warehouse`表可以看到数据库中的所有模型
|
||||
|
||||
通过使用`gs_explain_model`函数可以查看指定模型的详细参数：
|
||||
`select gs_explain_model('iris_m1');`
|
||||
|
||||
|
||||
|
||||
**PREDICT BY**
|
||||
```sql
|
||||
SELECT id,
|
||||
PREDICT BY iris_m1 (FEATURES sepal_length,sepal_width,petal_length,petal_width) as "PREDICT",
|
||||
target_id < 2 as "LABEL"
|
||||
FROM iris_train limit 20;
|
||||
```
|
||||
```sql
|
||||
SELECT id,
|
||||
PREDICT BY iris_m1 (FEATURES sepal_length,sepal_width,petal_length,petal_width) as "PREDICT",
|
||||
target_id < 2 as "LABEL"
|
||||
FROM iris_test;
|
||||
```
|
||||
|
||||
|
||||
|
||||
**计算分类准确率**
|
||||
准备率 = 分类正确的数量 / 样本总数
|
||||
```sql
|
||||
SELECT id,
|
||||
PREDICT BY iris_m1 (FEATURES sepal_length,sepal_width,petal_length,petal_width) as "PREDICT",
|
||||
target_id < 2 as "LABEL"
|
||||
INTO temp_pred
|
||||
FROM iris_train limit 20;
|
||||
|
||||
SELECT
|
||||
COUNT(*) AS total_count,
|
||||
SUM(CASE WHEN "PREDICT"= "LABEL" THEN 1 ELSE 0 END) AS correct_count,
|
||||
CASE
|
||||
WHEN COUNT(*) = 0 THEN 0.0
|
||||
ELSE (SUM(CASE WHEN "PREDICT" = "LABEL" THEN 1 ELSE 0 END)::FLOAT / COUNT(*)) * 100.0
|
||||
END AS accuracy
|
||||
FROM temp_pred;
|
||||
```
|
||||
|
||||
|
||||
### 多分类模型
|
||||
假定现在的任务是给定一个样本要预测是3种花型中的哪一种，这是一个多分类的问题。
|
||||
```sql
|
||||
CREATE MODEL iris_m2 USING multiclass
|
||||
FEATURES sepal_length, sepal_width,petal_length,petal_width
|
||||
TARGET target_id
|
||||
FROM iris_train
|
||||
WITH classifier="logistic_regression", batch_size=20,max_iterations=300,learning_rate = 1.0;
|
||||
|
||||
drop table temp_pred;
|
||||
|
||||
SELECT id,
|
||||
PREDICT BY iris_m2 (FEATURES sepal_length,sepal_width,petal_length,petal_width) as "PREDICT",
|
||||
target_id as "LABEL"
|
||||
INTO temp_pred
|
||||
FROM iris_test;
|
||||
|
||||
SELECT
|
||||
COUNT(*) AS total_count,
|
||||
SUM(CASE WHEN "PREDICT"= "LABEL" THEN 1 ELSE 0 END) AS correct_count,
|
||||
CASE
|
||||
WHEN COUNT(*) = 0 THEN 0.0
|
||||
ELSE (SUM(CASE WHEN "PREDICT" = "LABEL" THEN 1 ELSE 0 END)::FLOAT / COUNT(*)) * 100.0
|
||||
END AS accuracy
|
||||
FROM temp_pred;
|
||||
```
|
||||
batch_size=20
|
||||
|
||||
|
||||
|
||||
经过调节超参数batch_size=4，在训练集上的结果可以提升
|
||||
|
||||
|
||||
|
||||
## 小结
|
||||
最后总结下使用openGauss中AI模块的流程
|
||||
|
||||
1. 处理数据集
|
||||
1. 数据集中的每个样本通常包括多个属性列和标签列，建议添加一个id列用于标识每个样本
|
||||
2. 划分数据集，在ML中需要在训练集上训练模型，在测试集上评估
|
||||
3. 明确属于是哪种任务类型（分类or回归），才能选择后面相应的算法
|
||||
2. 创建模型
|
||||
1. `CREATE MODEL`关键字创建模型
|
||||
2. `DROP MODEL xxx` 删除模型
|
||||
3. `SELECT gs_explain_model('xxx')` 查看模型详细信息
|
||||
4. `SELECT modelname, createtime, processedtuples,iterations,modeltype, outputtype FROM gs_model_warehouse LIMIT 5;`查看数据库中所有模型
|
||||
3. 模型推理
|
||||
1. 通产推理结果应该包含样本的id, 预测值PREDICT, 标签值LABEL
|
||||
2. 使用`PREDICT BY`关键字进行模型的推理，得到PREDICT列
|
||||
4. 模型评估
|
||||
1. 可以通过SQL语句计算ACC准确率（for 分类）、MSE均方误差（for 回归）等指标
|
||||
2. 也可以将预测的结果导出到CSV文件使用其他工具来分析，`COPY(...)TO '/path' WITH(FORMAT CSV, HEADER)`
|
||||
5. 模型调参
|
||||
1. 通过观察模型在训练集上的结果，我们可以通过调整不同的超参数来创建不同的模型
|
||||
2. 不同的模型有这不同的可调参数，可以参考文档中列出的[这张表](https://docs.opengauss.org/zh/docs/5.0.0/docs/AIFeatureGuide/DB4AI-Query-%E6%A8%A1%E5%9E%8B%E8%AE%AD%E7%BB%83%E5%92%8C%E6%8E%A8%E6%96%AD.html#:~:text=%E8%A1%A8%203-,%E8%B6%85%E5%8F%82%E7%9A%84%E9%BB%98%E8%AE%A4%E5%80%BC%E4%BB%A5%E5%8F%8A%E5%8F%96%E5%80%BC%E8%8C%83%E5%9B%B4,-%E7%AE%97%E5%AD%90)。
|
||||
3. 当你觉的效果足够满意了，就将这组参数选定作为最终的模型
|
||||
|
||||
*参考资料*
|
||||
[openGauss Doc-Ai特性](https://docs.opengauss.org/zh/docs/5.0.0/docs/AIFeatureGuide/AI%E7%89%B9%E6%80%A7.html)
|
||||
[openGauss Doc-原生Db4Ai引擎](https://docs.opengauss.org/zh/docs/5.0.0/docs/AIFeatureGuide/%E5%8E%9F%E7%94%9FDB4AI%E5%BC%95%E6%93%8E.html)
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 44 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 72 KiB |
|
|
@ -0,0 +1,156 @@
|
|||
# 审计日志调研
|
||||
*Tang Wuguo, tangwg@csu.edu.cn*
|
||||
|
||||
因为后续工作需要对数据库用户的操作进行建模并进行用户画像，第一步就是需要获取到用户操作日志，即审计日志。在这篇博客中将介绍openGauss中的审计日志部分的内容，包括如何获取审计日志，设置审计日志，导出审计日志等内容。
|
||||
|
||||
## 1.日志位置
|
||||
查询到各种配置文件的路径：
|
||||
例如postgresql.conf文件路径：`/var/lib/opengauss/data/postgresql.conf`
|
||||
|
||||
查看审计日志的文件夹名：
|
||||
|
||||
审计日志完整路径：`/var/lib/opengauss/data/pg_audit`
|
||||
|
||||
|
||||
|
||||
## 2.比较OG和PG 中的日志管理
|
||||
**openGauss**
|
||||
- pg_log
|
||||
数据库运行日志，openGauss运行时数据库节点以及openGauss安装部署时产生的日志统称为系统日日志
|
||||
- pg_xlog
|
||||
WAL 日志，事务日志信息即重做日志
|
||||
- pg_clog
|
||||
事务提交日志，记录的是事务的元数据
|
||||
- pg_audit
|
||||
审计功能开启时会不断产生大量的审计日志，占用磁盘空间。用户可以根据磁盘空间的大小设置审计日志维护策略。
|
||||
- gs_profile
|
||||
性能日志指的是数据库系统在运行时检测物理资源的运行状态的日志，在对外部资源进行访问时的性能检测，包括磁盘等外部资源的访问检测信息。
|
||||
|
||||
**postgresql**
|
||||
- pg_log
|
||||
- pg_xlog
|
||||
- pg_clog
|
||||
|
||||
|
||||
|
||||
## 3.审计日志
|
||||
openGauss将用户对数据库的所有操作写入审计日志。数据库安全管理员可以利用这些日志信息，重现导致数据库现状的一系列事件，找出非法操作的用户、时间和内容等。
|
||||
默认时开启审计功能的[audit_enabled](https://docs.opengauss.org/zh/docs/5.0.0/docs/DatabaseReference//%E5%AE%A1%E8%AE%A1%E5%BC%80%E5%85%B3.html#zh-cn_topic_0237124745_zh-cn_topic_0059777744_sb3d1b703f24c49f096b36087a60fdfcd)，除了审计总开关，**各个审计项**也有对应的开关。只有开关开启，对应的审计功能才能生效。
|
||||
### 3.1审计相关的配置
|
||||
下面列出了所有审计配置及解释，其中就包括对各个审计项的配置：
|
||||
```bash
|
||||
audit_enabled = on
|
||||
audit_directory = 'pg_audit' # 审计文件的存储目录
|
||||
audit_data_format = 'binary' # 审计日志文件的格式，当前仅支持二进制格式（binary）
|
||||
audit_rotation_interval = 1d # 创建一个新审计日志文件的时间间隔，default 1 day
|
||||
audit_rotation_size = 10MB # 审计日志文件的最大容量, 超过此参数值时将生成一个新日志
|
||||
audit_space_limit = 1024MB # 审计文件占用磁盘空间的最大值
|
||||
audit_file_remain_threshold = 1048576 # 审计目录下审计文件的最大数量
|
||||
|
||||
# --------具体的审计项--------
|
||||
# 1, 7 表示开启，0表示关闭
|
||||
|
||||
#---用户和权限审计
|
||||
audit_login_logout = 7 # 用户登录、退出的审计
|
||||
audit_database_process = 1 # 数据库启动、停止、恢复和切换的审计
|
||||
audit_user_locked = 1 # 用户锁定和解锁
|
||||
audit_user_violation = 0 # 用户越权操作审计
|
||||
audit_grant_revoke = 1 # 用户权限授予和回收审计
|
||||
|
||||
#---各类操作审计
|
||||
# 数据库对象的CREATE，ALTER，DROP操作审计,
|
||||
# 由29个二进制位数表示，分别代表了26类数据库对象
|
||||
# 12295表示：审计DATABASE, SCHEMA, USER, DATA SOURCE
|
||||
audit_system_object = 12295
|
||||
|
||||
# 具体表的INSERT、UPDATE和DELETE操作审计, 默认关闭
|
||||
# 注意不包括SELECT
|
||||
audit_dml_state = 0
|
||||
# 是否对SELECT操作进行审计
|
||||
audit_dml_state_select = 0
|
||||
|
||||
# 在执行存储过程、匿名块或自定义函数（不包括系统自带函数）时是否记录审计信息
|
||||
audit_function_exec = 0
|
||||
# 是否对COPY操作进行审计
|
||||
audit_copy_exec = 0
|
||||
# 是否对SET操作进行审计
|
||||
audit_set_parameter = 1
|
||||
```
|
||||
### 3.2配置审计项
|
||||
这里我们在默认值的基础上，打开对表操作的审计，执行存储过程、函数的审计，copy操作的审计
|
||||
```bash
|
||||
# 12295表示：审计DATABASE, SCHEMA, USER, DATA SOURCE
|
||||
# 12303表示打开第4位的审计，即对表格创建审计
|
||||
audit_system_object = 12303
|
||||
|
||||
# 具体表的INSERT、UPDATE和DELETE操作审计
|
||||
audit_dml_state = 0
|
||||
# 是否对SELECT操作进行审计
|
||||
audit_dml_state_select = 0
|
||||
|
||||
# 在执行存储过程、匿名块或自定义函数（不包括系统自带函数）时是否记录审计信息
|
||||
audit_function_exec = 0
|
||||
# 是否对COPY操作进行审计
|
||||
audit_copy_exec = 0
|
||||
```
|
||||
```bash
|
||||
# 检查audit_dml_state配置值
|
||||
cat /var/lib/opengauss/data/postgresql.conf | grep audit_dml_state
|
||||
|
||||
# 修改配置
|
||||
gs_guc set -D /var/lib/opengauss/data -c "audit_dml_state=1"
|
||||
gs_guc set -D /var/lib/opengauss/data -c "audit_dml_state_select=1"
|
||||
gs_guc set -D /var/lib/opengauss/data -c "audit_function_exec=1"
|
||||
gs_guc set -D /var/lib/opengauss/data -c "audit_copy_exec=1"
|
||||
gs_guc set -D /var/lib/opengauss/data -c "audit_system_object=12303"
|
||||
|
||||
# 重启数据库使参数生效
|
||||
gs_ctl restart -D /var/lib/opengauss/data
|
||||
```
|
||||
我们也可以直接修改postgresql.conf文件来配置，修改后也需要重启数据库。
|
||||
### 3.3查询审计日志
|
||||
通过下面的查询语句，可以查询到指定时间段内的所有审计日志。
|
||||
```sql
|
||||
select * from pg_query_audit('2023-05-18 00:00:00','2025-05-18 08:00:00');
|
||||
```
|
||||
```sql
|
||||
time | type | result | userid | username | database | client_conninfo | object_name | detail_info | node_name | thread_id | local_port | remote_port
|
||||
------------------------+----------------+--------+--------+-----------+-----------+-------------------------+-------------------+--------------+-------------------+---------------------------------+------------+-------------
|
||||
2021-03-04 08:00:08+08 | login_success | ok | 10 | omm | postgres | gsql@::1 | postgres | login db(postgres) success, SSL=off | dn_6001_6002_6003 | 140477687527168@668131208211425 |17778 | 46946
|
||||
|
||||
```
|
||||
|
||||
### 3.4审计日志存储到表格
|
||||
在前面我们可以通过查询语句查看日志，但是仍不方便。我们可以使用下面的语句将查询到的日志信息存储到表格中，方便我们观察、过滤日志信息。
|
||||
```sql
|
||||
CREATE TABLE audit_log
|
||||
(
|
||||
log_time timestamp(3) with time zone,
|
||||
type text,
|
||||
result text,
|
||||
|
||||
userid integer,
|
||||
username text,
|
||||
database text,
|
||||
client_conninfo text,
|
||||
object_name text,
|
||||
detail_info text,
|
||||
node_name text,
|
||||
|
||||
thread_id text,
|
||||
local_port text,
|
||||
remote_port text,
|
||||
|
||||
);
|
||||
|
||||
SELECT * INTO audit_log FROM pg_query_audit('2023-05-18 00:00:00','2025-05-18 08:00:00');
|
||||
|
||||
```
|
||||
|
||||
|
||||
|
||||
*参考资料*
|
||||
[PostgreSQL日志管理-CSDN博客](https://blog.csdn.net/weixin_41287260/article/details/120471931)
|
||||
[openGauss Doc-审计概述](https://docs.opengauss.org/zh/docs/5.0.0/docs/DatabaseAdministrationGuide/%E5%AE%A1%E8%AE%A1%E6%A6%82%E8%BF%B0.html)
|
||||
[openGauss Doc-日志参考](https://docs.opengauss.org/zh/docs/5.0.0/docs/DatabaseOMGuide/%E6%97%A5%E5%BF%97%E5%8F%82%E8%80%83.html)
|
||||
|
||||
|
|
@ -0,0 +1,82 @@
|
|||
# 设计文档
|
||||
*Tang Wuguo, tangwg@csu.edu.cn*
|
||||
|
||||
首先我们明确一下将用户的画像问题看作一个多分类任务，确定好模型的输入输出。模型的输入是设计好的用户操作特征，模型输出的是该用户的类型。而要从原始的审计日志到用户特征需要经过特征工程模块；在用户操作模型中选用的是openGauss中定义好的机器学习模型。
|
||||
整体的流程如下图所示：
|
||||
|
||||
1. 首先需要收集指定时间段的审计审计日志
|
||||
2. 通过分析改日志来筛除一些无效的数据，设计一些用户特征，例如用户的一次操作需要包括主体、客体、操作类型（哪个用户对谁做了什么）等信息；
|
||||
3. 接着需要处理数据集，例如给类别数据编号，划分数据集；
|
||||
4. 使用为了评估模型分类的好坏，这里选择准确率作为指标；
|
||||
5. 根据指标的好坏，我们可以调整模型的超参数，使得模型在训练集上分类得更好；
|
||||
6. 选定好超参数后模型就可以部署上线，之后数据库中又会产生许多的用户操作日志，我们可以重新收集日志来迭代模型；
|
||||
|
||||
|
||||
|
||||
|
||||
## 1.模拟审计日志
|
||||
由于没有找到公开的审计日志可以使用，于是设计了一个多数据库用户操作的教务管理系统，并使用python脚本模拟各个用户操作数据库，从而得到审计日志。
|
||||
|
||||
模拟日志流程：
|
||||
|
||||
|
||||
### 1.1数据库db_school设计
|
||||
该数据库包括两个模式，教务管理系统中的各种数据存在public模式下；操作日志、操作模型相关的数据都存在pm模式下。
|
||||
|
||||
这个教务管理系统中由4个表组成：
|
||||
教师表（工号，姓名）
|
||||
学生表（学号，姓名，班级）
|
||||
课程表（课程号，授课老师工号，课程名，学分）
|
||||
成绩表（学号，课程号，成绩）
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
### 1.2多用户设计
|
||||
opengauss：超级管理员，创建用户pm
|
||||
- pm：项目架构师；负责设计、创建数据库、表格、视图；创建下面的用户
|
||||
- t_coder：老师端，主要对成绩表增、改
|
||||
- s_coder：学生端，成绩查询
|
||||
- affair：教务处，负责增删学生表、老师表、课程表
|
||||
|
||||
### 1.3各个用户的权限设计
|
||||
openGauss支持以下的权限：SELECT、INSERT、UPDATE、DELETE、TRUNCATE、REFERENCES、CREATE、CONNECT、EXECUTE、USAGE、ALTER、DROP、COMMENT、INDEX和VACUUM。不同的权限与不同的对象类型关联，对象所有者的权限是隐式拥有的，即只要拥有对象就可以执行对象所有者的这些隐式权限。
|
||||
|
||||
**pm**: sysadmin
|
||||
**s_coder**: SELECT ON ALL TABLES IN SCHEMA public
|
||||
**t_coder**: SELECT ON ALL TABLES IN SCHEMA public, UPDATE ON public.score
|
||||
**affair**: SELECT ON ALL TABLES IN SCHEMA public, UPDATE ON ALL TABLES IN SCHEMA public
|
||||
|
||||
|
||||
### 1.4模拟用户操作
|
||||
准备40名学生、5名老师、5门课程信息，使用python脚本模拟5天操作的数据
|
||||
具体是使用`psycopg2`库创建数据库连接对象，再使用数据库连接对象执行增删改查操作。
|
||||
|
||||
|
||||
## 2.用户操作模型
|
||||
### 2.1日志中的用户操作类型
|
||||
通过观察日志可以发现操作主要分为：
|
||||
- 系统级别的操作，如登录，创建用户，分配权限，设置参数
|
||||
- 数据库级操作，创建数据库，模式，表
|
||||
- 表级操作，对表的增删改查
|
||||
|
||||
因此，在设计操作特征的时候，也按照这些类别创建了各类操作的视图（所有视图均是统计每天的次数）：
|
||||
|
||||
- query_login，用户每天的登录次数
|
||||
- query_sys，除了登录操作，其他的系统级操作次数
|
||||
- query_db，数据库级的操作次数
|
||||
- query_insert_all，所有对表的插入操作
|
||||
- query_insert_score，所有对score表的插入操作
|
||||
- query_insert_score，所有对score表的插入操作
|
||||
- query_sel_info，各类信息表的查询
|
||||
- query_sel_score，对成绩表的查询
|
||||
|
||||
### 2.2模型应用场景
|
||||
**给某日用户的画像**
|
||||
首先通过计算训练数据中各个特征的下四分位值a_25作为我们的判定阈值。即有75%的数据是大于这个a_25，a_25排在后25名的位置；同时防止该特征出现的很少例如操作数据库的特征，绝大部分数据都会等于0此时，对于这种情况我们手动设置该阈值为1。
|
||||
|
||||
**识别危险用户操作**
|
||||
根据用户的操作行为，判断其是否行为异常。即该用户的操作特征与其实际用户标签不符。
|
||||
在测试时故意增加了一条s_coder的异常数据，例如该一条学生客户端端的数据的成绩表修改次数改为正值，但其实学生端是没法对成绩表进行增改的，看模型是否能预测出的异常行为。
|
||||
Loading…
Reference in New Issue