Remove sensitive configs from dolphinscheduler_env.sh (#11917)
This commit is contained in:
kezhenxu94
GitHub
parent
511149f644
commit
b73a9511a7
|
|
@ -339,28 +339,12 @@ The default configuration is as follows:
|
|||
### dolphinscheduler_env.sh [load environment variables configs]
|
||||
|
||||
When using shell to commit tasks, DolphinScheduler will export environment variables from `bin/env/dolphinscheduler_env.sh`. The
|
||||
mainly configuration including `JAVA_HOME`, mata database, registry center, and task configuration.
|
||||
mainly configuration including `JAVA_HOME` and other environment paths.
|
||||
|
||||
```bash
|
||||
# JAVA_HOME, will use it to start DolphinScheduler server
|
||||
export JAVA_HOME=${JAVA_HOME:-/opt/soft/java}
|
||||
|
||||
# Database related configuration, set database type, username and password
|
||||
export DATABASE=${DATABASE:-postgresql}
|
||||
export SPRING_PROFILES_ACTIVE=${DATABASE}
|
||||
export SPRING_DATASOURCE_URL
|
||||
export SPRING_DATASOURCE_USERNAME
|
||||
export SPRING_DATASOURCE_PASSWORD
|
||||
|
||||
# DolphinScheduler server related configuration
|
||||
export SPRING_CACHE_TYPE=${SPRING_CACHE_TYPE:-none}
|
||||
export SPRING_JACKSON_TIME_ZONE=${SPRING_JACKSON_TIME_ZONE:-UTC}
|
||||
export MASTER_FETCH_COMMAND_NUM=${MASTER_FETCH_COMMAND_NUM:-10}
|
||||
|
||||
# Registry center configuration, determines the type and link of the registry center
|
||||
export REGISTRY_TYPE=${REGISTRY_TYPE:-zookeeper}
|
||||
export REGISTRY_ZOOKEEPER_CONNECT_STRING=${REGISTRY_ZOOKEEPER_CONNECT_STRING:-localhost:2181}
|
||||
|
||||
# Tasks related configurations, need to change the configuration if you use the related tasks.
|
||||
export HADOOP_HOME=${HADOOP_HOME:-/opt/soft/hadoop}
|
||||
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-/opt/soft/hadoop/etc/hadoop}
|
||||
|
|
|
|||
|
|
@ -8,11 +8,12 @@ We here use MySQL as an example to illustrate how to configure an external datab
|
|||
which is `api-server/libs` and `alert-server/libs` and `master-server/libs` and `worker-server/libs`.
|
||||
|
||||
* First of all, follow the instructions in [datasource-setting](datasource-setting.md) `Pseudo-Cluster/Cluster Initialize the Database` section to create and initialize database
|
||||
* Set the following environment variables in your terminal or modify the `bin/env/dolphinscheduler_env.sh` with your database username and password for `{user}` and `{password}`:
|
||||
* Set the following environment variables in your terminal with your database address, username and password for `{address}`, `{user}` and `{password}`:
|
||||
|
||||
```shell
|
||||
export DATABASE=mysql
|
||||
export SPRING_PROFILES_ACTIVE=${DATABASE}
|
||||
export SPRING_DATASOURCE_URL="jdbc:mysql://{address}/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8&useSSL=false"
|
||||
export SPRING_DATASOURCE_USERNAME={user}
|
||||
export SPRING_DATASOURCE_PASSWORD={password}
|
||||
```
|
||||
|
|
@ -54,9 +55,9 @@ mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'%';
|
|||
mysql> CREATE USER '{user}'@'localhost' IDENTIFIED BY '{password}';
|
||||
mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'localhost';
|
||||
mysql> FLUSH PRIVILEGES;
|
||||
```
|
||||
```
|
||||
|
||||
For PostgreSQL:
|
||||
For PostgreSQL:
|
||||
```shell
|
||||
# Use psql-tools to login PostgreSQL
|
||||
psql
|
||||
|
|
@ -72,9 +73,10 @@ echo "host dolphinscheduler {user} {ip} md5" >> $PGDATA/pg_hba.conf
|
|||
pg_ctl reload
|
||||
```
|
||||
|
||||
Then, modify `./bin/env/dolphinscheduler_env.sh`, change {user} and {password} to what you set in the previous step.
|
||||
Then, set the database configurations by exporting the following environment variables, change {user} and {password} to what you set in the previous step.
|
||||
|
||||
For MySQL:
|
||||
|
||||
```shell
|
||||
# for mysql
|
||||
export DATABASE=${DATABASE:-mysql}
|
||||
|
|
@ -85,6 +87,7 @@ export SPRING_DATASOURCE_PASSWORD={password}
|
|||
```
|
||||
|
||||
For PostgreSQL:
|
||||
|
||||
```shell
|
||||
# for postgresql
|
||||
export DATABASE=${DATABASE:-postgresql}
|
||||
|
|
|
|||
|
|
@ -23,10 +23,10 @@ Stop all services of dolphinscheduler according to your deployment method. If yo
|
|||
|
||||
### Upgrade Database
|
||||
|
||||
Change configuration in `./bin/env/dolphinscheduler_env.sh` ({user} and {password} are changed to your database username and password), and then run the upgrade script.
|
||||
Set the following environment variables ({user} and {password} are changed to your database username and password), and then run the upgrade script.
|
||||
|
||||
Using MySQL as an example, change the value if you use other databases. Please manually download the [mysql-connector-java driver jar](https://downloads.MySQL.com/archives/c-j/)
|
||||
jar package and add it to the `./tools/libs` directory, then change `./bin/ env/dolphinscheduler_env.sh` file
|
||||
jar package and add it to the `./tools/libs` directory, then export the following environment variables
|
||||
|
||||
```shell
|
||||
export DATABASE=${DATABASE:-mysql}
|
||||
|
|
@ -45,7 +45,7 @@ Execute database upgrade script: `sh ./tools/bin/upgrade-schema.sh`
|
|||
- If you deploy with Pseudo-Cluster deployment, change it according to [Pseudo-Cluster](../installation/pseudo-cluster.md) section "Modify Configuration".
|
||||
- If you deploy with Cluster deployment, change it according to [Cluster](../installation/cluster.md) section "Modify Configuration".
|
||||
|
||||
And them run command `sh ./bin/start-all.sh` to start all services.
|
||||
And them run command `sh ./bin/start-all.sh` to start all services.
|
||||
|
||||
## Notice
|
||||
|
||||
|
|
@ -54,7 +54,7 @@ And them run command `sh ./bin/start-all.sh` to start all services.
|
|||
The architecture of worker group is different between version before version 1.3.1 until version 2.0.0
|
||||
|
||||
- Before version 1.3.1(include itself) worker group can be created through UI interface.
|
||||
- Since version 1.3.1 and before version 2.0.0, worker group can be created by modifying the worker configuration.
|
||||
- Since version 1.3.1 and before version 2.0.0, worker group can be created by modifying the worker configuration.
|
||||
|
||||
#### How Can I Do When I Upgrade from 1.3.1 to version before 2.0.0
|
||||
|
||||
|
|
@ -78,13 +78,13 @@ Assume bellow are the machine worker service to be deployed:
|
|||
To keep worker group config consistent with the previous version, we need to modify workers configuration as below:
|
||||
|
||||
```shell
|
||||
#worker service is deployed on which machine, and also specify which worker group this worker belongs to.
|
||||
#worker service is deployed on which machine, and also specify which worker group this worker belongs to.
|
||||
workers="ds1:service1,ds2:service2,ds3:service2"
|
||||
```
|
||||
|
||||
#### The Worker Group has Been Enhanced in Version 1.3.2
|
||||
|
||||
Workers in 1.3.1 can only belong to one worker group, but after version 1.3.2 and before version 2.0.0 worker support more than one worker group.
|
||||
Workers in 1.3.1 can only belong to one worker group, but after version 1.3.2 and before version 2.0.0 worker support more than one worker group.
|
||||
|
||||
```sh
|
||||
workers="ds1:service1,ds1:service2"
|
||||
|
|
|
|||
|
|
@ -147,7 +147,7 @@ export DOLPHINSCHEDULER_OPTS="
|
|||
|spring.datasource.hikari.leak-detection-threshold|0|连接泄露检测阈值|
|
||||
|spring.datasource.hikari.initialization-fail-timeout|1|连接池初始化失败timeout|
|
||||
|
||||
DolphinScheduler同样可以通过`bin/env/dolphinscheduler_env.sh`进行数据库连接相关的配置。
|
||||
DolphinScheduler同样可以通过设置环境变量进行数据库连接相关的配置, 将以上小写字母转成大写并把`.`换成`_`作为环境变量名, 设置值即可。
|
||||
|
||||
|
||||
## Zookeeper相关配置
|
||||
|
|
@ -322,28 +322,12 @@ common.properties配置文件目前主要是配置hadoop/s3/yarn相关的配置
|
|||
|
||||
## dolphinscheduler_env.sh [环境变量配置]
|
||||
|
||||
通过类似shell方式提交任务的的时候，会加载该配置文件中的环境变量到主机中。涉及到的 `JAVA_HOME`、元数据库、注册中心和任务类型配置，其中任务类型主要有: Shell任务、Python任务、Spark任务、Flink任务、Datax任务等等。
|
||||
通过类似shell方式提交任务的的时候，会加载该配置文件中的环境变量到主机中。涉及到的 `JAVA_HOME` 任务类型的环境配置，其中任务类型主要有: Shell任务、Python任务、Spark任务、Flink任务、Datax任务等等。
|
||||
|
||||
```bash
|
||||
# JAVA_HOME, will use it to start DolphinScheduler server
|
||||
export JAVA_HOME=${JAVA_HOME:-/opt/soft/java}
|
||||
|
||||
# Database related configuration, set database type, username and password
|
||||
export DATABASE=${DATABASE:-postgresql}
|
||||
export SPRING_PROFILES_ACTIVE=${DATABASE}
|
||||
export SPRING_DATASOURCE_URL
|
||||
export SPRING_DATASOURCE_USERNAME
|
||||
export SPRING_DATASOURCE_PASSWORD
|
||||
|
||||
# DolphinScheduler server related configuration
|
||||
export SPRING_CACHE_TYPE=${SPRING_CACHE_TYPE:-none}
|
||||
export SPRING_JACKSON_TIME_ZONE=${SPRING_JACKSON_TIME_ZONE:-UTC}
|
||||
export MASTER_FETCH_COMMAND_NUM=${MASTER_FETCH_COMMAND_NUM:-10}
|
||||
|
||||
# Registry center configuration, determines the type and link of the registry center
|
||||
export REGISTRY_TYPE=${REGISTRY_TYPE:-zookeeper}
|
||||
export REGISTRY_ZOOKEEPER_CONNECT_STRING=${REGISTRY_ZOOKEEPER_CONNECT_STRING:-localhost:2181}
|
||||
|
||||
# Tasks related configurations, need to change the configuration if you use the related tasks.
|
||||
export HADOOP_HOME=${HADOOP_HOME:-/opt/soft/hadoop}
|
||||
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-/opt/soft/hadoop/etc/hadoop}
|
||||
|
|
|
|||
|
|
@ -7,11 +7,12 @@
|
|||
> 如果使用 MySQL 需要手动下载 [mysql-connector-java 驱动][mysql] (8.0.16) 并移动到 DolphinScheduler 的每个模块的 libs 目录下，其中包括 `api-server/libs` 和 `alert-server/libs` 和 `master-server/libs` 和 `worker-server/libs`。
|
||||
|
||||
* 首先，参照 [数据源配置](datasource-setting.md) `伪分布式/分布式安装初始化数据库` 创建并初始化数据库
|
||||
* 在你的命令行或者修改 bin/env/dolphinscheduler_env.sh 设定下列环境变量，将 `{user}` 和 `{password}` 改为你数据库的用户名和密码
|
||||
* 在你的命令行设定下列环境变量，将 `{address}`, `{user}` 和 `{password}` 改为你数据库的地址, 用户名和密码
|
||||
|
||||
```shell
|
||||
export DATABASE=mysql
|
||||
export SPRING_PROFILES_ACTIVE=${DATABASE}
|
||||
export SPRING_DATASOURCE_URL="jdbc:mysql://{address}/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8&useSSL=false"
|
||||
export SPRING_DATASOURCE_USERNAME={user}
|
||||
export SPRING_DATASOURCE_PASSWORD={password}
|
||||
```
|
||||
|
|
@ -72,9 +73,10 @@ echo "host dolphinscheduler {user} {ip} md5" >> $PGDATA/pg_hba.conf
|
|||
pg_ctl reload
|
||||
```
|
||||
|
||||
然后修改`./bin/env/dolphinscheduler_env.sh`，将username和password改成你在上一步中设置的用户名{user}和密码{password}
|
||||
然后设置以下环境变量，将username和password改成你在上一步中设置的用户名{user}和密码{password}
|
||||
|
||||
对于 MySQL：
|
||||
|
||||
```shell
|
||||
# for mysql
|
||||
export DATABASE=${DATABASE:-mysql}
|
||||
|
|
@ -82,9 +84,10 @@ export SPRING_PROFILES_ACTIVE=${DATABASE}
|
|||
export SPRING_DATASOURCE_URL="jdbc:mysql://127.0.0.1:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8&useSSL=false"
|
||||
export SPRING_DATASOURCE_USERNAME={user}
|
||||
export SPRING_DATASOURCE_PASSWORD={password}
|
||||
```
|
||||
```
|
||||
|
||||
对于 PostgreSQL：
|
||||
|
||||
```shell
|
||||
# for postgresql
|
||||
export DATABASE=${DATABASE:-postgresql}
|
||||
|
|
|
|||
|
|
@ -16,7 +16,7 @@ DolphinScheduler 支持时区设置。
|
|||
|
||||
服务时区
|
||||
|
||||
使用脚本 `bin/dolphinshceduler_daemon.sh`启动服务， 服务的默认时区为UTC， 可以在`bin/env/dolphinscheduler_env.sh`中进行修改， 如`export SPRING_JACKSON_TIME_ZONE=${SPRING_JACKSON_TIME_ZONE:-Asia/Shanghai}`。<br>
|
||||
使用脚本 `bin/dolphinshceduler_daemon.sh`启动服务， 服务的默认时区为UTC， 可以在 `application.yaml` 文件中进行修改，或通过环境变量修改, 如`export SPRING_JACKSON_TIME_ZONE=${SPRING_JACKSON_TIME_ZONE:-Asia/Shanghai}`。<br>
|
||||
IDEA 启动服务默认时区为本地时区，可以加jvm参数如`-Duser.timezone=UTC`来修改时区。 时区选择详见[List of tz database time zones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones)
|
||||
|
||||
用户时区
|
||||
|
|
|
|||
|
|
@ -100,8 +100,6 @@ deployUser="dolphinscheduler"
|
|||
|
||||
* DolphinScheduler 的数据库配置，详细配置方法见[初始化数据库](#初始化数据库)
|
||||
* 一些任务类型外部依赖路径或库文件，如 `JAVA_HOME` 和 `SPARK_HOME`都是在这里定义的
|
||||
* 注册中心`zookeeper`
|
||||
* 服务端相关配置，比如缓存，时区设置等
|
||||
|
||||
如果您不使用某些任务类型，您可以忽略任务外部依赖项，但您必须根据您的环境更改 `JAVA_HOME`、注册中心和数据库相关配置。
|
||||
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
本教程使用三种不同的方式通过 Docker 完成 DolphinScheduler 的部署，如果你想要快速体验，推荐使用 standalone-server 镜像，
|
||||
如果你想要体验比较完成的服务，推荐使用 docker-compose 启动服务。如果你已经有自己的数据库或者 Zookeeper 服务
|
||||
你想要沿用这些基础服务，你可以参考沿用已有的 PostgreSQL 和 ZooKeeper 服务完成部署。
|
||||
你想要沿用这些基础服务，你可以参考沿用已有的 PostgreSQL 和 ZooKeeper 服务完成部署。
|
||||
|
||||
## 前置条件
|
||||
|
||||
|
|
@ -56,7 +56,7 @@ $ docker-compose --profile schema up -d
|
|||
$ docker-compose --profile all up -d
|
||||
```
|
||||
|
||||
> 提醒：通过 docker-compose 启动服务时，除了会启动 DolphinScheduler 对应的服务外，还会启动必要依赖服务，如数据库 PostgreSQL(用户
|
||||
> 提醒：通过 docker-compose 启动服务时，除了会启动 DolphinScheduler 对应的服务外，还会启动必要依赖服务，如数据库 PostgreSQL(用户
|
||||
> `root`, 密码 `root`, 数据库 `dolphinscheduler`) 和 服务发现 ZooKeeper。
|
||||
|
||||
### 沿用已有的 PostgreSQL 和 ZooKeeper 服务
|
||||
|
|
@ -126,4 +126,4 @@ $ docker run -d --name dolphinscheduler-alert-server \
|
|||
## 环境变量
|
||||
|
||||
可以通过环境变量来修改 Docker 运行的配置，我们在沿用已有的 PostgreSQL 和 ZooKeeper 服务中就通过环境变量修改了 Docker 的数据库配置和
|
||||
注册中心配置，关于全部的配置环境可以查看[全部的配置文件](https://github.com/apache/dolphinscheduler/blob/<version>/script/env/dolphinscheduler_env.sh) 了解 <!-- markdown-link-check-disable-line -->
|
||||
注册中心配置，关于全部的配置环境可以查看对应组件的 application.yaml 文件了解 <!-- markdown-link-check-disable-line -->
|
||||
|
|
|
|||
|
|
@ -22,10 +22,10 @@
|
|||
|
||||
### 数据库升级
|
||||
|
||||
修改 `./bin/env/dolphinscheduler_env.sh` 中的如下配置（{user}和{password}改成你数据库的用户名和密码），然后运行升级脚本。
|
||||
设置相关环境变量（{user}和{password}改成你数据库的用户名和密码），然后运行升级脚本。
|
||||
|
||||
下面以 MySQL 为例，别的数据库仅需要修改成对应的配置即可。请先手动下载 [mysql-connector-java 驱动 jar](https://downloads.MySQL.com/archives/c-j/)
|
||||
jar 包 并添加到 `./tools/libs` 目录下，修改 `./bin/env/dolphinscheduler_env.sh` 文件
|
||||
jar 包 并添加到 `./tools/libs` 目录下，设置以下环境变量
|
||||
|
||||
```shell
|
||||
export DATABASE=${DATABASE:-mysql}
|
||||
|
|
|
|||
|
|
@ -18,21 +18,8 @@
|
|||
# JAVA_HOME, will use it to start DolphinScheduler server
|
||||
export JAVA_HOME=${JAVA_HOME:-/opt/soft/java}
|
||||
|
||||
# Database related configuration, set database type, username and password
|
||||
export DATABASE=${DATABASE:-postgresql}
|
||||
export SPRING_PROFILES_ACTIVE=${DATABASE}
|
||||
export SPRING_DATASOURCE_URL
|
||||
export SPRING_DATASOURCE_USERNAME
|
||||
export SPRING_DATASOURCE_PASSWORD
|
||||
|
||||
# DolphinScheduler server related configuration
|
||||
export SPRING_CACHE_TYPE=${SPRING_CACHE_TYPE:-none}
|
||||
export SPRING_JACKSON_TIME_ZONE=${SPRING_JACKSON_TIME_ZONE:-UTC}
|
||||
export MASTER_FETCH_COMMAND_NUM=${MASTER_FETCH_COMMAND_NUM:-10}
|
||||
|
||||
# Registry center configuration, determines the type and link of the registry center
|
||||
export REGISTRY_TYPE=${REGISTRY_TYPE:-zookeeper}
|
||||
export REGISTRY_ZOOKEEPER_CONNECT_STRING=${REGISTRY_ZOOKEEPER_CONNECT_STRING:-localhost:2181}
|
||||
# Never put sensitive config such as database password here in your production environment,
|
||||
# this file will be sourced everytime a new task is executed.
|
||||
|
||||
# Tasks related configurations, need to change the configuration if you use the related tasks.
|
||||
export HADOOP_HOME=${HADOOP_HOME:-/opt/soft/hadoop}
|
||||
|
|
|
|||
Loading…
Reference in New Issue