从零到一：手把手教你用Docker Compose快速部署DolphinScheduler 3.x集群（含避坑指南）

从零到一手把手教你用Docker Compose快速部署DolphinScheduler 3.x集群含避坑指南在当今数据驱动的时代高效的任务调度系统已成为企业数据流水线不可或缺的组成部分。DolphinScheduler作为Apache顶级开源项目以其可视化DAG工作流、丰富的任务类型支持和去中心化架构正在成为越来越多企业的选择。本文将带你从零开始通过Docker Compose这一轻量级容器编排工具快速搭建一个功能完整的DolphinScheduler 3.x集群环境。1. 环境准备与规划在开始部署之前我们需要明确几个关键点硬件需求即使是测试环境也建议至少4GB内存和2核CPU软件依赖Docker 20.10Docker Compose 2.0现代浏览器推荐Chrome或Edge注意生产环境需要根据任务量和并发度适当调整资源配置1.1 目录结构规划合理的目录结构能避免后续的配置混乱。建议创建如下目录mkdir -p dolphinscheduler/{data,logs,resource}这个结构将用于data存放PostgreSQL数据库持久化数据logs存储各组件运行日志resource作为资源中心挂载点2. Docker Compose文件解析下面是经过优化的docker-compose.yml文件包含了所有必要服务version: 3 services: postgresql: image: postgres:13 container_name: ds-postgres environment: POSTGRES_PASSWORD: dolphinscheduler POSTGRES_USER: dolphinscheduler POSTGRES_DB: dolphinscheduler volumes: - ./data/postgres:/var/lib/postgresql/data ports: - 5432:5432 healthcheck: test: [CMD-SHELL, pg_isready -U dolphinscheduler] interval: 5s timeout: 5s retries: 5 zookeeper: image: zookeeper:3.8 container_name: ds-zookeeper ports: - 2181:2181 environment: ZOO_MY_ID: 1 healthcheck: test: [CMD, zkServer.sh, status] interval: 10s timeout: 5s retries: 3 dolphinscheduler-master: image: apache/dolphinscheduler:3.1.0 container_name: ds-master depends_on: postgresql: condition: service_healthy zookeeper: condition: service_healthy environment: SPRING_DATASOURCE_URL: jdbc:postgresql://postgresql:5432/dolphinscheduler SPRING_DATASOURCE_USERNAME: dolphinscheduler SPRING_DATASOURCE_PASSWORD: dolphinscheduler REGISTRY_ZOOKEEPER_CONNECT_STRING: zookeeper:2181 MASTER_EXEC_THREADS: 10 MASTER_EXEC_TASK_NUM: 20 volumes: - ./logs/master:/opt/dolphinscheduler/logs ports: - 5678:5678 command: [master-server] dolphinscheduler-worker: image: apache/dolphinscheduler:3.1.0 container_name: ds-worker depends_on: postgresql: condition: service_healthy zookeeper: condition: service_healthy environment: SPRING_DATASOURCE_URL: jdbc:postgresql://postgresql:5432/dolphinscheduler SPRING_DATASOURCE_USERNAME: dolphinscheduler SPRING_DATASOURCE_PASSWORD: dolphinscheduler REGISTRY_ZOOKEEPER_CONNECT_STRING: zookeeper:2181 WORKER_EXEC_THREADS: 20 WORKER_HOST_WEIGHT: 100 volumes: - ./logs/worker:/opt/dolphinscheduler/logs - ./resource:/tmp/dolphinscheduler/resource ports: - 1234:1234 command: [worker-server] dolphinscheduler-api: image: apache/dolphinscheduler:3.1.0 container_name: ds-api depends_on: postgresql: condition: service_healthy zookeeper: condition: service_healthy environment: SPRING_DATASOURCE_URL: jdbc:postgresql://postgresql:5432/dolphinscheduler SPRING_DATASOURCE_USERNAME: dolphinscheduler SPRING_DATASOURCE_PASSWORD: dolphinscheduler REGISTRY_ZOOKEEPER_CONNECT_STRING: zookeeper:2181 volumes: - ./logs/api:/opt/dolphinscheduler/logs ports: - 12345:12345 command: [api-server] dolphinscheduler-alert: image: apache/dolphinscheduler:3.1.0 container_name: ds-alert depends_on: postgresql: condition: service_healthy zookeeper: condition: service_healthy environment: SPRING_DATASOURCE_URL: jdbc:postgresql://postgresql:5432/dolphinscheduler SPRING_DATASOURCE_USERNAME: dolphinscheduler SPRING_DATASOURCE_PASSWORD: dolphinscheduler REGISTRY_ZOOKEEPER_CONNECT_STRING: zookeeper:2181 volumes: - ./logs/alert:/opt/dolphinscheduler/logs command: [alert-server] dolphinscheduler-ui: image: apache/dolphinscheduler:3.1.0-ui container_name: ds-ui depends_on: dolphinscheduler-api: condition: service_started ports: - 8888:88882.1 关键配置说明数据库配置使用PostgreSQL 13作为元数据存储密码建议在生产环境中修改为更复杂的值数据卷挂载确保数据持久化ZooKeeper配置作为服务注册中心单节点模式适合测试生产环境建议集群部署Master ServerMASTER_EXEC_THREADS控制主节点执行线程数MASTER_EXEC_TASK_NUM控制每个批次处理的任务数量Worker ServerWORKER_EXEC_THREADS工作节点并发线程数WORKER_HOST_WEIGHT负载均衡权重资源目录挂载实现任务间文件共享3. 部署与初始化3.1 启动服务执行以下命令启动所有服务docker-compose up -d可以通过以下命令观察启动日志docker-compose logs -f3.2 数据库初始化DolphinScheduler会在首次启动时自动初始化数据库表结构。可以通过以下命令检查初始化是否完成docker exec -it ds-postgres psql -U dolphinscheduler -c \dt应该能看到约50张以t_ds_开头的表被创建。3.3 访问Web UI服务启动完成后访问http://localhost:8888使用默认凭据登录用户名admin密码dolphinscheduler1234. 常见问题与解决方案4.1 端口冲突处理如果出现端口冲突可以修改docker-compose.yml中的端口映射。常见需要修改的端口包括服务默认容器端口默认主机端口PostgreSQL54325432ZooKeeper21812181API Server1234512345UI888888884.2 资源不足问题如果Worker节点任务执行失败可能是资源限制导致。可以通过以下方式调整增加Worker线程数environment: WORKER_EXEC_THREADS: 50调整Docker资源限制docker update --cpus 2 --memory 4g ds-worker4.3 日志查看技巧各组件日志位于挂载的logs目录下可按服务查看tail -f logs/master/dolphinscheduler-master.log关键日志标记INFO正常操作日志WARN潜在问题警告ERROR需要立即处理的错误5. 生产环境优化建议5.1 高可用配置对于生产环境建议Master节点至少部署2个实例dolphinscheduler-master-1: # ...配置同前 ports: - 5678:5678 dolphinscheduler-master-2: # ...配置同前 ports: - 5679:5678Worker节点根据任务量横向扩展5.2 监控集成建议集成Prometheus监控添加以下配置environment: SPRING_METRICS_ENABLED: true SPRING_METRICS_EXPORT_PROMETHEUS_ENABLED: true MANAGEMENT_ENDPOINTS_WEB_EXPOSURE_INCLUDE: prometheus,health,info5.3 备份策略关键数据备份方案数据库定期备份docker exec ds-postgres pg_dump -U dolphinscheduler dolphinscheduler_backup.sql资源中心备份tar czvf resource_backup.tar.gz dolphinscheduler/resource/6. 进阶配置技巧6.1 邮件告警设置在UI界面配置SMTP信息后修改alert-server环境变量environment: ALERT_SMTP_SERVER: smtp.example.com ALERT_SMTP_PORT: 587 ALERT_SMTP_USER: your_emailexample.com ALERT_SMTP_PASSWORD: your_password ALERT_SMTP_SSL: true6.2 多租户配置创建租户INSERT INTO t_ds_tenant (tenant_code, description) VALUES (dev, 开发环境);分配Worker分组environment: WORKER_GROUPS: default,dev6.3 自定义任务插件要添加自定义任务类型创建插件目录mkdir -p dolphinscheduler/plugins修改Worker配置volumes: - ./plugins:/opt/dolphinscheduler/plugins7. 版本升级策略当需要升级到新版本时备份数据库和资源文件修改docker-compose.yml中的镜像版本执行滚动更新docker-compose pull docker-compose up -d注意大版本升级前务必检查官方升级文档可能有数据迁移要求