基于 Rainbond 部署 DolphinScheduler 高可用集群

❝本文描述通过 Rainbond 云原生应用管理平台 一键部署高可用的 DolphinScheduler 集群,这种方式适合给不太了解 Kubernetes、容器化等复杂技术的用户使用,降低了在 Kubernetes 中部署 DolphinScheduler 的门槛。

Apache DolphinScheduler 是一个分布式易扩展的可视化 DAG 工作流任务调度开源系统。解决数据研发 ETL 错综复杂的依赖关系,不能直观监控任务健康状态等问题。DolphinScheduler 以 DAG 流式的方式将 Task 组装起来,可实时监控任务的运行状态,同时支持重试、从指定节点恢复失败、暂停及 Kill 任务等操作

「简单易用」:DAG 监控界面,所有流程定义都是可视化,通过拖拽任务定制 DAG,通过 API 方式与第三方系统对接, 一键部署

「高可靠性」:去中心化的多 Master 和多 Worker, 自身支持HA功能, 采用任务队列来避免过载,不会造成机器卡死

「丰富的使用场景」:支持暂停恢复操作.支持多租户,更好的应对大数据的使用场景. 支持更多的任务类型,如 spark, hive, mr, python, sub_process, shell

「高扩展性」:支持自定义任务类型,调度器使用分布式调度,调度能力随集群线性增长,Master 和 Worker 支持动态上下线

前提条件

可用的 Rainbond 云原生应用管理平台,请参阅文档 Rainbond 快速安装

DolphinScheduler 集群一键部署

  1. 对接并访问内置的开源应用商店,搜索关键词 dolp 即可找到 DolphinScheduler 应用。
  1. 点击 DolphinScheduler 右侧的 安装 进入安装页面,填写对应的信息,点击确定即可开始安装,自动跳转至应用视图。
选择项 说明
团队名称 用户自建的工作空间,以命名空间隔离
集群名称 选择 DolphinScheduler 被部署到哪一个 K8s 集群
选择应用 选择 DolphinScheduler 被部署到哪一个应用,应用中包含有若干有关联的组件
应用版本 选择 DolphinScheduler 的版本,目前可选版本为 3.0.0-beta2
  1. 等待几分钟后,DolphinScheduler 集群就会安装完成,并运行起来。
  1. 点击访问 DolphinScheduler-API 组件,默认的用户密码是admin / dolphinscheduler123

API Master Worker 节点伸缩

DolphinScheduler API、Master、Worker 都支持伸缩多个实例,多个实例可以保证整个集群的高可用性。

以 Worker 为例:

  • 进入组件内 -> 伸缩,设置实例数量。
  • 验证 Worker 节点,进入 DolphinScheduler UI -> 监控中心 -> Worker 查看节点信息。

<<< 左右滑动见更多 >>>

配置文件

API 和 Worker 服务共用 /opt/dolphinscheduler/conf/common.properties ,修改配置时只需修改 API 服务的配置文件。

如何支持 Python 3?

Worker 服务默认安装了 Python3,使用时可以添加环境变量 PYTHON_HOME=/usr/bin/python3

如何支持 Hadoop, Spark, DataX 等?

以 Datax 为例,使用通用数据初始化插件实现:

  1. 安装插件。Rainbond 团队视图 -> 插件 -> 从应用商店安装插件 -> 搜索 通用数据初始化插件 并安装。
  2. 开通插件。进入 Worker 组件内 -> 插件 -> 开通 通用数据初始化插件 ,并修改配置

    • FILE_URL:http://datax-opensource.oss-cn-hangzhou.aliyuncs.com/datax.tar.gz
    • FILE_PATH:/opt/soft
    • LOCK_PATH:/opt/soft
  3. 更新组件,初始化插件会自动下载 Datax 并解压到 /opt/soft目录下。

关于 Rainbond

Rainbond 核心100%开源,使用简单,不需要懂容器和Kubernetes,支持管理多种Kubernetes集群,提供企业级应用的全生命周期管理。

🌟 Github:https://github.com/goodrain/rainbond

💻 官网:https://www.rainbond.com

🏠 微信群:关注 Rainbond 公众号加入技术交流群

🏡 钉钉群:请搜索钉钉群号 31096419

K8S中文社区微信公众号

评论 4

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址
  1. #4

    I’m excited to write about Henry Hacker, he is a great and brilliant hacker who penetrated my spouse’s phone without a physical installation app. And I was able to access my spouse’s phone, SMS, Whatsapp, Instagram, Facebook, Wechat, Snapchat, Call Logs, Kik, Twitter and all social media. The most amazing thing there is that he restores all phone deleted text messages. And I also have access to everything including the phone gallery without touching the phone.I can see the whole secret of my spouse. Contact him for any hacking service. He is also a genius in repairing Credit Score, increasing school grade, Clear Criminal Record etc. His service is fast. Contact on: Henryclarkethicalhacker@gmail.com and you can text, call him on whatsapp him on +18134211326…

    Eliott Sharon2个月前 (09-19)回复
  2. #3

    Hello Guys, I just completed my divorce with my cheating husband of 10 years with two beautiful kids when i got a solid evidence of his unfaithfulness on extra marital affairs and his infidelity lifestyle, And the various applications he used to hide chats and lot of secret on his cell phone. Then, i decided to hire (Henry clark) an hacker and a PI just to be 100% sure because i don’t want to confront him or take any kinda step without proof, fact and figures because that could leads to defamation… It was with this great, honest, professional and trustworthy Man i got to know my husband is a professional cheater and has been using this app to hide most of his chats. This great hacker helped me broke to into his cell phone activities and wired everything he does on his cell phone directly to my phone and i was able to monitor and track him directly from my phone remotely without him knowing,
    Contact the Ethical hacker via email, Henryclarkethicalhacker@gmail.com you can also reach him on Whatsapp 18134211326..

    Steve Jenna2个月前 (09-19)回复
  3. #2

    Contact him for any type of hacking, he is a professional hacker that specializes in exposing cheating spouses, and every other hacking related issues. he is a cyber guru, he helps catch cheating spouses by hacking their communications like call, Facebook, text, emails, Skype, whats-app and many more. I have used this service before and he did a very good job, he gave me every proof I needed to know that my fiancee was cheating. You can contact him on his email to help you catch your cheating spouse, or for any other hacking related problems, like hacking websites, bank statement, grades and many more. he will definitely help you, he has helped a lot of people, contact him on, Henryclarkethicalhacker@gmail.com, and you can Text/Call &WhatsApp: +1 (262)-236-7526, and figure out your relationship status. I wish you the best.

    Lucy Janet3周前 (11-13)回复
  4. #1

    Tracking cellphones and getting an accurate report has been a bit worrisome, thanks to Jeffrey whose service helped me locate certain cellphones without breaking a sweat. His service made me know that the internet has become the most common method of cheating nowadays, either emotionally or physically. Taking a few extra steps in getting what would serve as proof for leaving a toxic relationship, to see who your spouse texts or chats with on social media isn’t a bad idea, I’d recommend you reach out to Jeffreyethicalhacker@gmail.com
    Text,call or whatsapp on: +1 (747)345-9036
    TYU

    dda liey5541周前 (11-23)回复