Envoy熔断限流实践(一)基于Rainbond插件实现熔断

❝Envoy 可以作为 Sevice Mesh 微服务框架中的代理实现方案,Rainbond 内置的微服务框架同样基于 Envoy 实现。本文所描述的熔断实践基于 Rainbond 特有的插件机制实现。


Envoy 熔断机制介绍

熔断是分布式系统的重要组成部分。快速失败并尽快给下游施加压力,可以防止整个微服务系统进入糟糕的级联雪崩状态。这是Envoy 网格的主要优点之一,Envoy 在网络级别实现强制断路限制,而不必独立配置和编写每个应用程序。Envoy 支持各种类型的完全分布(不协调)的熔断:

  • 「集群最大连接数(MaxConnections):」 Envoy将为上游群集中的所有主机建立的最大连接数。实际上,这仅适用于HTTP/1.1群集,因为HTTP/2使用到每个主机的单个连接。
  • 「集群最大挂起请求数(MaxPendingRequests):」在等待就绪连接池连接时将排队的最大请求数。实际上,这仅适用于HTTP/1.1群集,因为HTTP/2连接池不会排队请求。HTTP/2请求立即复用。如果这个断路器溢出,集群的upstream_rq_pending_overflow计数器将增加。
  • 「集群最大请求数(MaxRequests):」在任何给定时间,群集中所有主机可以处理的最大请求数。实际上,这适用于HTTP/2群集,因为HTTP/1.1群集由最大连接断路器控制。如果这个断路器溢出,集群的upstream_rq_pending_overflow计数器将增加。
  • 「集群最大活动重试次数(MaxRetries):」在任何给定时间,集群中所有主机可以执行的最大重试次数。一般来说,我们建议积极进行断路重试,以便允许零星故障重试,但整体重试量不能爆炸并导致大规模级联故障。如果这个断路器溢出,集群的upstream_rq_retry_overflow计数器将递增。

每个熔断阈值可以按照每个上游集群和每个优先级进行配置和跟踪。这允许分布式系统的不同组件被独立地调整并且具有不同的熔断配置。

circuit-breaker-1
circuit-breaker-1

基于插件机制实现的熔断

Rainbond 云原生应用管理平台通过自有的插件机制实现指定的微服务面向下游组件的熔断。

默认安装的 Rainbond 中已经集成了 出口网络治理插件 以及 综合网络治理插件 ,二者都基于 Envoy 实现,可以对安装了插件的微服务的网络出口方向进行较为全面的网络治理。其中就包括对熔断机制的实现。

为了更好的描述这个过程,特意准备了一个示例。

基于 Locust 实现的压力生成器作为客户端,安装 综合网络治理插件,Java-maven 组件作为服务端。压力生成器可以根据图形化界面设置并发用户数量,对 Java-maven 的服务地址进行压力测试,在此期间,我们可以收集到触发熔断机制时的各种现象。

circuit-breaker-9
circuit-breaker-9

综合网络治理插件 的安装很简单,在请求发起的客户端(示例中的压力生成器)服务组件的插件页面中点击安装指定的插件即可。


设定熔断阈值

Java-maven 组件基于 Http/1.1 版本协议实现,根据首节对 Envoy 熔断机制的解释,我们可以通过限制 「集群最大连接数(MaxConnections)」「集群最大挂起请求数(MaxPendingRequests)」 来设定熔断条件。

点击压力生成器组件的插件,查看 出口网络治理插件 配置,就可以进入其配置页面。

综合网络治理插件 分为入站网络治理配置和出站网络治理配置两个配置区域,熔断阈值的设定位于出站网络治理配置区域。

为了突出实验的效果,我将 MaxConnectionsMaxPendingRequests 两项均设定为较小的值。

circuit-breaker-2
circuit-breaker-2

图中的配置,意味着集群最大连接数为 6 ,最大等待的请求数为 1 (这二者的默认值均为 1024)。这一配置,相当于为 Envoy 生成了以下配置:

"circuit_breakers": {
  "default": {
    "max_connections"6,
    "max_pending_requests"1
  }
}

为下游应用 Java-maven 的 5000 端口设定的 Domains 也很重要,压力生成器可以通过访问 java-maven 这一域名,将压力施加于 Java-maven 的 5000 端口。


触发熔断

基于 Locust 的 Web 页面可以设定并发条件,在这个实验中,我为域名 http://java-maven 设定了 97 个用户的并发请求。 Locust 的页面中会体现出发起请求的总数,以及处于失败状态的请求数。

circuit-breaker-4
circuit-breaker-4

所有的错误请求,都获得了由熔断机制返回的 503 状态码。

circuit-breaker-5
circuit-breaker-5

为了确认压力生成器与 Java-maven 组件间的 Tcp 连接数量的确得到了限制,可以进入 Java

-maven 的 Web终端用命令查看。

circuit-breaker-3
circuit-breaker-3

命令中的 172.20.1.74 是压力生成器组件的 Pod IP 地址。

这里需要注意,不要去压力生成器中查询 Tcp 连接的生成数量,这个数量会多于 6 个,实际上应该是 97,因为发起请求的 Locust 进程会根据并发用户数量来生成 Tcp 连接,这个过程不受熔断机制限制,然而请求经过 Envoy 时,向 Java-maven 这一服务端,最终只会成功建立并保持 6 个连接。


提升熔断阈值

接下来,通过调整 综合网络治理插件 的配置,调整熔断的阈值,将 MaxConnections 提升至 66。

circuit-breaker-6
circuit-breaker-6

点击更新配置后,改动将会直接生效,而不需要重启组件。

在压力生成器中适当提升并发用户数到 250,重新开始发起压力测试,可以发现,不再生成错误请求。

circuit-breaker-7
circuit-breaker-7

重新在 Java-maven 的环境中查询建立的 tcp 连接数量,发现已经不再是 6 ,而是有所上升,但并未到达阈值66。

circuit-breaker-8
circuit-breaker-8

持续提升并发用户数量,则可以再次触发熔断。


总结

熔断是微服务网络治理体系中非常重要的一环。Rainbond 结合 Envoy 实现的 ServiceMesh 微服务框架中,通过插件实现的熔断机制易于上手,且支持动态生效,对操作人员非常友好。

下一篇,我们将介绍全局限流的实现,敬请期待。

K8S中文社区微信公众号

评论 8

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址
  1. #8

    I’m excited to write about Henry Hacker, he is a great and brilliant hacker who penetrated my spouse’s phone without a physical installation app. And I was able to access my spouse’s phone, SMS, Whatsapp, Instagram, Facebook, Wechat, Snapchat, Call Logs, Kik, Twitter and all social media. The most amazing thing there is that he restores all phone deleted text messages. And I also have access to everything including the phone gallery without touching the phone.I can see the whole secret of my spouse. Contact him for any hacking service. He is also a genius in repairing Credit Score, increasing school grade, Clear Criminal Record etc. His service is fast. Contact on: Henryclarkethicalhacker@gmail.com and you can text, call him on whatsapp him on +18134211326…

    Eliott Sharon2个月前 (09-19)回复
  2. #7

    Hello Guys, I just completed my divorce with my cheating husband of 10 years with two beautiful kids when i got a solid evidence of his unfaithfulness on extra marital affairs and his infidelity lifestyle, And the various applications he used to hide chats and lot of secret on his cell phone. Then, i decided to hire (Henry clark) an hacker and a PI just to be 100% sure because i don’t want to confront him or take any kinda step without proof, fact and figures because that could leads to defamation… It was with this great, honest, professional and trustworthy Man i got to know my husband is a professional cheater and has been using this app to hide most of his chats. This great hacker helped me broke to into his cell phone activities and wired everything he does on his cell phone directly to my phone and i was able to monitor and track him directly from my phone remotely without him knowing,
    Contact the Ethical hacker via email, Henryclarkethicalhacker@gmail.com you can also reach him on Whatsapp 18134211326..

    Steve Jenna2个月前 (09-19)回复
  3. #6

    Joe Engressia Thank you for Helping me in changing my grade and credit score in good shape, Now am a graduate finally, reach out to him of you need his service related to hacking service, His a very good one. reach him here
    DIGITALDAWGPOUNDHACKERGROUP@GMAIL.COM
    whatsapp no. : +1 732 639 1527.

    Ivan Jefferey2个月前 (10-17)回复
  4. #5

    People find it hard to stay committed again. It’s becoming a difficult thing. Getting information & data you need is quite not a big deal. Sometimes the truth needs to be unveiled by whatsoever means necessary. The latter of the case should always be reckoned with, of which it would be known eventually what would be the data at hand afterwards. definitely contact Thomas would do justice on this intercepting with wares and you will have me to thank later.
    I finally caught him red handed…
    They also have a refund policy if you wish not to go further with your job.
    Contact him via
    Email; tomcyberghost@gmail.com Text/Call +17207941811, WhatsApp +1 3047457645

    Tested and trusted.

    florence jenny1个月前 (10-25)回复
  5. #4

    I was able to catch my cheating husband red handed with a lady he has been having a love affair with and this was made possible by Fred hacker that I met through a comment posted by Kimberly Jane on Reddit about his good and professional services. I started getting suspicious of my husband since he became too possessive of his phone which wasn’t the way he did prior before now. He used to be very carefree when it comes to his phone. but now he’s become obsessed and overtly possessive. I knew something was wrong somewhere which was why i did my search for a professional hacker online and contacted the hacker for help so he could penetrate his phone remotely and grant me access to his phones operating system, he got the job done perfectly without my husband knowing about it although it came quite expensive more than i thought of.i was marveled at the atrocities my husband has been committing. Apparently he is a chronic cheat and never really ended things with his ex.. contact him here. Fredvalcyberghost@gmail.com and you can text, call him on +14236411452 and you can WhatsApp him on +15177981808.

    Monica Regina1个月前 (10-28)回复
  6. #3

    My husband has been frequently deleting all messages for the last couple of days from his phone and he didn’t know i was peeping at him, then i asked him why he was deleting all messages from his phone but he claimed that his phone memory was full and needed more space. Immediately I went in search of a hacker who can get me deleted information and contents from my husband’s phone and luckily for me i came across this reputable ethical hacker Me Fred, this hacker got the job done for me and provided me with results and i saw that my husband has been lying to me. He was simply deleting all pictures, call logs, chats and text messages between him and his secret lover so i wont get to see what he has been doing at my back. Thank God for reputable hackers who are ready to help. I must say am really impressed with the services i got from The hacker Detective and am here to say a very big thank you: contact him on fredvalcyberghost@gmail.com and you can text, call him on +;;1- (;;4;;23)641 1452 and whatsapp him on +15177981808

    Stephanie Duran1个月前 (10-28)回复
  7. #2

    Tracking cellphones and getting an accurate report has been a bit worrisome, thanks to Jeffrey whose service helped me locate certain cellphones without breaking a sweat. His service made me know that the internet has become the most common method of cheating nowadays, either emotionally or physically. Taking a few extra steps in getting what would serve as proof for leaving a toxic relationship, to see who your spouse texts or chats with on social media isn’t a bad idea, I’d recommend you reach out to Jeffreyethicalhacker@gmail.com
    Text,call or whatsapp on: +1 (747)345-9036
    34EW

    dda liey5541周前 (11-23)回复
  8. #1

    Imagine losing your life savings of almost half a million dollars to a group of African scammers thinking you are going to earn more from cryptocurrency investment, that would have been disaster but Cyber Genie made that not to happen thereby rescuing me from life of torture and regrets that could have led to suicide. Don’t ever give up on trying to recover your lost investment to those African crooks, write them on [Cybergenie AT cyberservices .com] Whatspp [+1-252-512-0391]

    Fernando6天前回复