---
title: Operation and Custom Plugins for High Availability
tags: 
author: [Yahoo!デベロッパーネットワーク](https://convert.docswell.com/user/ydnjp)
site: [Docswell](https://www.docswell.com/)
thumbnail: https://bcdn.docswell.com/page/4VJPDMLWJ8.jpg?width=480
description: 2020.1.7に開催されたIngress勉強会での登壇。 イベントページ：https://cnt.connpass.com/event/160029/
published: January 08, 20
canonical: https://convert.docswell.com/s/ydnjp/ZX33R5-2020-01-08-152944
---
# Page. 1

![Page Image](https://bcdn.docswell.com/page/4VJPDMLWJ8.jpg)

Operation and Custom Plugins
for High Availability
Yahoo Japan Corporation
Jun Cui
January 7th, 2020


# Page. 2

![Page Image](https://bcdn.docswell.com/page/L2EV19Q9JQ.jpg)

About me
Jun Cui
• Site reliability engineer
at Yahoo! Japan(September 2017~)
• API Gateway developer
Hobby
• Playing Games(PS4, Nintendo Switch)
• Playing Monster Hunter World-Iceborne recently
image: screenshot of CAPCOM CO., LTD. monster hunter world-iceborne
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved.
2


# Page. 3

![Page Image](https://bcdn.docswell.com/page/Z57G2ZW47L.jpg)

Agenda
1. Kong usage states in Yahoo! Japan
2. Kong system configuration in Yahoo! Japan
3. Deploy Kong with Ansible
4. Custom plugins
5. Future work
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved.
3


# Page. 4

![Page Image](https://bcdn.docswell.com/page/V4EQPL3RJP.jpg)

1.Kong usage states in Yahoo! Japan
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved.
4


# Page. 5

![Page Image](https://bcdn.docswell.com/page/DKJ4YD1QE1.jpg)

About Yahoo! JAPAN
100+
services
150,000+
servers
Reference: https://promotionalads.yahoo.co.jp/online/yj_mediasheet201812.html
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved.
74,500,000,000+
PV/month
(average in 2018)
image: aflo
5


# Page. 6

![Page Image](https://bcdn.docswell.com/page/PLE16ZGLJG.jpg)

Kong usage states
・Release timetable
Kong EE 0.33-1
2018.10.1
Kong EE 0.33-2 Kong EE 0.34
2019.3
2019.5
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved.
NEW!
Kong EE 0.36-1
2019.9
6


# Page. 7

![Page Image](https://bcdn.docswell.com/page/8GEWW9KQE2.jpg)

Kong usage states
・ Configurations on Kong
・220+ Workspaces
・590+ Services
・690+ Routes
・ Req/sec
・7000+ average per Kong cluster
・12000+ peak per Kong cluster
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved.
7


# Page. 8

![Page Image](https://bcdn.docswell.com/page/W47Z59Z3E3.jpg)

2. Kong system configuration
in Yahoo! Japan
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved.
8


# Page. 9

![Page Image](https://bcdn.docswell.com/page/KYJ69KZZJV.jpg)

Kong for Backend
internal
client
Kong
(east)
Backend
(east)
Kong
(west)
Backend
(west)
GSLB
・2 Kong clusters for backend developers.
・GSLB (Global Server Load Balancer) is a network equipment which directs network
traffic to a group of data centers in various geographical locations.
・Developers could choose one or two Kong clusters depending on their availability
requirements.
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved.


# Page. 10

![Page Image](https://bcdn.docswell.com/page/5GJ5NPW9E4.jpg)

Kong for Frontend
external
client
Kong
(east)
Frontend
(east)
Kong
(west)
Frontend
(west)
GSLB
・2 Kong clusters for frontend developers.
・It proxies external requests to internal frontend servers.
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved.


# Page. 11

![Page Image](https://bcdn.docswell.com/page/4LE32342J5.jpg)

Kong clusters
Kong
(east)
external
client
Frontend
(east)
GSLB
Kong
(east)
Backend
(east)
Kong
(west)
Backend
(west)
GSLB
Kong
(west)
Frontend
(west)
・4 Kong clusters in production environment.
・2 clusters for frontend and 2 clusters for backend.
・Total 240 Kong nodes in 4 clusters.
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved.


# Page. 12

![Page Image](https://bcdn.docswell.com/page/V8EDZ4Q67G.jpg)

Active/standby in Kong clusters
Kong
(east)
external
client
Frontend
(east)
GSLB
Kong
(east)
Backend
(east)
Kong
(west)
Backend
(west)
GSLB
Kong
(west)
Frontend
(west)
・Kong checks upstreamʼs health on a regular basis.
・If one of FE/BE is not able to function normally, active/standby plugin failovers all the
requests to the other FE/BE.
・After failed FE/BE has been recovered, proxies will be returned to the original FE/BE.
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved.


# Page. 13

![Page Image](https://bcdn.docswell.com/page/4V7PDMGZ78.jpg)

3.Deploy Kong with Ansible
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved.
13


# Page. 14

![Page Image](https://bcdn.docswell.com/page/L2JV196M7Q.jpg)

Deploy Kong with Ansible
Screwdriver.cd
Artifactory
yum
publish
commit
YJ network
Production environment
git
ansible-playbook
ansible
resource
ssh
ansible
deploy
server
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved.
Kong
server
Kong
server
Kong
server


# Page. 15

![Page Image](https://bcdn.docswell.com/page/Z5EG2ZNXEL.jpg)

Package version and config file management
group_vars/all/all.yml
# kong.conf
kong_log_level: error
kong_proxy_listen_port: 80
# package version
kong_custom_plugin: kong-plugin-active-standby-1.x.x,
kong-plugin-sorry-page-2.x.x,
kong-plugin-custom-1.0.0
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved.


# Page. 16

![Page Image](https://bcdn.docswell.com/page/V4JQPLQ57P.jpg)

4.Custom plugins
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved.
16


# Page. 17

![Page Image](https://bcdn.docswell.com/page/DK74YDNVJ1.jpg)

#1 Traffic abuse prevention
Suspected to be abuse
API node1
Kong
Server
API node2
API node3
request deny
send event
Traffic abuse
prevention server
・Check accesses from Yahoo ID/remote IP/other attribute in fixed period.
・Deny requests when meet the limit, which configured by developers.
・Protect from DoS attack to reduce server load.
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved.
17


# Page. 18

![Page Image](https://bcdn.docswell.com/page/PLJ16Z547G.jpg)

#2 Sorry page
API node1
Kong
Server
API node2
API node3
5xx
return customize html/json
・Kong would return 503 if all targets of upstream become unhealthy states.
・Sorry page could respond to users with customized content and status code.
・Developers could customize content/status for each endpoint.
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved.
18


# Page. 19

![Page Image](https://bcdn.docswell.com/page/8GJWW95ZJ2.jpg)

#3 Active/standby
API node1
Kong server
upstream1
cluster1
API node2
API node3
route
service
API node4
active/standby
plugin
upstream2
cluster2
API node5
API node6
・Proxy all requests to API node of cluster1 in normal.
・Switch all requests to cluster2 if cluster1 crashed.
・Enable to offer our services as a whole continually.
・Simpler implementation of active/standby than canary release plugin.
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved.
19


# Page. 20

![Page Image](https://bcdn.docswell.com/page/W4EZ59NLJ3.jpg)

Active/standby vs Canary
・ Canary
・ Active/standby
&quot;config&quot;: {
&quot;standby_host”
}
&quot;config&quot;: {
&quot;duration&quot;,
&quot;groups&quot;,
&quot;hash&quot;,
&quot;percentage&quot;,
&quot;start&quot;,
&quot;steps”,
&quot;upstream_fallback&quot;,
&quot;upstream_host&quot;,
&quot;upstream_port&quot;,
&quot;upstream_uri&quot;
}
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved.
20


# Page. 21

![Page Image](https://bcdn.docswell.com/page/KY769K8M7V.jpg)

5. Future work
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved.
21


# Page. 22

![Page Image](https://bcdn.docswell.com/page/5G75NP9QJ4.jpg)

Future work
・Current Issues
・Users cannot see metrics data of their own service.
・Frontend developers do not expect to display Kong error to end-users.
・Deployment is a big work.
・Future work
・Provide Grafana to developers, using Kong vitals.
・Develop custom plugin to show customized error page instead of Kong error.
・Implement automatic deployment to reduce maintenance cost.
・Increase Kong nodes.
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved.
22


# Page. 23

![Page Image](https://bcdn.docswell.com/page/V9J2562WJR.jpg)

Thank you !!
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved.
23


