I recently talked about Ingress in Kubernetes at the Pune Kubernetes
meetup. This
post goes over that topic along with steps to demonstrate ingress
capabilities.
The British built Martello
towers all over the coast
inspired by one at Mortella, Italy where in 1794 British warships could
do little damage to these towers. The tower’s design was unique and
robust enough to hold a strong enemy attack. Load Balancers in my
opinion are the same – they are the firefighters of your applications
and hold the fort in case of an attack, intended or otherwise.
First, the warnings!
There are multiple ways to expose applications running in Kubernetes
pods outside the cluster. So first things first, let’s talk about *how
not to expose **your application. **Port
Forwarding ***is
fine for quick POCs but cannot be used for any serious application as it
simply forwards requests from a port o the client machine, directly to a
pod. Thus among many other limitations (which we will cover below) it
allows communication only between client and the individual pod.
HostPort or Host Network is a mechanism where ports on a worker
node are mapped to ports on pods. This means that those ports on the
host have to be opened up for outside communication and there cannot be
more than one pod per node. Moreover the scalability of such pods can be
only be as many as the number of nodes. With newer versions of
Kubernetes, the CNI plugin completely ignores the
HostPort
directive. Read
here
for more information.
Exposing Kubernetes Applications the right way
Services are the right way to expose an application within or outside a
cluster. If there is a public-facing application, there are following
options to expose it:
NodePort exposes the service over a port from range (30000-32767)
the port value can be provided explicitly in the service spec else it
gets chosen by Kubernetes automatically. This allows configuration of
physical Load Balancers over a combination of nodes and ports. If the
nodes have public IPs and access to node port is allowed, the
application can be accessed directly by sending the request to
NodeIP:NodePort. This means for each service exposed via NodePort, the
node port itself needs to be opened on all nodes. This is not a good
thing as we would be exposing ports of our worker nodes to the world.
Service type LoadBalancer is a wrapper over NodePort which simply
creates a load balancer in the cloud provider’s infrastructure that
support creation of an external Load Balancer. It adds all nodes as the
members of the Load Balancer. This is a far better solution but as the
number of public facing services grows, each service needs its own
LoadBalancer. This decision won’t go very well with your finance
department. Moreover, if the public facing service is not accessed
frequently, the service still needs to be available all the time.
This is where an Ingress comes into the picture. For more details on
above methods, refer to this excellent post by Mark
Betz
Ingress Components
When we talk about ingress, we refer to following parts:
**Ingress Resource **is a Kubernetes resource which defines rules that
are used by ingress controller to route incoming traffic.
**Ingress Controller **is an application which captures incoming
requests and routes them according to rules mentioned in the Ingress
resource.
**Default Backend **is another small application that simply catches
traffic for which there are no relevant rules defined via ingress
resources and presents a 404 page.
How Ingress controllers route a request
Now let’s see how a request gets routed to an internal service via an
Ingress Controller. Consider a Kubernetes cluster with all its worker
nodes in a private subnet. We have an application running in a
deployment exposed via a service of type ClusterIP which cannot be
accessed outside the cluster. In order to expose this service outside of
the cluster, we add an Ingress Controller which means we add a
Deployment of Ingress Controller and a
service (There
are other resources such as config maps, roles, role mapping etc but we
skip them for bravity, you can refer to the demo git
repo for more details on
each deployed resource).
Now we create an Ingress resource in our cluster which says *For any
packet with destination as demo.infracloud.space route it to service
demo-go-app-svc *(which is the blue circle in our diagram). The Ingress
controller keeps watching for creation/updation/removal of Ingress
resources. It immediately notices the creation of Ingress resource. The
Ingress controller reads the rule mentioned in the Ingress resource and
updates its configuration. Now the cluster is ready to serve requests.
We update our Route53 or similar DNS service to map the name
“demo.infracloud.space” to the Load Balancer for Ingress service
(Which we did manually but can be automated with Terraform or similar
utility. More on this a bit later in article).
Now a request arrives with destination set to “demo.infracloud.space”.
The LoadBalancer forwards it to the Ingress service. The Ingress service
then forwards the request to one of the Ingress pods. The pod which
receives the request will consult its existing routing rules and route
it to the internal service. In case a request arrives on the Load
Balancer for which there are no rules defined, it gets routed to a
default backend that serves a 404 page. There can be multiple internal
services to which routes can be created via different ingress
resources/rules in a single ingress resource.
We have a demo which works on AWS published in this ingress-demo repo. This demo requires
a Kubernetes cluster to be created on AWS (preferably via kops) and then
goes on to install an Ingress controller and installs a demo app which
can be accessed via different ingress resources. This should serve as a
good hands-on practice for first ingress implementation.

DNS update
Remember we had to manually add an entry for the subdomain into Route53
and point it to the load balancer associated with Ingress Controller? So
there are two distinct cases here.
Path-Based Routing
If your domain is going to be fixed (demo.infracloud.space) and if you
are going to add additional services based on path (Ex.
demo.infracloud.space/billing and demo.infracloud.space/auth and so on)
then it is just one time addition of pointing the domain to LB and rest
of updates will be taken care by ingress controller. You can choose to
automate the DNS update based on the number of environments you have,
but otherwise above setup should just work fine.
Sub Domain-Based Routing
If you are going to add subdomains for new services, for example, you
might have multiple tenants and each tenant wants a unique URL (Ex.
sherry.infracloud.space, john.infracloud.space and so on). Then every
time you have a new tenant, you will have to add a new entry to Route53
pointing to the same load balancer for ingress controller. You can
automate this as part of your workflow using Terraform/Ansible or
similar tooling. But we felt this is a problem which is better solved at
Kubernetes level. So we thought about
Ingress-Route53-Controller
(As of this is writing we just have a blank repo, more updates soon)
What about Scaling?
So in request path between the end user and the service, there are three
things: ELB/ALB, Ingress controller and the pod that serves the request.
The ELB scales well for up to 1M requests per second (With some
pre-warning to AWS in case of sudden bursts). The pods should be scaled
well with multiple replicas and
HPA.
Now how do we make sure that ingress controller pod is scaled well. We
can scale the replication controller & HPA so that there are enough
numbers of pods serving the ingress routing requests. Another strategy
is to use a daemon set of ingress controller pods so that a large number
of pods are handling the load.
Secondly, if your traffic is more than what a single ELB can serve
(More than 1M requests per second) – then you can have more than one LB
and associated ingress controllers. In this configuration, you can split
traffic based on load or evenly spread across load balancers.
ELB/ALB
For now, we have used an ELB (classic) in the demo, but it is possible
to use an ALB as well. We will update the demo soon to talk more about
this so keep watching the Github repo.
Conclusion
To summarize, an Ingress controller-based setup can act as a load
balancer for multiple services and save on management and cost of using
multiple load balancers. This setup is more secure and also more
scalable. There are also some interesting projects like the CoreOS
ALB and Traefik
ingress controller.
Looking for help with Kubernetes adoption or Day 2 operations? learn more about our capabilities and why startups & enterprises consider as one of the best Kubernetes consulting services companies.