‘It’s not how you start it’s how you finish’, is a phrase you may hear a coach\nutter to his team but it is surprisingly applicable in the Tech industry as\nwell. All of us were newbies at some point in time. We’ve all faced challenges\nwhile trying to learn something new. However, reaching the end of the road to\nproficiency requires you to push through those tough times and I am here to let\nyou know that it is possible even when entering Kubernetes (k8s) as a\nKubernewbie. My background prior this undertaking was Design and Development\n(DevSigner), and not Ops. However, by the end of this journey I made the\ntransition and gained a new moniker, DevOpsSigner.
\nInitially, this blog post was going to serve as a comprehensive intro to\nKubernetes but, after some thought (and a few rewrites), I realized that this\nwould not be possible in a single post. It was better to approach this post from\nthe angle of what this is not meant to be. This post is not a comprehensive\nintroduction to Kubernetes, nor is it a tutorial or even a guide. It is more\nakin to a testimonial, a recap of our experience coming into Kubernetes and how\nwe went from Kubernewbies to a fully deployed production application in a\nKubernetes cluster in a few weeks.
\nWith no prior experience with Kubernetes or Devops it made the process of\nlearning and ramping up even more challenging. When entering a new domain, like\nDevops, there is a load of domain specific knowledge (DSK) you do not possess\nthat others have acquired over time. That DSK is what you use to evaluate,\nexplore, build solutions and use technologies in that domain. Without it you\ncould go down a path of reinventing the wheel. Every domain has very complex\nproblems that people are trying to solve, have solved or are improving on.\nFortunately for us and you that solution maybe primed and applicable to the need\n/ issue / requirement that you have. Kubernetes was that soltion for us.
\nKubernetes is a solution to a complex problem in its domain and by using\nKubernetes you cut a chunk of complexity in this initial DSK acquisition and can\nstart your Kubernewbie and Devops journey, ‘standing on the shoulders of\ngiants’. Kubernetes gifts you a system with a set of primitives, an API, and a\nset of hooks to build and focus on things unique to your needs. Joe Beda said it\nbest in that ‘Kubernetes gives you a Platform to make a Platform’. We will talk\nabout how we used this ‘Platform Builder’ (Kubernetes) if you will, later, but\nfirst let us define what Kubernetes is in general.
\nKubernetes.io introduces Kubernetes as “an open-source system for automating\ndeployment, scaling, and management of containerized applications.” Kubernetes\nbuilds upon 15 years of experience of running production workloads at Google\nwhich was influenced by Google’s internal systems Borg & Omega. So what does\nGoogle know about production workloads you ask? Well around 2015, Google was\nspinning up approximately 2 billion containers per week. Yes, you read that\ncorrectly, 2 billion per week. I don’t know about you, but I have not seen 2\nbillion of anything in even a year.
\nSo, it is fair to say that Google has a tremendous amount of experience\nutilizing containers in production. Fortunately, some brave souls started to\nopen-source that expertise. Harnessing that experience, then Google employees Joe\nBeda, Brendan Burns andCraig McLuckie took on the challenge. Brendan Burns\nstarted the initial prototypes which, combined with work from Joe Beda and Craig\nMcLuckie, resulted in the first commit to the Kubernetes repo in 2014.
\nKubernetes was created to bring the idea of dynamic, container-centric, managed,\nscheduled-cluster thinking outside of Google. With the exponential growth and\nusage of containers globally, Kubernetes strives to give everyone a production\nlevel solution fashioned from Google itself. There is no denying Google’s\nexperience with containers and containerized applications. But what is a\ncontainer and what does a containerized application mean in this context?
\nFor now, let’s say that a containerized application is an application that is\nwithin a container. A container, by definition from Webster’s Dictionary, is “an\nobject that can be used to hold or transport something.” In relation to\ntechnology, that “something” is everything that a piece of software needs to run\nin a particular environment, such as application code, utilities, configuration\nand dependencies. So a proposed Tech definition is: a container is an\nabstraction: an Operating-system-level-virtualization that allows you to run an\napplication and its dependencies in resource-isolated processes.
\nDiving even deeper into this tech definition, an Operating-system-level\nvirtualization, also known as containerization (containerized application),\nrefers to an operating system feature in which the kernel allows the existence\nof multiple resource-isolated user-space instances. Those instances are the\ncontainers that and share one Host Operating system across them. Since the\ncontainers encapsulate everything an application needs to run this makes moving\nand running the software in different environments possible. So with all these\ndefinitions in hand, my final description of a container would be: a Container\nis a isolated-resource that is used to transport everything an application needs\nto run.
\nThat was an intense explanation of what a container is, but container technology\nis a fundamental technology and a catalyst for the building of Kubernetes.\nKelsey Hightower said, “People will soon learn that containers only solve the\nsoftware packaging and distribution problem. Containers don't manage anything;\nthey need to be managed”. Kubernetes presents a management solution for\ncontainers. Lets now assume you have some containers built and ready for use,\nyou can have anywhere from one to one-thousand of them to deploy -- and you’ll\nneed a way to deploy, manage and even scale the system using these containers.
\nThat was the position we found ourselves in with a new project that we were\nhired to develop. We were hired to build a React application and a backend\napplication. The backend application was developed using Ruby, which would be a\nMicroservice / module living in an overall system of modules, powered by an\nAPI-Gateway called Okapi. We needed our own production and sandbox environments\nthat were fully functional versions of the overall Microservice Architecture.\nHaving this would allow us rapid development and iteration on our module, a\nplace to demo all the new features we would be implementing, and a sandbox\nlocation for experimentation. Since our module was going to be part of a larger\nsystem made up of multiple modules/containers we needed a deployment and\nmanagement solution. Enter Kubernetes.
\nWe researched a few options for a container management solution prior to\nsettling on Kubernetes. Many of which are capable of being a solution to\ncontainer management but ultimately we went with Kubernetes because of a few\nthings:
\nNow that we’d settled on using Kubernetes, we needed to decide on where to host\nour Kubernetes clusters, and whether we wanted to go with a managed or unmanaged\nKubernetes hosting solution. A managed solution has Provisioning, Security, Load\nBalancing, Upgrading, and Monitoring that is handled by them. An unmanaged\nsolution will require you to do more of the leg work to setup all the previously\nmentioned points. However, not all managed Kubernetes hosting solutions are the\nsame, so we researched a few. One that was pretty awesome was Platform9, which\nis a good turn key managed solution, but we ended up going with Google\nKubernetes Engine (previously Google Container Engine).
\nGoogle Kubernetes Engine (GKE) is a managed solution that abstracts away the\naforementioned Provisioning, Security, Load Balancing, Upgrading, and Monitoring\nof running a Kubernetes cluster, but not so much that you cannot configure (and\neven swap) an alternate technology for one of those parts. However, staying as\nclose as you can in the GKE system and using all the features within GKE will\nmake your life much easier. Now that we’ve decided on using Kubernetes for\norchestration, and GKE as the managed solution for hosting the cluster, let’s\nShip it. Well, not so fast.
\nBefore we actually shipped to GKE, we flexed our Kubernetes muscles and\nexperimented with Kubernetes using Minikube. The Minikube readme says that,
\n\n\n“Minikube is a tool that makes it easy to run Kubernetes locally. Minikube runs
\n
a single-node Kubernetes cluster inside a VM on your laptop for users looking to\ntry out Kubernetes or develop with it day-to-day,”.
\nWe used Minikube to initially test standing up and deploying all the modules.\nMinikube allowed us to get familiar with certain parts of Kubernetes, known as\nResource Types such as Pods, Services, Replica Sets, Deployments, Config Maps\netc and go through some of the examples in the Kubernetes documentation and\naround the web.
\nDuring this k8s exploration period using MiniKube we became very familiar with\nusing another tool called 'kubectl'. kubectl is a command-line interface (CLI)\nfor executing commands against a running Kubernetes cluster in this case the\nrunning cluster was the single-node Minikube instance but could be any running\nk8s cluster. kubectl has a laundry list of commands but our initial use was to\n‘apply’ our YAML files and as time progress we started to use more of the\nadvanced commands of kubectl. One resource that I wish I had during this\nexploration phase was a book called ‘Kubernetes Up and Running’ which I read\nlater but would suggest it to anyone that is new to k8s. Despite not having the\nbook at the time this step was crucial to gaining a basic understanding of\nKubernetes and standing up a cluster before shipping the full ancolata to GKE,\nwhich we did soon thereafter.
\nWe popped some champagne and celebrated the win of successfully standing up a\nKubernetes cluster with a fully functional API-Gateway and modules (including\nour module). Our clusters were running great, but after a few weeks and having\nto swap out and redeploy parts of it (and on occasion having to redeploy the\nentire cluster), we needed to find a solution to automate the process of tearing\ndown and standing up our cluster. Again, we did some market research, but we\ncouldn’t find anything that would fit our needs at the time; so we ended up\nbuilding Okubi, as mentioned earlier using the ‘Platform to build a Platform’.\nOkubi gives us a command line interface for tearing down and standing up our\ncluster.
\nWe use Okubi for things like:
\nGenerating module YAML files from a template.
\nChanging configuration values foreach environment from a configuration file.
\nSetting priority level to modules.This will allow explicit ordering of when certain modules are deployed.
\nOrder the deployment of the certain parts of the system to the cluster.
\nRun execution steps during the standup specific to this infrastructure.
\nOkubi is great, but it took hitting the pain points of doing all this manually\nbefore we found a solution that worked for us and built an abstraction.\nDefinitely have to give a big shout out to Joe\nLaSala for all his hardwork in Okubi. We\nhave some enhancements that we will like to introduce within Okubi but currently\nit is serving us well with cluster deployment. While having to redeploy modules\nin the cluster or redeploy the cluster we often found ourselves debugging\ncontainers.
\nTo debug containers one method that we use was to tail the logs of the\ncontainer. Which essentially watches the logs and prints them to the terminal.\nThough you can run a command in kubectl to tail the log of a container. We\nstarted to use Stern because of some added features. Stern simply describes\nitself as Multi pod container log tailing for Kubernetes. This was a feature\nthat we used along with tailing multiple containers within a Kubernetes “Pod”.\nIt maybe possible to do the same with kubectl but do not quote me. However,\nusing Stern made this task fairly simple with easy to remember CLI commands.
\nTailing logs is very useful but then we wondered if we could somehow interact\nwith the code while it was running in the cluster. Then we found Telepresence.\nTelepresence is an open source tool that lets you run a single service locally,\nwhile connecting that service to a remote Kubernetes cluster. So, by using\nTelepresence we were able to change the reality in your running cluster and\ntests code locally as though our local code was running in the cluster. In\naddition, we were able to find usage for Telepresence during active feature\ndevelopment. Yes, this all sounds quite magical and it did to me as first so let\nme try to explain how we used it.
\nFirst, use case was when we saw some buggy behavior and wanted to debug that\ncode. We would then switch to our sandbox cluster context utilizing another tool\ncalled Kubectx. Kubectx says it is a way to switch between clusters and\nnamespaces in kubectl and that is exactly right. Since our sandbox mirrors our\nproduction cluster we would then Telepresence into the sandbox cluster and then\nset debuggers within our local in order to step through the code during\nexecution. Once we found the offending code that was causing the bug we would\nthen create a PR with the fix.
\nSecond use-case as I mentioned earlier was the usage of Telepresence during\nactive feature development. This is one usage that came in handy for me several\ntimes. Let’s say I was task with developing a feature in the module. That new\nfeature would then be consumed by the React frontend that I aforementioned React\napplication that we had underdevelopment as well. Now, call me crazy but even\nthough I have all these awesome testing tools at my disposal and full test\ncoverage around my code because we do not ship code without tests. I still want\nto test it live. Hooray for Telepresence.
\nI would follow the same steps of making sure I was in the correct sandbox\ncontext and then Telepresence in that cluster. In addition, I would then spin up\nthe React application locally and make sure it was making it calls to the\nsandbox URL. This would give me a live React, a Telepresenced Ruby Module and a\nfully running Kubernetes cluster. I could manually test out what I was\ndeveloping and perform a full end to end test before I pushed to out a PR. I did\nnot use this all the time but on the occasions when I had to reach for this\ntechnique it was worth its weight in Gold.
\nWell, that is all that I have for this post. Honestly, I could keep writing but\nthis is supposed to be a blogPost and not a blogBook. So, I will keep the\nadditional topics for another blogPost. Overall, I would say that the journey\nfrom a Kubernewbie was very challenging and extremely rewarding. You have to\npush your limits and go into uncharted territory in order to grow. Have faith in\nyour abilities, ask questions and don’t get discouraged when you can’t figure\nsomething out. There is still much to learn but that is nature of the Beast\nwhich we call technology. If you have any question you can always hit me on\ntwitter at @elrickvm or send an\nemail. Be blessed and happy coding!
\nPhoto by chuttersnap on\nUnsplash.
\n