There's a learning curve, and new features are being added, but at this point I would not hesitate to recommend Kubernetes to just about anyone.

CI: We standardize on CircleCI and it gets the job done, but has some serious shortcomings. I've also come close to building my own on top of the k8s cluster and it's not the correct time investment for me right now, but I'd consider building my own in the future. I've yet to find a CI framework I really like.

We switched over from Circle and it's been awesome to scale our build process without fear of resource constraints. It's pretty great :). They even have a CF setup that will auto scale boxes based on CI load.

Do you have any advice, or gotchas on getting into kubernetes development?

For me, getting GitLab up and running on kubernetes was a breeze (using the popular docker image [1]) and my `pg_dump; duplicity;` backups are chugging right along. I haven't played with their CI yet, but I'm pretty excited to see how much it can do for me automatically managing my cluster.

[1] https://github.com/sameersbn/docker-gitlab

I've worked with a self hosted Gitlab instance before and that was nice, but I'm not quite sure how well the Gitlab CI would integrate when the code is on github.

[0] https://github.com/vyshane/concourse-kubernetes

It is also very nice in that you can execute pipelines, and even just jobs, on the Concourse installation from your local CLI. This makes it much simpler to test pipeline/job tweaks without a single commit being necessary until it is verified to work. Yes, you can do something similar with the Jenkins Pipeline plugin suite using the Replay functionality but it is significantly more tedious (cut and paste from editor into a textbox in the Jenkins UI, bleh).

I use Gitlab CI and helm[1] for deploying. The last step of the ci process checks out the helm chart which is just another git repo and executes a helm install/upgrade CHART-NAME. Making things accessible is done through kubernetes ingress with nginx[2](which includes getting let's encrypt automatically for all external endpoints) so when I want to deploy a new staging version of the app I can do helm install app --set host=my-stage.domain.com .

There still a few gotchas like the pods won't update when a configmap was changed which is important because I keep the container configuration maps as configmaps. A crude workarround for this is [3] which triggers a configuration reload of the application running inside the container.

This solution has no licensing cost unlike Openshift(Tectonic[4] is another enterprise Kubernetes distribution which is free for 10 nodes) and the cost are based on the amount of time to set this up. But after you got into helm and more complex kubernetes deployments it should be easy.

[1] https://github.com/kubernetes/helm

[2] https://github.com/jetstack/kube-lego

[3] https://github.com/jimmidyson/configmap-reload

[4] https://coreos.com/tectonic/

[1]: https://github.com/Eneco/landscaper

thanks for the info. As far as I understand it you configure the state of the helm release + configured values in a file and apply it to a kube cluster. Depending on your cluster setup this is really helpful. Do you have a solution for triggering a rolling update of pods if a configmap has changed? In your examples I didn't see any configmaps.

I moved to Kubernetes about 6 months ago and have been really enjoying it. My first production cluster was hand rolled on AWS, where I found the cloud-provider load balancer integrations extremely helpful (https://kubernetes.io/docs/user-guide/load-balancer/).

I'm now using Google Container Engine which is effectively just a hosted Kubernetes cluster on GCP, which has really been 0 effort setup, and have been deploying to it with Wercker (http://www.wercker.com) [Disclaimer: I currently work at Wercker as of the last few months, but was a fan/user for many years before joining]

One thing I noticed across Openshift, Mesos, and Kubernetes: none of them handle the Docker daemon on a node hanging particularly well, which in my experience happens fairly often.

They also have CI/CD out of the box and builds can be triggered in your existing cluster with a 'convox build' or triggered on pushes to your private github repos.

Overall, unless you have a team that actually sees benefit in managing your own container and cluster manager(you better be big), id recommend embracing Convox, or something like it. The complexity still exposed by Kubernetes, OpenStack or ECS is still significant.

I'm one of two developers at a very young startup, and the one responsible for backend + devops stuff. I simply don't have the time to learn a more complex tool like Kubernetes (not that I didn't consider it) while also working on the actual product. Its simplicity has been a bit limiting on occasion, but they're happy to accept PRs for well thought out changes. I recently had a PR merged regarding UDP ports and ELBs that should make microservice architectures much easier and cheaper to implement.

A nice bonus is that they only charge you for the minion nodes. The Kube master is free.

https://cloud.google.com/container-engine/pricing

(I work for Google Cloud)

The biggest challenge right now is the ingress/loadbalancer abstraction. Hopefully, that should get resolved over the next few months.

https://cloudcraft.co/view/5582ddd4-c6f8-4354-8f5b-9fb0a3744...

* Development: docker + docker-compose. Ideally, we would want to get rid of docker-compose for development.

* CI: Travis (planning on switching to something that is more on the CD side)

* Infrastructure management: terraform

* Prod: AWS, CoreOs, Kubernetes 1 master node and 5-6 worker nodes (m4.large) in an autoscaling group.

Infrastructure deployments and updates are done by Terraform. Blue/Green deployments thanks to the autoscaling group.

Kubernetes deployments and updates are done by kubectl.

There's still problems with each piece, but for the most part they work great without much trouble.

Example kube-deploy files: https://github.com/UKHomeOffice/kube-piwik

Example app / drone files: https://github.com/UKHomeOffice/docker-piwik

Platform Documentation: https://github.com/UKHomeOffice/hosting-platform

KD - our deployment tool https://github.com/UKHomeOffice/kd

I can't really comment on whether or not this specific pipeline actually works as I've just picked a random open source example but the workflow is there.

We also have a legacy tool and use jenkins sometimes, but mostly that won't be open sourced.

Legacy deployment tool - don't use this. https://github.com/UKHomeOffice/kb8or

The build jobs creates images that are published to ECS repositories, and there are auto scaling groups that add and remove engine hosts to and from ALB target groups for each deployed service. It makes service discovery, scaling, etc. really easy.

Definitely try swarm out if you haven't already. 1.12 was good, 1.13 is amazing (secrets, health-based VIP membership, etc).

we have been considering playing with swarm. How many images and instances do you have, etc

And especially, how have you leveraged 1.13

I'd much prefer doing simple bare-metal deployments again.

I'd take the opposite stance, really. As far as the image format, it's the major differentiation Docker has, and IMO a really clean way of keeping image pulls DRY. Once your hosts have pulled a single image, given that you don't actively undermine it, subsequent pulls, even for different images, only need to retrieve the absolute minimum since they already have hopefully pulled the majority of the file system.

Disclaimer: I am the founder and CEO of Boxfuse

Surprised it wasn't already in the long list of suggestions. Have been using the tool when it was called Tutum. The guys behind Docker bought Tutum and renamed it to Docker Cloud. It's currently set up to redeploy services when I push an image to my repositories. Really loving the simplicity, even tough it's got some quirks.

You can now link your bitbucket or github repositories. Let it build your containers and deploy it to production. This way you can build an easy CI/CD pipeline.

With some personal flavor:

- I use autoredepleoy only in a test environment - We have a locally running old server with drone.io that is web-hooked to GitHub / BitBucket

Dokku's main advantage is that it's a no-brainer : if you're used to deploying heroku apps, it's very similar. It also automates the creation of data containers for database services, for example. On top of that, while I can use heroku's buildpacks for small sideprojects, I can also take full control of the build using a Dockerfile (which is what I do for bigger projects). The main inconvenient is that it can't manage multi host container deployments, like docker-swarm or kubernetes (I don't need it, so no need to compromise on simplicity).

Gitlab's pipeline both offer CI and CD, with a lot of cool features around it, like being able to tell on a commit page when it has been deployed on production, for no configuration cost.

Regarding costs : well, it's the cost of a dedicated server.

Next steps now is getting these projects deployed through a CI/CD. I evaluated a few, and it looks like I'm down to drone.io or jenkins.

This will bring some much needed sanity I need to keeping all these side / personal projects in order. I can go weeks or months without touching them, but then know EXACTLY how they will get tested and deployed.

Cost management is easy, all the projects are open-source and since we can spin Nomad up against any cloud provider or internal machine hosts, depending on what's the cheapest at the time. It's pretty easy to wrap your head around Nomad and make it do what you need.

0: https://www.nomadproject.io/

I also attended the DevConf.cz and saw a lot of presentations regarding Openshift. They have most of the talks on youtube (https://www.youtube.com/channel/UCmYAQDZIQGm_kPvemBc_qwg) in case somebody is interested

I was leveraging EFS as NFS mounts for my persistent volumes and had good results.

You might check out fabric8 if only for their visualizations of what is going on in your openshift / kubernetes environment.

Thanks for the youtube link! I'll be sure to check it out

https://github.com/redhat-openstack/openshift-on-openstack/

http://rancher.com/

We deploy all our containerized applications to Rancher (using Cattle for orchestration) via Jenkins jobs with a standardized Makefile for build, test, and deploy, making things consistent.

We look at running straight k8s, but it was like using a chainsaw to sharpen a pencil for our use case.

In addition their devs are extremely helpful and also have a hobby of getting things to run on ARM.

Rancher is also a breeze to deploy, I could manually deploy it with one hand tied behind my back in 10mn on AWS.

Rancher was the first thing I'd seen that claimed to be able to pull it off. I'll definitely give this more of a look.

I, personally, prefer the bare-metal deploys of automated scripts. I usually just spin up a VM and write a bash script to "prep" it the way I want. After that, I just run "./deploy" and it pushes where I want. I like this because I feel like I have more control and it actually feels easier. Plus, I've run into weird issues with Docker that take so long to debug that it completely cancels out the benefit of using it for me.

The bash script I have works for every side project I create, and is simply copied from project to project. :)

You're mixing terms here. A VM is not bare-metal.

I meant just an old-school deploy without containers and the sort.

Jenkins has a plugin that integrates with elastic beanstalk. This makes ci/cd straightforward.

There's no extra cost for elastic beanstalk, other than what you'd pay for ec2, s3 and elastic load balancer.

We've a starter template with a bunch of .ebextensions scripts that simplify common installation tasks.

If your application is a run-of-the-mill web app speaking to a database - elastic beanstalk is pretty much all you need.

At my current place, we are using Teamcity to run tests and build images, and Rancher for the orchestration part. I built a simple tool to handle auto-deployments to our different environments.

I cannot recommend Rancher enough. Especially for small teams, it's just a breeze to set up and use.

I have github hooked up to travis. When a new PR (or commit) is pushed travis shoves the app into its container, and runs the test suite inside the container.

If that passes AND the branch is master we push the image to docker hub. As of now we manually update the app inside of rancher but I think automating that will be a simple API call. Once we get more stable I'll be investigating that.

I still haven't quite figured out secret management but outside of that and a tiny learning curve it's been pretty smooth sailing.

An example travis config: https://github.com/OperationCode/operationcode_bot/blob/mast...

Automating the upgrades (i.e. redeploys) in Rancher is pretty straight forward - their API is super easy to use. I ended up writing a simple tool in mostly Bash to handle it, and threw it in a Docker container to run on Teamcity.

Started using Juju[1]. Basically, Juju handles bootstrapping/creating instances you need in the public clouds and you can use Juju Charms to specify how to deploy your services. So our deployment looks like this:

  - juju bootstrap google # Get instance in GCE
  - juju deploy my-app    # My app is deployed to GCE

  juju deploy wiki-simple

In our case, we have a production and development environments. Both are actually running in clouds in different regions

  - juju bootstrap google/us-east1-a production
  - juju deploy my-app

  - juju bootstrap google/europe-west1-c development
  - juju deploy my-app

We don't use any containers. Juju allows us to deploy our services in any clouds (aws, gce, azure, maas ...) including local using lxd.

[1] https://www.ubuntu.com/cloud/juju

[2] https://jujucharms.com/wiki-simple

https://kontena.io/

Flux is OSS [3], and we use it to deploy our commercial product, Weave Cloud, itself which runs on Kubernetes.

1: https://www.weave.works/continuous-delivery-weave-flux

2: https://cloud.weave.works

3: https://github.com/weaveworks/flux

<plug>we've been working a bit on an open source tool, pib, that supports setting up multiple environments because we ran into this problem (behind the scenes it uses terraform, k8s, and minikube). would love to hear if anyone here has seen anything similar or has thoughts! https://github.com/datawire/pib</plug>

Also, while I have a vested interest in saying this, you don't always want to solve this yourself. Look at hosted solutions like GCP and CircleCI to make things even more painless.

[0] http://getgandalf.com/

There's still a decent amount of human intelligence involved though, since obviously we want to give customers a good experience. This mainly comes in upfront (where we tune the implementation for each customer) and for any tasks which Gandalf hasn't learned to do yet. I've also invested in making it easier to train Gandalf to do things—for example, I can say "watch me" and then do a bunch of things with the AWS console/API and they get turned into a parametrizable playbook.

Any good DevOps engineer invests heavily in automation. Gandalf is just one level up of automating the process of automation.

If you have any other questions, feel free to email morgante@getgandalf.com.

A single commit to our master branch on GitHub can result in multiple service accounts, config maps, services, deployments, etc. to be created/updated. Making all of that work is complicated enough but then we also need to deal with things like canary deployments and letting us build and deploy to k8s from our local workstations. And then there are details like automatically deleting old images from ECR so your CICD process doesn’t fill that up without you knowing. Incorporating CICD processes with Kubernetes is kind of new so there is a lot of different projects and services starting to address this area.

I love Ansible. Chef is alright. I've been using AWS OpsWorks recently, and it's not bad. Elastic beanstalk is ok, too.

I've spun up some Kubernetes clusters, and it's nice, although I have no need for it yet. I remember the database situation was difficult when I was trying it last year. Something about persistent storage being difficult, so you had to run Postgres on a separate server.

I still like Capistrano. You can automate it with any CI pipeline. For one client, I used the "elbas" [1] gem for autoscaling on AWS. It automatically created new AMIs after deployment. Not super elegant, but it worked fine.

I don't see much of a middle ground between Heroku and Kubernetes. Just start with the one free dyno. Maybe ramp it up to 3 or 4 with hirefire.io. Once you're spending a few hundred per month on Heroku, that's probably the time to spin up a small kubernetes cluster and deploy stuff in containers.

[1] https://github.com/lserman/capistrano-elbas

Our current approach is using an Operable (https://operable.io) Cog we wrote which takes the kubernetes yaml and applies it to a running cluster. It's not perfect, but I'm pretty happy with the direction it's going. We built this cog in a public repo (https://github.com/retracedhq/k8s-cog) so you are welcome to use any of it, if it's useful. Then we have our CI service send a message (using SQS) after a build is done to deploy to staging.

Lately I've been using home-brew chatops to manage products so it's nice to hear what other people are using. Operable looks really interesting, I'm going to give it a shot. Thanks for the example cog!

Its a SaaS (and enterprise) platform for automated pipelines and deployments to Kubernetes clusters anywhere.

Previous discussion: https://news.ycombinator.com/item?id=13160218

disclaimer: I'm the founder at distelli.

For our dockerized services we use Nomad internally and for a different product we've built in AWS we're using Elastic Beanstalk with all of the resources defined in terraform.

We use jenkins to manage the CI/CD for each method.

For CI, we use our own product (Runnable [0]), which allows us to test our branches with their own full-stack environments, which is great for solid integration tests. We often use it for e2e too. We're planning on adding more CD features in the near future though.

[0] http://runnable.com

Docker Swarm (especially 1.13 https://www.infoq.com/news/2017/01/docker-1.13) is like Cassandra for me. Yes it has a few shortcomings, but it allows you to have a fairly reasonable cluster using a stupid compose.yml file and very quickly.

It's surprising the number of people who want to build a homebrew Kubes PaaS. When I first started working in development, every company was building its own CMS, until it invariably realised that it was hard and that they were better off using a commercial or open source solution. Seems that container-based platforms are history repeating itself.

This is Kubernetes, plus monitoring of your choice, running on your infrastructure, remotely managed by our team. The side benefit is that the same setup works on different infrastructure options, so you deploy and run the same stack on AWS and also on-premise/bare metal.

> How do you minimize costs with your solution?

Autoscaling groups triggered off of "cluster capacity".

Back in 2015, I implemented a Kubernetes by hand in AWS. I'm not going to do something like that again. GKE is fairly painless and it has most of the sensible defaults that I want. Networking just works -- pods can talk to each other as well as to any VM instances from any availability zone and region. Integrating with GCP service accounts just works. Spinning up experimental clusters is easy, as is horizontally scaling the clusters. One gotcha is that Google has not made K8S 1.5 generally available in all regions or availability zones. Otherwise, upgrades are pretty easy.

I have deployed with Docker Compose (not doing that again -- it is easier to use shell scripts). I have deployed with AWS ECS service (not doing that again; it does not have the concept of pods which severely constrains how you deploy). I used to deploy with Chef. I've heard of Chef's Habitat, but have not played with it.

Back for the 2015 project, I wrote Matsuri as a framework to manage the different Kubernetes templates. It's useful if you know Ruby. It uses idiomatic Ruby to generate and manage K8S specifications, and run kubectl commands. I wanted a single tool that could work with all the different environments (production, staging, etc.) as well as manage the dev environment. For example, if I want to diff my version-controlled spec on dev with what Kubernetes master currently has, I would use `bin/dev diff pod myapp`. If I want to diff the deployment resource by the same name, I would use `bin/production diff deployment myapp`. I can write hooks specific to the app. For example, `bin/production console mongodb` uses hooks to query Kubernetes to find a pod to attach to, determine the current Mongodb master, and invoke the command to go directly into the Mongodb shell. But I could have invoked `bin/staging console mongodb` or `bin/dev console mongodb`. I could do this because I have been developing software for a long time and I have enough ops experience to be able to put it all together. YMMV.

We're using Go.cd for the CD. I could have used Jenkins, but decided to give Go.cd a try. Go.cd has some advantages (such as much better topologies and tracking value streams) though there are also things it does not do as well as Jenkins (Go.cd auth mechanisms blow, and I had to write my own custom proxy to get Github hooks working more securely and reliably). Setting up GCP service accounts so that go.cd agents can deploy was a lot easier than I thought, once I read through the GCP docs. (Much easier than AWS).

Docker containers are still difficult to make. You want to vet things before using them. Handling this stuff is still going to be a full-time job for someone, both in terms of designing the infrastructure as well as the development tools. There are a lot of issues that come up because dev might throw things over the wall that might impact the overall reliability and performance of the system.

What have you found are the biggest advantages of pods over containers? How does ECS constrain how you deploy? Are you simply referring to rollout/rollback, scale up/down?

That meant I could not horizontally scale one container more than the other. I can scale the whole group, but there is a lot of wasted resources at that point.

Kubernetes pods group containers together under a single IP address. Containers from one pod (one IP address) can talk to any other pod. Docker did not even have this functionality until 1.12, and that is too little, too late. (And I am not sure this is something ECS supports right now). Combined with label selectors, long-running services (which binds a DNS name to the set selected by the label selectors), I can horizontally-scale pods and still maintain service discoverability. Using DNS makes service discovery stupid-easy. This means I can scale Kubernetes pods independently from each other.

Another consequence of using Service objects to select a set based on label selectors is routing can now be dynamic. Pods that need to talk to another pod goes through the service. I can then scale the dependency up and down, and it doesn't really affect the pod that requires that service. I can do rolling upgrades to the dependency, and it works because Service abstracts that through label selectors.

There are still some warts related to this setup. Stateful sets still needs a lot of work. I've also found that many applications caches IP addresses (redis sentinel being a notorious example). To work well with Kubernetes, it's better to always query DNS when making a connection. Ruby drivers for Mongodb and Redis, for example, will cache DNS lookup, making failover fragile (if you are running Mongodb and Redis inside Kubernetes; if you're not, you won't have this problem).

I was choosing between Kubernetes and Mesos after ECS, but had not looked into either deeply. It was random chance that took me to Kubernetes instead of Mesos. Kubernetes solved many of the pain points of Docker Compose and ECS.

It's largely undocumented. I've heard some interest in places to use it. Time constraints is such that creating examples for it is low on my priority list. Frankly, if you don't know Ruby, you're probably better off looking at Helm.

Do you have large workload spikes, or traffic spikes?

You should just use helm.

To deploy code with Salt, we had an SSH account on the Salt server configured with a bunch of deploy keys. Each of those had a forced command that would read $SSH_ORIGINAL_COMMAND and forward this information to an agent (running as root) that would execute Salt with the correct arguments, based on information in $SSH_ORIGINAL_COMMAND. This let us use a build step in TeamCity that basically did ssh deploy@mgmt-gateway [env] [project] [version]. Deployments were logged to New Relic and Slack.

In a different team that are fond of PHP we use a private GitLab CE for code management, GitLab CI Multi-runner as the build agent for CI/CD, Ansible for configuration management and code deploys to different environments running on AWS EC2. Like in the previous team, we have configured our .gitlab-ci.yml to pass some arguments in $SSH_ORIGINAL_COMMAND over SSH to a management node that in turns talks to Ansible.

Something I like with having a private GitLab CE instance is that development doesn't stop because your public Git host is DDoSed or have other problems (like the recently discussed one here on HN).

Test and staging servers are shutdown/destroyed off-hours and restarted/recreated by cron jobs that execute Ansible plays which identify eligible EC2 instances via EC2 tags. Production environments with multiple servers are similarly scaled down during off-hours. By simply modifying/removing the "shutdown" tag from the AWS resources, teams are able to exclude their test/staging environments from the scheduled shutdowns, something which is useful for upcoming releases. ;)

In the Node/Golang shop I loved how simple Docker images were and how good it felt to deploy it to isolated containers. Unfortunately, I don't see how that's possible (in a clean way, preferrably without using two images) when both an Nginx process (static file serving, e.g. frontend resources) and a PHP-FPM process needs access to the same code release.

(If you have experience with Nginx/PHP-FPM apps and Docker, feel free to enlighten me!)

Things I'm not entirely fond of about GitLab CI is that:

- each branch in each repo must have a .gitlab-ci.yml that is up-to-date (administrative challenge!)

- it's entirely driven from a git push (though the web gui provides buttons for existing builds to retry/manually execute steps to e.g. deploy code)

GitLab has no support for a centrally managed .gitlab-ci.yml file on a project group and/or project level. There's no way to define variables on a project group and/or project level. There's no way to schedule jobs so that you can execute daily/weekly tests, or to manage jobs (in a user-friendly way via the web gui) that perform cron-like tasks, so you can avoid putting these tasks on the server themselves in /etc/cron.d (which becomes a problem when you restore backups / bake AMIs / do auto-scaling).

I'd love to look more into K8 and Google's cloud offerings, especially since I believe this might be the future and because I believe Google are lightyears ahead of the competition when it comes to security and protecting the privacy of its customers. Unfortunately I'm afraid it's not viable given my team's current investment in Nginx/PHP-FPM apps and various AWS services.