We’re going to setup a private on premises k8s cluster. It’s supposed to be provisioned in a vSphere.
To be able to deploy custom applications we need a private Docker registry somewhere accessible by this cluster. So, is it a good idea to host this private registry in the cluster itself (e.g. Nexus)? Or it’s better to isolate it from the cluster and host at some external VM?
Moreover, we have plans to migrate all CI/CD services in a k8s cluster (Git server, CI server, artifacts storage, etc.) Does it make sense to provision a dedicated cluster for them or separate namespace in the shared cluster will be enough?
CodePudding user response：
I would highly advise you to not host the registry in the cluster.
The main reason for it is, what happens if your cluster is broken? To be able to do a disaster recovery, you sometimes need to set up a new cluster. Or redeploy (a lot of) your applications.
You can't do this if your registry is also down with your cluster.
To your second question: In my opinion, this depends on your setup:
In case you rely on CI/CD for a cluster restore, then use a different cluster. In case not, I would ask you what the benefit is of putting it in the same cluster. As CI/CD has nothing to do with productive code, why would you put it in the same cluster (environment) at all. On the other hand, if CI gets the lowest priority in the cluster, I can understand that this saves a lot of money and can be preference at first.
Namespaces, in my opinion, are good for a separation of concerns (customer A from customer B). Though environments (staging, production, services) should not be mixed.
Bonus answer: If you are going to spin up a new cluster for CI/CD, you could also host your registry for your production/staging images. Though, keep in mind that you still should not host images for that new cluster on that registry.
CodePudding user response：
is it a good idea to host this private registry in the cluster itself
First question you should ask yourself: do you want to mirror images that are crucial to start Kubernetes up (eg: etcd, kube-apiserver, ...). Or is your registry meant to host application images? Assuming the latter: you may host your registry in that Kubernetes.
In theory, hosting your registry in Kubernetes would work just as well. Though when you'll have to troubleshoot some issue, maintain/upgrade that registry: depending on who has that responsibility, it is not "wrong" to go with a bare-metal/dedicated instance either.
(e.g. Nexus)?
Community edition? Or do you have a subscription?
Community comes without HA: I would rather look into Kraken, Harbor or Portus.
If you don't need to implement RBAC/privileges segmentation on a per-project or per-image basis: docker-registry works just as fine. Can be plugged to some s3 storage for HA
Moreover, we have plans to migrate all CI/CD services in a k8s cluster (Git server, CI server, artifacts storage, etc.)
Basing your CI/CD on Kubernetes is indeed a good idea.
Moving your Git server, CI server or artifacts storage to Kubernetes might sound like a good idea to begin with: don't rush into those.
Kubernetes can first be leverage running ephemeral workloads, your CI builds, tests, ... and eventually, running your dev/preprod/prod stages involving some CD, maybe some blue-green, ...
Before you consider migrating stateful workloads in Kubernetes: there's a whole other topic to assess: persisting data storage & CSI integration.
Does it make sense to provision a dedicated cluster for them or separate namespace in the shared cluster will be enough?
It doesn't necessarily make sense to create a separate cluster. You could start with two namespaces AND ResourceQuotas LimitRanges, ensuring your CI doesn't consume too much.
You could also make sure your CI runs on distinct set of nodes (run production workloads on nodes matching label foo=bar, and my CI on nodes matching foo=CI).