Post

Deploy Jenkins on K3s

Posted
By Gastón Gallo
8 min read

We have seen on a previous article how to configure a K3s cluster on a VM. In this post we will see how to install Jenkins on it and use the cluster to dynamically assign agents as pods. Remember that each YAML file that you create can be applied to the K3s by using kubectl apply -f [yaml-path].

Jenkins deployment

First of all, we are going to create the namespace, you can create it with the command kubectl create namespace jenkins or using a YAML file:

1
2
3
4
5
6

kind: Namespace
apiVersion: v1
metadata:
  name: jenkins
  labels:
    name: jenkins

Next, we are going to create the Service Account so our Jenkins deployment can create pods for the agents:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins
  namespace: jenkins

---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: jenkins
  namespace: jenkins
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get","list","watch"]
- apiGroups: [""]
  resources: ["events"]
  verbs: ["watch"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: jenkins
  namespace: jenkins
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: jenkins
subjects:
- kind: ServiceAccount
  name: jenkins

Finally, we are going to create the deployment, service, Ingress and Persistent Volume:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins
  namespace: jenkins
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jenkins
      app.kubernetes.io/name: jenkins
      app.kubernetes.io/instance: jenkins
  template:
    metadata:
      labels:
        app: jenkins
        app.kubernetes.io/name: jenkins
        app.kubernetes.io/instance: jenkins
    spec:
      serviceAccountName: jenkins
      containers:
        - name: jenkins
          securityContext:
            runAsUser: 0
          image: "jenkins/jenkins:lts-jdk11"
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
            - name: agent
              containerPort: 50000
              protocol: TCP
          volumeMounts:
            - mountPath: "/var/jenkins_home/"
              name: jenkins-vol
      volumes:
        - name: jenkins-vol
          persistentVolumeClaim:
            claimName: jenkins-pvc
          
---
apiVersion: v1
kind: Service
metadata:
  name: jenkins
  namespace: jenkins
  labels:
    app: jenkins
spec:
  type: ClusterIP
  ports:
  - port: 80
    protocol: TCP
    targetPort: 8080
    name: http
  - port: 50000
    protocol: TCP
    targetPort: 50000
    name: agent
  selector:
    app: jenkins
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: jenkins
  namespace: jenkins
  labels:
    app: jenkins
spec:
  ingressClassName: nginx
  rules:
    - host: jenkins.local
      http:
        paths:
          - backend:
              service:
                name: jenkins
                port: 
                  number: 80
            path: /
            pathType: ImplementationSpecific
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: jenkins-pv
  namespace: jenkins
spec:
  storageClassName: jenkins-pv
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 20Gi
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: /data/jenkins-volume/
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-pvc
  namespace: jenkins
spec:
  storageClassName: jenkins-pv
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi

If your K3s is running in a VM, remember to add to your hosts file the correct entry so you can access. In this case I have assigned it jenkins.local as the host name, of course you can change to anything you like.

Also, I have assigned 20Gi for the PersistentVolume, feel free to modify this value to your liking, I’m not sure what is the minimum size required for Jenkins, but this is the value that I found in the official documentation when creating PersistentVolumes. This is important otherwise if your K3s or pod get restarted, you will lose all your configuration.

Jenkins initialization

Once you have your Jenkins running, you have to initialize it. If you enter to your Jenkins URL, you will be prompted for an Administrator password, to get retrieve that password you have to see the Jenkins log. You can see your Jenkins pods by running kubectl get pods -n jenkins, lookup for the pod name and then run kubectl logs [pod-name] -n jenkins. You should see the automatically-generated alphanumeric password between the sets of asterisks. Copy and paste it into the page prompt and click on Continue.

Now it will ask you to create an administrator account, take your time to create one, decide on its username and password, the continue. On the next window it will ask you if you want to Install suggested plugins or to Select plugins to install. If you are not sure, choose Install suggested plugins. The installation process may take a few minutes, but once it finished you can start using Jenkins.

Agent configuration

Setting up Kubernetes Cloud Provider

Once you have finished doing the initialization of Jenkins, you will be sent to the Dashboard. To the top right you will see a notification informing you that using the built-in node for running tasks is dangerous because it runs with the same privileges that the Jenkins instance. We will use our Kubernetes cluster to dynamically allocate agents to run tasks.

Click on the top right icon and select Manage Cloud provider, you will be sent to the Plugin Manager page showing you all cloud providers available. Choose Kubernetes by clicking its checkbox to the left and click the Download now and install after restart button. Wait for the download to finish and then click on the Restart Jenkins when no jobs are pending.

When it boots up, login with your account, go to Manage Jenkins > Manage Nodes and Clouds > Configure Clouds. On the Configure Clouds page click on Add a new cloud > Kubernetes.

A new section named Kubernetes will appear, name it kubernetes and click on Kubernetes Cloud details….

One important value is Kubernetes URL, in there you have to specify your Kubernetes API URL, to retrieve it run the following command:

1

kubectl config view

Then check the URL that appears on clusters:cluster:server, it could be something like https://127.0.0.1:6443, you have to change the 127.0.0.1 part with the IP of the VM where your K3s is installed, remember that you can use ip a s | grep 'inet ' to check the IP address. Paste https://[k3s-ip-address]:6443 in the Kubernetes URL field.

On the Kubernetes Namespace field input the same namespace of our Jenkins instance: jenkins.

You can click on the Test Connection button to make sure that Jenkins can access the Kubernetes API correctly.

Next on the Jenkins URL field insert your Jenkins Service name with an http prefix: http://jenkins. Since our agents will run on the same cluster and same namespace, the agents can find our Jenkins controller just by specifying the service name as URL.

Now we have to configure our Pod Templates, here we can define how our pods will be created. Click on the name and give it a name, for example: jenkins-agent. Click on Pod Template details… to configure other details:

  • On Namespace write the namespace of our Jenkins instance: jenkins.
  • On Labels, add the labels you want to use to identify our pod, if you are not sure about this, use the same value as the name: jenkins-agent.
  • On Usage choose Use this node as much as possible.

There are a lot of options to configure here to make your agent pods behave exactly as you want, and even add more pod templates. For now we will leave everything else as is, but you are free to check the official documentation for further configuration.

Click on Save and Apply.

Disabling the built-in node

Another important step is to disable the built-in node so it doesn’t keep assigning tasks to it. Go to Manage Jenkins > Manage Nodes and Clouds, click on the Settings gear icon on the right of the Built-In Node row.

Change the value of Number of executors to 0, then click on Save. With this you will have disabled the Built-In Node executors.

Testing the agents

We will make a really simple Pipeline to test that our agents are working. Click on New Item, give it a name (for example test-pipeline), click on Pipeline then on the Ok button.

On the next page, go to the Pipeline section on the bottom of the page, and on the Script textarea you will see a dropdown list that says try sample Pipeline…, click on it and select Hello World.

Click on Save, and on the next page click Build Now from the left. It will start building, you can run the following command on the terminal of your K3s to watch any change on your pod instances of the jenkins namespace: kubectl get pods -n jenkins -w It may take some time, but you should see a pod with the prefix jenkins-agent being created and then deleted. You can exit from this view by pressing Ctrl + C. If you don’t see anything maybe it has created and deleted the pod too fast before you ran the command, you can always check the Status page of the pipeline to check if it ran successfully, if you still want to check if it’s creating pods correctly, run the kubectl get pods -n jenkins -w command before pressing the Build Now button, then check again.

DevOps, Kubernetes
k3s kubernetes jenkins
This post is licensed under CC BY 4.0 by the author.