--- title: 'QuickTip: Tanzu TKC/Guest Cluster: Error "Permission denied" when creating file with securityContext' url: 'https://patrik.kernstock.net/blog/quicktip-tkc-permission-denied-creating-file' markdown: 'https://patrik.kernstock.net/blog/quicktip-tkc-permission-denied-creating-file.md' date: '2021-07-09' description: 'This time, I’m not getting into much detail about specific details around this topic. Consider this as a quick write-up/FYI. If you’ve ever encountered this issue or worked with securityContext‘s in Kubernetes, you might know what I’m talking about. Disclaimer: Before proceeding, please don’t see t…' taxonomy: category: - VMware - 'Tanzu & Kubernetes' tag: - QuickTip - Tanzu --- # [QuickTip: Tanzu TKC/Guest Cluster: Error "Permission denied" when creating file with securityContext](https://patrik.kernstock.net/blog/quicktip-tkc-permission-denied-creating-file) July 8, 2021 (updated August 22, 2021) 3 min [VMware](https://patrik.kernstock.net/category:VMware) [Tanzu & Kubernetes](https://patrik.kernstock.net/category:Tanzu%20%26%20Kubernetes) [QuickTip](https://patrik.kernstock.net/tag:QuickTip) [Tanzu](https://patrik.kernstock.net/tag:Tanzu) Note This article is more than 5 year(s) old – information may be outdated. ![QuickTip: Tanzu TKC/Guest Cluster: Error "Permission denied" when creating file with securityContext](https://patrik.kernstock.net/images/2/0/2/1/-/2021-07-0823-35-52-xwhp8ceg-88c6af88.png?g-b3a1b014 "QuickTip: Tanzu TKC/Guest Cluster: Error "Permission denied" when creating file with securityContext") This time, I’m not getting into much detail about specific details around this topic. Consider this as a quick write-up/FYI. If you’ve ever encountered this issue or worked with `securityContext`‘s in Kubernetes, you might know what I’m talking about. **Disclaimer**: Before proceeding, please don’t see this is as an official response from VMware. You continue on your own risk. ## Some context When deploying Grafana via TKG Extensions on your TKG(m) or TKGS/TKC, you might have encountered some "Permission denied" issues. This error is described in [kb.vmware.com/s/article/82108](https://kb.vmware.com/s/article/82108) pretty well. This also affects some TKGs releases, for example TKGS 1.20.2. ## Reproduction For some testing, I’ve written a yaml (inspired by [kubernetes docs](https://v1-20.docs.kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod)) to reproduce this error. I’ve used following: ```yaml apiVersion: v1 kind: Pod metadata: name: fstypetest spec: securityContext: runAsUser: 1000 runAsGroup: 3000 fsGroup: 2000 volumes: - name: fstypetest-volume persistentVolumeClaim: claimName: fstypetest-volume-claim containers: - name: fstypetest-pod image: busybox command: [ "sh", "-c", "sleep 1h" ] volumeMounts: - name: fstypetest-volume mountPath: /data securityContext: allowPrivilegeEscalation: false --- kind: PersistentVolumeClaim apiVersion: v1 metadata: name: fstypetest-volume-claim spec: storageClassName: vsan-raid1-1-failure accessModes: - ReadWriteOnce resources: requests: storage: 100Mi ``` Then get into the shell of this pod via `kubectl exec -it fstypetest -- sh` and you will see this: ```sh / $ ls -lsh /data total 12K 12 drwx------ 2 root root 12.0K Jul 8 21:49 lost+found / $ ls -lsh /data^C / $ touch /data/test touch: /data/test: Permission denied ``` ### How you want it to look like Now, when running the same specification on a GuestCluster with `TKGS 1.20.7` you however will see this: ```sh /data $ touch test /data $ ls -lsh total 0 0 -rw-r--r-- 1 1000 2000 0 Jul 8 21:22 test ``` (Does look better, doesn’t it?) So long story short: This issue seems to be finally solved in TKGS 1.20.7. ## The solution Updating your Tanzu Kubernetes Cluster/Guest Cluster to recently (5th of July 2021) released `TKGS 1.20.7` does fix the issue finally. ```sh $ kubectl get tanzukubernetesreleases | tail -n 5 v1.19.7---vmware.1-tkg.1.fc82c41 1.19.7+vmware.1-tkg.1.fc82c41 True True 20d [1.20.7+vmware.1-tkg.1.7fb9067 1.19.7+vmware.1-tkg.2.f52f85a] v1.19.7---vmware.1-tkg.2.f52f85a 1.19.7+vmware.1-tkg.2.f52f85a True True 20d [1.20.7+vmware.1-tkg.1.7fb9067] v1.20.2---vmware.1-tkg.1.1d4f79a 1.20.2+vmware.1-tkg.1.1d4f79a True True 20d [1.20.7+vmware.1-tkg.1.7fb9067] v1.20.2---vmware.1-tkg.2.3e10706 1.20.2+vmware.1-tkg.2.3e10706 True True 20d [1.20.7+vmware.1-tkg.1.7fb9067] v1.20.7---vmware.1-tkg.1.7fb9067 1.20.7+vmware.1-tkg.1.7fb9067 True True 3d2h ``` ## The workaround When you can’t or don’t want to update, there are two workarounds. ### Workaround #1 This workaround is inspired by [the steps provided for TKG(m) here](https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.2.1/rn/VMware-Tanzu-Kubernetes-Grid-121-Release-Notes.html#knownissues)), so they **should** be fine. For TKGS, you can use following: ```sh kubectl patch deployment -n vmware-system-csi vsphere-csi-controller --type=json -p='[{"op": "add", "path": "/spec/template/spec/containers/4/args/-", "value": "--default-fstype=ext4"}]' kubectl delete pod -n vmware-system-csi -l app=vsphere-csi-controller ``` This will patch your deployment named `vsphere-cis-controller`, then we remove all pods labeled with `app=vsphere-cis-controller`. As the deployment enforces multiple pods of the controller running at a given time, Kubernetes will start re-creating those deleted pods. This step is required to restart the pods, so that the deployment change takes effect. However to note, this could result in some interruption when interacting with PVCs (e.g. during creationg/deletion) in a busy environment. When issues appear, please consider raising an offical service request with VMware. ### Workaround #2 The second workaround is the one I was using alternatively and should be less disruptive, as it’s going to modify Storage Classes we already have. However the caveat is, that all storage classes needs to be modified. Check `kubectl get sc` for the names of your storage classes in use. 1. Export the Storage Class definition: ```sh kubectl get sc vsan-raid1-1-failure -o yaml > tmp_sc.yaml ``` 2. Modify Storage Class by adding this below `parameters` (watch out for indentation!) ```text csi.storage.k8s.io/fstype: "ext4" ``` 3. Replace the Storage Class: ```sh kubectl replace -f tmp_sc.yaml --force ``` The more automated way: ```sh kubectl get sc vsan-raid1-1-failure -o yaml > tmp_sc.yaml sed '/^parameters:.*/a\ \ csi.storage.k8s.io/fstype: "ext4"' -i tmp_sc.yaml kubectl replace -f tmp_sc.yaml --force ``` Beside this, getting to the latest version for the final fix might be the best idea on a longer run. Hope this helps some people out there! (Fun-Fact: I was writing this post as a solution for the current 1.20.2 release. During reproduction I figured out there’s 1.20.7 with a fix released.) ## Comments Sort by: Loading comments... --- ## Navigation - Parent: [Blog](https://patrik.kernstock.net/index.md) - Previous: [QuickTip: Enabling Fault Tolerance fails with "This device does not support vSphere Fault Tolerance."](https://patrik.kernstock.net/blog/quicktip-enabling-fault-tolerance-fails-with-this-device-does-not-support-vsphere-fault-tolerance.md) - Next: ["Downscaling" SupervisorControlPlaneVM's from three to two nodes](https://patrik.kernstock.net/blog/downscaling-supervisorcontrolplanevms-from-three-to-two-nodes.md)