Mount failed with error: rpc error: [failed to authenticate credentials for azstorage] #1689
Comments
|
@Hidayathullashaik there is auth error, how did you create the you could use following way to create the secret with accoutname and key: kubectl create secret generic industrialqa-secret --from-literal=azurestorageaccountname="xxx" --from-literal azurestorageaccountkey="xxx" --type=Opaque |
|
@andyzhangx - Hi, Thank you for the suggestion. I have already created the industrialqa-secret before creating the PV, PVCs in both the ways using the Rancher UI Portal & through the imperative kubectl commands with the preveleged credentials but I'm still encountering the same error: MountVolume.MountDevice failed for volume "industrial-pv-blob" : rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exit status 1 Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage] , output: Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors. Please let me know if there are any additional steps or configurations I should try. |
|
@Hidayathullashaik to narrow down the issue, you could follow this guide to check whether blobfuse mount works on the agent node: https://github.com/kubernetes-sigs/blob-csi-driver/blob/master/docs/csi-debug.md#troubleshooting-connection-failure-on-agent-node |
|
Hi @andyzhangx, After re-installing K3s, I am now encountering a new error: Error: This is a different error compared to the previous one. Could you please provide guidance on how to resolve this new issue? Any insights would be greatly appreciated. |
|
@Hidayathullashaik pls follow this guide to check whether blobfuse mount works on the agent node: https://github.com/kubernetes-sigs/blob-csi-driver/blob/master/docs/csi-debug.md#troubleshooting-connection-failure-on-agent-node |
|
@andyzhangx - Getting the same authentication failed error for the blobfuse2 mount in all the nodes. I have exported the Storage account name and key as well. [root@peplapsete01 ~]# [pepsense@peplapsete03 ~]$ |
|
pls check the /var/log/blobfuse2.log to get the detailed error msg |
|
@andyzhangx - I’ve reviewed the logs in /var/log/blobfuse2.log, and here are the relevant error details I found: Nov 14 03:57:20 peplapsete01 blobfuse2[2502872]: [/root/test] LOG_CRIT [mount.go (432)]: Starting Blobfuse2 Mount : 2.3.2 on [Red Hat Enterprise Linux 8.8 (Ootpa)] |
|
have you provided the right account key? you could file issue here: https://github.com/Azure/azure-storage-fuse/issues since it's a blobfuse auth issue. |
|
@andyzhangx - Yes, I have reviewed the storage account name & key and it is correct. Sure will log the issue in the above shared url. Thankyou for your support. |
|
@andyzhangx - Thankyou for all your support and follow-up. This issue has been resolved now and can mark this issue as completed. Resolution There was time difference/lag in the VM/Server due to which the blobfuse2 request to storageaccount while authenticating is considering as too old and clearly seen in the blobfuse2.log file sample shared below. blobfuse2.log file Error
Once again Thankyou ! |
|
This issue has been resolved |
Hi, I am getting a repitative error for azure blob credentials while provisioning the PV, PVCs in the Kubernetes [K3s] Cluster using blobfuse protocol. I have verified the az blob storage account credentials working in other clusters. Need assistance/support to fix the issue.
Error:
MountVolume.MountDevice failed for volume "industrial-pv-blob" : rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exit status 1 Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage] , output: Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors.
CSI Blob Node Logs
volumeId _pepsensecst-demo
context map[containerName:pepsensecst-demo protocol:fuse2 resourceGroup: storageAccount:]
mountflags [-o allow_other --file-cache-timeout-in-seconds=120]
mountOptions [-o allow_other --file-cache-timeout-in-seconds=120 --empty-dir-check=false --tmp-path=/mnt/-demo --container-name=pepsensecst-demo --pre-mount-validate=true --use-https=true --cancel-list-on-mount-seconds=10]
args /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/ac57e1b1e08147bd8749693401e850fc7d31875873642a7150780425d14de8bc/globalmount -o allow_other --file-cache-timeout-in-seconds=120 --empty-dir-check=false --tmp-path=/mnt/_pepsensecst-demo --container-name=pepsensecst-demo --pre-mount-validate=true --use-https=true --cancel-list-on-mount-seconds=10
serverAddress .blob.core.windows.net
I1111 12:05:04.424943 1244193 nodeserver.go:154] start connecting to blobfuse proxy, protocol: fuse2, args: /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/ac57e1b1e08147bd8749693401e850fc7d31875873642a7150780425d14de8bc/globalmount -o allow_other --file-cache-timeout-in-seconds=120 --empty-dir-check=false --tmp-path=/mnt/_pepsensecst-demo --container-name=pepsensecst-demo --pre-mount-validate=true --use-https=true --cancel-list-on-mount-seconds=10
I1111 12:05:04.430847 1244193 nodeserver.go:163] begin to mount with blobfuse proxy, protocol: fuse2, args: /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/ac57e1b1e08147bd8749693401e850fc7d31875873642a7150780425d14de8bc/globalmount -o allow_other --file-cache-timeout-in-seconds=120 --empty-dir-check=false --tmp-path=/mnt/_pepsensecst-demo --container-name=pepsensecst-demo --pre-mount-validate=true --use-https=true --cancel-list-on-mount-seconds=10
E1111 12:05:06.910818 1244193 nodeserver.go:166] GRPC call returned with an error:rpc error: code = Unknown desc = exit status 1 Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage]
E1111 12:05:06.910971 1244193 nodeserver.go:392] rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exit status 1 Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage]
, output:
Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors.
E1111 12:05:06.911130 1244193 utils.go:80] GRPC error: rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exit status 1 Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage]
, output:
Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors.
I1111 12:05:07.446630 1244193 utils.go:75] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 12:05:07.446664 1244193 utils.go:76] GRPC request: {"secrets":"stripped","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/ac57e1b1e08147bd8749693401e850fc7d31875873642a7150780425d14de8bc/globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["-o allow_other","--file-cache-timeout-in-seconds=120"]}},"access_mode":{"mode":5}},"volume_context":{"containerName":"pepsensecst-demo","protocol":"fuse2","resourceGroup":"","storageAccount":""},"volume_id":"_pepsensecst-demo"}
I1111 12:05:07.447554 1244193 blob.go:386] parsing volumeID(_pepsensecst-demo) return with error: error parsing volume id: "_pepsensecst-demo", should at least contain two #
I1111 12:05:07.447591 1244193 blob.go:458] volumeID(_pepsensecst-demo) authEnv: []
I1111 12:05:07.447654 1244193 nodeserver.go:366] target /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/ac57e1b1e08147bd8749693401e850fc7d31875873642a7150780425d14de8bc/globalmountprotocol fuse2
Validations
I have validated the connectivity from the vm to storage account and shared the details below
[root@peplapsete01 ~]#
[root@peplapsete01 ~]# nc -zv .blob.core.windows.net 443
Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Connected to 20.209.146.193:443.
Ncat: 0 bytes sent, 0 bytes received in 0.57 seconds.
[root@peplapsete01 ~]#
[root@peplapsete01 ~]#
Configurations
PV.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
pv.kubernetes.io/provisioned-by: blob.csi.azure.com
name: industrial-pv-blob
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
storageClassName: blob-fuse
mountOptions:
- -o allow_other
- --file-cache-timeout-in-seconds=120
csi:
driver: blob.csi.azure.com
readOnly: false
volumeHandle: _pepsensecst-demo
volumeAttributes:
resourceGroup:
storageAccount:
containerName: pepsensecst-demo
protocol: fuse2
nodeStageSecretRef:
name: industrialqa-secret
namespace: default
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: industrial-pvc-blob
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
volumeName: industrial-pv-blob
storageClassName: blob-fuse
kind: Pod
apiVersion: v1
metadata:
name: nginx-blob
spec:
nodeSelector:
"kubernetes.io/os": linux
containers:
- image: mcr.microsoft.com/oss/nginx/nginx:1.17.3-alpine
name: nginx-blob
command:
- "/bin/sh"
- "-c"
- while true; do echo $(date) >> /mnt/blob/outfile; sleep 1; done
volumeMounts:
- name: blob01
mountPath: "/mnt/blob"
resources:
limits:
cpu: 0.5
memory: "512Mi"
requests:
cpu: 0.1
memory: "128Mi"
volumes:
- name: blob01
persistentVolumeClaim:
claimName: industrial-pvc-blob
Expected Behaviour
Blobfuse should make connection from the vm to the azurestorageaccount and provision a volume successfully in the container.
The text was updated successfully, but these errors were encountered: