London's bridge is falling down...

OMG! Last few days one of my LUN at PVE cluster is falling down after night backup!

Symptoms:


root@cl02-n02:~# pvscan
/dev/NAS01LUN0VG0/vzsnap-cl02-n02-0: read failed after 0 of 4096 at 21478965248: Input/output error
/dev/NAS01LUN0VG0/vzsnap-cl02-n02-0: read failed after 0 of 4096 at 21479022592: Input/output error
/dev/NAS01LUN0VG0/vzsnap-cl02-n02-0: read failed after 0 of 4096 at 0: Input/output error
/dev/NAS01LUN0VG0/vzsnap-cl02-n02-0: read failed after 0 of 4096 at 4096: Input/output error
/dev/sdb: Checksum error
PV /dev/sdk VG NAS01LUN9VG0 lvm2 [2.00 TiB / 2.00 TiB free]
PV /dev/sdj VG NAS01LUN8VG0 lvm2 [2.00 TiB / 2.00 TiB free]
PV /dev/sdi VG NAS01LUN7VG0 lvm2 [511.98 GiB / 511.98 GiB free]
PV /dev/sdh VG NAS01LUN6VG0 lvm2 [511.98 GiB / 511.98 GiB free]
PV /dev/sdg VG NAS01LUN5VG0 lvm2 [255.99 GiB / 159.99 GiB free]
PV /dev/sdf VG NAS01LUN4VG0 lvm2 [255.99 GiB / 157.99 GiB free]
PV /dev/sde VG NAS01LUN3VG0 lvm2 [127.99 GiB / 29.99 GiB free]
PV /dev/sdd VG NAS01LUN2VG0 lvm2 [127.99 GiB / 95.99 GiB free]
PV /dev/sdc VG NAS01LUN1VG1 lvm2 [127.99 GiB / 26.99 GiB free]
PV /dev/sda2 VG pve lvm2 [297.59 GiB / 16.00 GiB free]
PV /dev/sdb lvm2 [128.00 GiB]
Total: 11 [6.29 TiB] / in use: 10 [6.17 TiB] / in no VG: 1 [128.00 GiB]




Oh!
"/dev/sdb" is my LUN0 connected at PVE as VG named NAS01LUN0VG0

So, the solution is:
1) vgcfgrestore NAS01LUN0VG0
2) vgchange -ay NAS01LUN0VG0

So, next time we can see:

root@cl02-n02:~# pvscan
/dev/NAS01LUN0VG0/vzsnap-cl02-n02-0: read failed after 0 of 4096 at 21478965248: Input/output error
/dev/NAS01LUN0VG0/vzsnap-cl02-n02-0: read failed after 0 of 4096 at 21479022592: Input/output error
/dev/NAS01LUN0VG0/vzsnap-cl02-n02-0: read failed after 0 of 4096 at 0: Input/output error
/dev/NAS01LUN0VG0/vzsnap-cl02-n02-0: read failed after 0 of 4096 at 4096: Input/output error
PV /dev/sdk VG NAS01LUN9VG0 lvm2 [2.00 TiB / 2.00 TiB free]
PV /dev/sdj VG NAS01LUN8VG0 lvm2 [2.00 TiB / 2.00 TiB free]
PV /dev/sdi VG NAS01LUN7VG0 lvm2 [511.98 GiB / 511.98 GiB free]
PV /dev/sdh VG NAS01LUN6VG0 lvm2 [511.98 GiB / 511.98 GiB free]
PV /dev/sdg VG NAS01LUN5VG0 lvm2 [255.99 GiB / 159.99 GiB free]
PV /dev/sdf VG NAS01LUN4VG0 lvm2 [255.99 GiB / 157.99 GiB free]
PV /dev/sde VG NAS01LUN3VG0 lvm2 [127.99 GiB / 31.99 GiB free]
PV /dev/sdd VG NAS01LUN2VG0 lvm2 [127.99 GiB / 95.99 GiB free]
PV /dev/sdc VG NAS01LUN1VG1 lvm2 [127.99 GiB / 26.99 GiB free]
PV /dev/sdb VG NAS01LUN0VG0 lvm2 [127.99 GiB / 31.98 GiB free]
PV /dev/sda2 VG pve lvm2 [297.59 GiB / 16.00 GiB free]
Total: 11 [6.29 TiB] / in use: 11 [6.29 TiB] / in no VG: 0 [0 ]


But, WHY it happening? I don't know....my fair lady

PS. At storage side have no errors. All fine.