Vault fails with "input/output error"


If you receive an error similar to:

Block vault (copy) of 123456789ABCDEF1 failed with error cp: failed to close '/mnt/ods1/blocks/123/123456789abcdef1': Input/output error

this means that there is likely a problem communicating with your offsite storage. If you are using convention disk or network storage, there may be additional root cause errors listed in dmesg. Please see the following KB for more information:

https://docs.quadricsoftware.com/kb/troubleshooting/troubleshooting-storage-issues/


If you are using S3 or similar, however, you will need to drop to the shell (option 4) and check the log /var/log/syslog using the following command:


tail /var/log/syslog |less


look for a timestamp on the left of the log that correlates with the time of your error above. Because the A2 will retry several times after an error, it could be up to ~15 minutes prior to your error that the storage issue occurs.


Look for errors such as the following, which indicate an issue communicating with your cloud storage:

3 Oct 8 08:45:35 /usr/local/bin/goofys[745]: s3.ERROR code=Service Unavailable msg=503 request=58ba04a2-097e-1fbd-9b03-b4a9fc33694b

3 Oct 8 09:55:37 /usr/local/bin/goofys[745]: s3.ERROR code=The server is temporarily unavailable. Please try again later. msg=503 request=7c0cbad0-1320-1f66-af6c-a81e84b3fa58


Other errors may indicate other types of communication or outage scenarios as well, so be sure to read and evaluate all possible errors carefully.