I’ve been trying to use OneDrive as my restic backend – which is supported through the rclone backend. Using this, restic can use any backend that rclone supports, which is very cool. However, it has revealed that the OneDrive API is very buggy and unreliable! I’m surprised by how many issues I’ve been getting.
I’m mostly seeing issues using the restic check command, which confirms your backup files are not corrupted. When running the same check command 4 times, 3 out of 4 I get an error that one data block is missing, however I can see it is there, the fetching of it just fails.
The check command I am using is: restic -vvv check --read-data-subset 68/112
The errors that I’m getting look like this:
rclone: 2020/07/16 23:26:16 DEBUG : pacer: low level retry 10/10 (error Get "https://charlesdarwinuni-my.sharepoint.com/personal/jay_tuckey_cdu_edu_au/_layouts/15/download.aspx?UniqueId=1c9a5ed1-06a1-4385-b501-fc84b2bd4c1f&Translate=false&tempauth=eyJ0eXAiOiJKV1QiLCJhbGciOiJub25lIn0.eyJhdWQiOiIwMDAwMDAwMy0wMDAwLTBmZjEtY2UwMC0wMDAwMDAwMDAwMDAvY2hhcmxlc2RhcndpbnVuaS1teS5zaGFyZXBvaW50LmNvbUA5ZjI0ODc2Ny04ZTFhLTQyZjMtODM2Zi1jMDkyYWI5NWZmNzAiLCJpc3MiOiIwMDAwMDAwMy0wMDAwLTBmZjEtY2UwMC0wMDAwMDAwMDAwMDAiLCJuYmYiOiIxNTk0OTA3NzQ1IiwiZXhwIjoiMTU5NDkxMTM0NSIsImVuZHBvaW50dXJsIjoidXNnZnprcWFsajhGT3EybEtUTi9VSDQwamV2QXZSb0xhVHI2STVLRWZNTT0iLCJlbmRwb2ludHVybExlbmd0aCI6IjE2MSIsImlzbG9vcGJhY2siOiJUcnVlIiwiY2lkIjoiWWpVM1kyVmxZekF0TnpFM01DMDBOakZqTFRnek9EZ3ROekF3TVRoa1ptUTRNMkl5IiwidmVyIjoiaGFzaGVkcHJvb2Z0b2tlbiIsInNpdGVpZCI6Ik1UWmtPR0kzTTJFdE5UazJOQzAwT1RKaUxUazFaREl0TVdNMk5UUTVNMlkwWmpVMiIsImFwcF9kaXNwbGF5bmFtZSI6InJjbG9uZSIsImdpdmVuX25hbWUiOiJKYXkiLCJmYW1pbHlfbmFtZSI6IlR1Y2tleSIsImFwcGlkIjoiYjE1NjY1ZDktZWRhNi00MDkyLTg1MzktMGVlYzM3NmFmZDU5IiwidGlkIjoiOWYyNDg3NjctOGUxYS00MmYzLTgzNmYtYzA5MmFiOTVmZjcwIiwidXBuIjoiamF5LnR1Y2tleUBjZHUuZWR1LmF1IiwicHVpZCI6IjEwMDM3RkZFOTM2MjY3MzUiLCJjYWNoZWtleSI6IjBoLmZ8bWVtYmVyc2hpcHwxMDAzN2ZmZTkzNjI2NzM1QGxpdmUuY29tIiwic2NwIjoibXlmaWxlcy5yZWFkIGFsbGZpbGVzLnJlYWQgbXlmaWxlcy53cml0ZSBhbGxmaWxlcy53cml0ZSBhbGxzaXRlcy5yZWFkIiwidHQiOiIyIiwidXNlUGVyc2lzdGVudENvb2tpZSI6bnVsbH0.YjJocVNUdjN1RCtHWC9VSW85Q3Y5YkdYRjZZY1hDZWRhTUtJK0NPbWRVQT0&ApiVersion=2.1": stream error: stream ID 899; CANCEL)You can see that the low-level retries has hit 10/10 on this request, so it will be rejected entirely. As far as I can tell it seems to be entirely just bugs in the OneDrive API backend. When I retry the whole check later, I will often get a data block error, but saying a different data block is missing.
Testing Tips:
# In Fish Shell
# Before running the checks you can export some environment variables to get extra info
# To set rclone to DEBUG mode - will generate a lot of logs
set -x RCLONE_LOG_LEVEL DEBUG
# To set the rclone transaction limiter (doesn't seem to help, but I tried it)
set -x RCLONE_TPSLIMIT 1
# Can use tee to capture the very long log files created
restic -vvv check --read-data-subset 68/112 2>&1 | tee ~/68_112_check_attempt_4.logI have not found a solution, apart from just moving away from OneDrive, but thought I would post so others could find this post if they have similar errors.