To backup my server I uses a tool called restic (https://restic.net/) – It’s great. I’ve recently been testing out using MEGA (https://mega.io/) as the backend storage for my backups, and to integrate that with restic, you use rclone (https://rclone.org/) as a storage driver. Sounds complicated, but it’s really not:restic --talks-to--> rclone --talks-to--> mega
However, during my testing I’ve been having issues with backups often failing, with an error about timeouts – this is the error in my logs:
running: restic backup -v /var/lib/docker/volumes/unificontroller_config
open repository
Fatal: unable to open repo at rclone:mega:/backups/restic_repo: error talking HTTP to rclone: Get "http://localhost/file-5577006791947779410": context deadline exceeded (Client.Timeout exceeded while awaiting headers)After doing some investigation, it seems that when rclone first opens a connection to mega, it needs to download+decrypt all the metadata for your account, which can take some time – this means the first request might be slow, but future requests should be faster. I looked in the restic documentation for any configuration on setting timeouts, but didn’t find anything.
“No problem!” I thought to myself “I can add a longer timeout myself! It should be possible, just need to download the code for restic, find where the requests to rclone are made, and change the settings there to make the timeout longer”
The fact that I can even download, view, and modify the source for restic makes FOSS awesome, but even so, I’ve not done any coding in the “go” language before, so this might be a small challenge. I do have experience writing a lot of Python, as well as some C/C++, so I’m not completely unfamiliar with code.
I spent about an hour reading the code for restic on how it talks to the backend (which is super well written – thanks restic team), and as I’m looking and understanding it, I discover it that the rclone timeout is actually already configurable?!? What?!?
After some more investigating, I realise that this feature has been added between the stable version of restic (where I was looking at the docs) and the in-development version of restic (the code I downloaded)! So I compiled the in-development version, do some testing, and “YES!” everything works as expected, now I can run restic like this, and set the timeout for the rclone backend:
restic --option=rclone.timeout=10m backup -v /var/lib/docker/volumes/unificontroller_configJust to really put the icing on the cake, ONE DAY LATER restic releases a new stable version, with the rclone timeout feature in it!