I wrote a tool recently that allows taking log files from the Cisco-managed S3 bucket, and outputs each log line as a Syslog message, with the content of the log line as a JSON-formatted message. Details of the Cisco-managed S3 bucket are here: https://docs.umbrella.com/deployment-umbrella/docs/cisco-managed-s3-bucket
Once logged onto disk, the output of this tool looks like:
This tool combines rclone, the excellent tool for accessing cloud-storage services. It mounts the S3 storage using the rclone mount command to mount a fuse-filesystem of the S3 storage, then looks through that storage for files to send to syslog.
Once it finds files, it checks in a local SQLite database to see if it has already logged out each file. If not, it opens and logs out the file.
You might ask why we would want to do this? Well we use Umbrella for DNS Security, but are trying to integrate the logs from Umbrella into our SEIM solution, and one of the best ways to integrate is to use syslog message as an intermediary.
The full code is in a gitlab repository here: https://gitlab.com/jaytuck/umbrella-logs-to-syslog . Feel free to copy and re-purpose for your own use. If you have any thoughts or run into issues, you can contact me via a method on my About Me page.