Backup log display failing (Error grabbing logs: invalid character 'l' after object key:value pair)

[logan893]

Steps to reproduce

1. Running Nextcloud 9.4.0 Beta, stop all containers, run backup
2. For unknown reason, server (VM) crash after ~30 minutes, before backup completed
3. Backup Failed (maybe it caused the server restart somehow, nothing useful in journalctl logs)
4. Start backup again after server reboot
5. Backup Failed (no server restart this time)
6. Open Logs from AIO admin web interface
7. Logs only partially shown

Expected behavior

Logs should be visible in their entirety

Actual behavior

Only a few lines of log shown, then an error. The number of log lines shown varies. Sometimes 1, sometimes as much as 6.

2024-10-24T08:30:33.611291438Z Performing backup...
2024-10-24T08:30:33.611490598Z Starting the backup...
2024-10-24T08:30:34.493100585Z Killed stale lock nextcloud-aio-borgbackup.35-0.
Error grabbing logs: invalid character 'l' after object key:value pair

Other information

Host OS

Ubuntu 22.0.4 LTS

Output of sudo docker info

Client: Docker Engine - Community
 Version: 25.0.3
 Context: default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version: v0.12.1
    Path: /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version: v2.24.5
    Path: /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 13
  Running: 1
  Paused: 0
  Stopped: 12
 Images: 27
 Server Version: 25.0.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-124-generic
 Operating System: Ubuntu 22.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 3
 Total Memory: 5.783GiB
 Name: nextcloud44
 ID: e188dd7e-dacf-40b6-8ee7-2ee7bb18e9cb
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  [127.0.0.0/8](http://127.0.0.0/8)
 Live Restore Enabled: false

Docker run command or docker-compose file that you used

Other valuable info

[logan893]

Since displaying the log is not working via the admin web interface, how do I fetch the backup log to find out why the backup is failing?

[szaimen]

Hi,

Since displaying the log is not working via the admin web interface, how do I fetch the backup log to find out why the backup is failing?

sudo docker logs nextcloud-aio-borgbackup. Can you post the logs here? We would like to make displaying them more resilient if possible.

[logan893]

Hi,

Since displaying the log is not working via the admin web interface, how do I fetch the backup log to find out why the backup is failing?

sudo docker logs nextcloud-aio-borgbackup. Can you post the logs here? We would like to make displaying them more resilient if possible.

The logs from this command gives the same output as the "logs" viewed from the AIO admin web interface.
The addition is text on the line of the error:

"error from daemon in stream:", then the "Error grabbing logs: invalid character ..."

[szaimen]

Okay. Then this is a docker issue and nothing we can fix in AIO. You might want to try to do the backup again.

[logan893]

I tried to add my comments but since you were quick to change this from an issue to a discussion, I couldn't. Adding here for more details.

Okay. Then this is a docker issue and nothing we can fix in AIO. You might want to try to do the backup again.

FYI, I found I can fetch the log location via: docker inspect --format='{{.LogPath}}' nextcloud-aio-borgbackup

Looks like one log line in this file is malformed, with a new {"log": ...} entry being started before the previous one is finished. That's why the error notes the invalid character is a lower-case L. Unclear to me why this could happen; perhaps related to why the backup itself ends up failing, running out of storage space? Odd that it's able to continue logging after this, though.

The cause for the backup failing is clear from the log now that I can see it; "no space left on device." I presume the backup is trying to store all user files, including trash. And, this is then related to another issue I have with Nextcloud in general (not specifically AIO), where Nextcloud is moving deleted files from external storage, to the trash folder in the root of the user's files. The external storage is not flagged for backup, while the "local" files including the trash are. I need to remember to clean the trash folder after deleting huge files.

For now, I'll make sure to store backups on my NAS instead of the VM's local drive, to avoid running out of space for backups.