How to attach to unprivileged podman containers?

[Ma27]

Suppose you start an unprivileged podman container like this:

podman run -it debian /bin/sh

Now, running sudo cntr attach $(podman ps | tail -n1 | awk '{ print $1 }') fails with the following error:

no suitable container found, got the following errors:
  - ProcessId: not a valid pid: `eb77cae38179`: invalid digit found in string
  - Podman: Failed to list containers. 'podman inspect --format {{.State.Running}};{{.State.Pid}} eb77cae38179' exited with exit status: 125: Error: inspecting object: no such object: "eb77cae38179"
  - Docker: Failed to list containers. 'docker inspect --format {{.State.Running}};{{.State.Pid}} eb77cae38179' exited with exit status: 125: Error: inspecting object: no such object: "eb77cae38179"
  - Nspawn: Failed to list containers. 'machinectl show --property=Leader eb77cae38179' exited with exit status: 1: Could not get path to machine: No machine 'eb77cae38179' known

This is because running podman inspect as root doesn't give you information about a container started as another user. Obviously, running cntr without sudo won't work because the user has insufficient permissions.

Another thing I tried is to obtain the PID of the process in the container and run cntr attach against it:

$ sudo cntr attach 1396643
[sudo] password for ma27:
cannot create container mountpoint /var/lib/cntr: Permission denied (os error 13)

^C
$

Is it even possible nowadays to achieve that? If yes, how?

Currently using cntr 1.5.1 from NixOS 22.11.

[Mic92]

I think the issue is that cntr is running the podman cli as root, so it does not find your container running as unprivileged user. For now you can get the pid of your container using podman inspect --format '{{.State.Pid}}' eb77cae381 and than pass the PID to cntr as you already wrote. Does creating /var/lib/cntr upfront on the host fixes the issue? Otherwise I have to have a look where it goes wrong.

[Ma27]

Does creating /var/lib/cntr upfront on the host fixes the issue?

It already exists. So, I did some digging with strace (just an excerpt):

$ sudo strace -f cntr attach 731074
[...]
setxattr("/tmp/cntr.cvLinQ/cntr-exec", "security.capability", "\1\0\0\2\0\0\4\0\0\0\0\0\0\0\0\0\0\0\0", 20, 0) = 0
openat(AT_FDCWD, "/tmp/cntr.cvLinQ/pid", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 4
write(4, "1", 1) = 1
close(4)              = 0
openat(AT_FDCWD, "/dev/fuse", O_RDWR) = 4
prlimit64(0, RLIMIT_NOFILE, {rlim_cur=1024*1024, rlim_max=1024*1024}, NULL) = 0
openat(AT_FDCWD, "/", O_RDONLY|O_CLOEXEC) = 5
mkdir("/var/lib/cntr", 0777) = -1 EEXIST (File exists)
statx(AT_FDCWD, "/var/lib/cntr", AT_STATX_SYNC_AS_STAT, STATX_ALL, {stx_mask=STATX_ALL|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFDIR|0755, stx_size=2, ...}) = 0
socketpair(AF_UNIX, SOCK_DGRAM|SOCK_CLOEXEC, 0, [6, 7]) = 0
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f4d108f2ad0) = 733260
strace: Process 733260 attached
[pid 733259] recvmsg(6,  <unfinished ...>
[...]
[pid 733260] readlink("/proc/731074/ns/uts", "uts:[4026537357]", 256) = 16
[pid 733260] readlink("/proc/self/ns/uts", "uts:[4026531838]", 256) = 16
[pid 733260] openat(AT_FDCWD, "/proc/731074/ns/uts", O_RDONLY|O_CLOEXEC) = 9
[pid 733260] readlink("/proc/731074/ns/cgroup", "cgroup:[4026537353]", 256) = 19
[pid 733260] readlink("/proc/self/ns/cgroup", "cgroup:[4026531835]", 256) = 19
[pid 733260] openat(AT_FDCWD, "/proc/731074/ns/cgroup", O_RDONLY|O_CLOEXEC) = 10
[pid 733260] readlink("/proc/731074/ns/pid", "pid:[4026537359]", 256) = 16
[pid 733260] readlink("/proc/self/ns/pid", "pid:[4026531836]", 256) = 16
[pid 733260] openat(AT_FDCWD, "/proc/731074/ns/pid", O_RDONLY|O_CLOEXEC) = 11
[pid 733260] readlink("/proc/731074/ns/net", "net:[4026537263]", 256) = 16
[pid 733260] readlink("/proc/self/ns/net", "net:[4026531840]", 256) = 16
[pid 733260] openat(AT_FDCWD, "/proc/731074/ns/net", O_RDONLY|O_CLOEXEC) = 12
[pid 733260] readlink("/proc/731074/ns/ipc", "ipc:[4026537358]", 256) = 16
[pid 733260] readlink("/proc/self/ns/ipc", "ipc:[4026531839]", 256) = 16
[pid 733260] openat(AT_FDCWD, "/proc/731074/ns/ipc", O_RDONLY|O_CLOEXEC) = 13
[pid 733260] readlink("/proc/731074/ns/user", "user:[4026537260]", 256) = 17
[pid 733260] readlink("/proc/self/ns/user", "user:[4026531837]", 256) = 17
[pid 733260] openat(AT_FDCWD, "/proc/731074/ns/user", O_RDONLY|O_CLOEXEC) = 14
[pid 733260] setns(8, 0) = 0
[pid 733260] mkdir("var/lib/cntr", 0777) = -1 EACCES (Permission denied)
[pid 733260] statx(AT_FDCWD, "var/lib/cntr", AT_STATX_SYNC_AS_STAT, STATX_ALL, 0x7fff47406e40) = -1 EACCES (Permission denied)
[pid 733260] close(8) = 0
[pid 733260] close(9) = 0
[pid 733260] close(10) = 0
[pid 733260] close(11) = 0
[pid 733260] close(12) = 0
[pid 733260] close(13) = 0
[pid 733260] close(14) = 0
[pid 733260] close(5) = 0
[pid 733260] close(3) = 0

I don't know much about how this works internally, so I can't really tell why 733260 also tries to mkdir /var/lib/cntr. Also lacking the time to actually dig into this right now (otherwise I would've tried), sorry! But perhaps this helps to narrow it down :)

[Mic92]

Not sure if it tries to setup a bogus mountpoint for bind mount and creates the directory for it. Have to re-read the source code again.