Sendkey API call randomly ending in EOF failing packer build

[newedgex]

https://github.com/Telmate/proxmox-api-go/blob/master/proxmox/client.go#L411

About the environment :

• Server side : This has been verified on Proxmox v6, v7 and the latest 8.0.4 , and also with different versions of packer. I always connect with https through a valid letsencrypt certificate, the connection is also encapsulated through wireguard VPN.
• Client side : This happened with different versions of packer and also packer-plugin-proxmox , and it also happens with different templates wherever I use the "Typing the boot command" which relies on a PUT request to the Sendkey API endpoint.
  What happens :
  short log :

2023/10/12 10:12:35 packer-plugin-proxmox_v1.1.5_x5.0_linux_amd64 plugin: 2023/10/12 10:12:35 [INFO] Waiting 2s
2023/10/12 10:12:37 packer-plugin-proxmox_v1.1.5_x5.0_linux_amd64 plugin: 2023/10/12 10:12:37 [INFO] Waiting 2s
2023/10/12 10:12:44 packer-plugin-proxmox_v1.1.5_x5.0_linux_amd64 plugin: 2023/10/12 10:12:44 [INFO] Waiting 5s
==> proxmox-iso.ubuntu-server: Error running boot command: Put "https://proxmox-02-01.upstream.domain.com:8006/api2/json/nodes/proxmox-02-01/qemu/102/sendkey": EOF
==> proxmox-iso.ubuntu-server: Stopping VM
==> proxmox-iso.ubuntu-server: Deleting VM

long log :

2023/10/12 10:12:06 packer-plugin-proxmox_v1.1.5_x5.0_linux_amd64 plugin: 2023/10/12 10:12:06 Plugin address: unix /tmp/packer-plugin4053739862
2023/10/12 10:12:06 packer-plugin-proxmox_v1.1.5_x5.0_linux_amd64 plugin: 2023/10/12 10:12:06 Waiting for connection...
2023/10/12 10:12:06 packer-plugin-proxmox_v1.1.5_x5.0_linux_amd64 plugin: 2023/10/12 10:12:06 Serving a plugin connection...
2023/10/12 10:12:06 packer-plugin-proxmox_v1.1.5_x5.0_linux_amd64 plugin: 2023/10/12 10:12:06 [TRACE] starting builder iso
2023/10/12 10:12:06 packer-plugin-proxmox_v1.1.5_x5.0_linux_amd64 plugin: 2023/10/12 10:12:06 Number of sockets 0 is too small, using default: 1
2023/10/12 10:12:06 packer-plugin-proxmox_v1.1.5_x5.0_linux_amd64 plugin: 2023/10/12 10:12:06 OS not set, using default 'other'
2023/10/12 10:12:06 packer-plugin-proxmox_v1.1.5_x5.0_linux_amd64 plugin: 2023/10/12 10:12:06 Disk 0 cache mode not set, using default 'none'
2023/10/12 10:12:07 Build debug mode: false
2023/10/12 10:12:07 Force build: false
2023/10/12 10:12:07 On error: 
2023/10/12 10:12:07 Waiting on builds to complete...
2023/10/12 10:12:07 Starting build run: proxmox-iso.ubuntu-server
proxmox-iso.ubuntu-server: output will be in this color.
2023/10/12 10:12:07 Running builder: 
2023/10/12 10:12:07 [INFO] (telemetry) Starting builder proxmox-iso.ubuntu-server

2023/10/12 10:12:07 packer-plugin-proxmox_v1.1.5_x5.0_linux_amd64 plugin: 2023/10/12 10:12:07 using token auth
2023/10/12 10:12:07 packer-plugin-proxmox_v1.1.5_x5.0_linux_amd64 plugin: 2023/10/12 10:12:07 No URLs were provided to Step Download. Continuing...
==> proxmox-iso.ubuntu-server: Creating VM
==> proxmox-iso.ubuntu-server: No VM ID given, getting next free from Proxmox
==> proxmox-iso.ubuntu-server: Starting VM
==> proxmox-iso.ubuntu-server: Waiting 20s for boot
==> proxmox-iso.ubuntu-server: Typing the boot command
2023/10/12 10:12:35 packer-plugin-proxmox_v1.1.5_x5.0_linux_amd64 plugin: 2023/10/12 10:12:35 [INFO] Waiting 2s
2023/10/12 10:12:37 packer-plugin-proxmox_v1.1.5_x5.0_linux_amd64 plugin: 2023/10/12 10:12:37 [INFO] Waiting 2s
2023/10/12 10:12:44 packer-plugin-proxmox_v1.1.5_x5.0_linux_amd64 plugin: 2023/10/12 10:12:44 [INFO] Waiting 5s
==> proxmox-iso.ubuntu-server: Error running boot command: Put "https://proxmox-02-01.upstream.domain.com:8006/api2/json/nodes/proxmox-02-01/qemu/102/sendkey": EOF
==> proxmox-iso.ubuntu-server: Stopping VM
==> proxmox-iso.ubuntu-server: Deleting VM
2023/10/12 10:12:55 [INFO] (telemetry) ending proxmox-iso.ubuntu-server
==> Wait completed after 48 seconds 224 milliseconds
2023/10/12 10:12:55 machine readable: error-count []string{"1"}
==> Some builds didn't complete successfully and had errors:
2023/10/12 10:12:55 machine readable: proxmox-iso.ubuntu-server,error []string{"Error running boot command: Put \"https://proxmox-02-01.upstream.domain.com:8006/api2/json/nodes/proxmox-02-01/qemu/102/sendkey\": EOF"}
==> Builds finished but no artifacts were created.
2023/10/12 10:12:55 [INFO] (telemetry) Finalizing.
Build 'proxmox-iso.ubuntu-server' errored after 48 seconds 224 milliseconds: Error running boot command: Put "https://proxmox-02-01.upstream.domain.com:8006/api2/json/nodes/proxmox-02-01/qemu/102/sendkey": EOF

==> Wait completed after 48 seconds 224 milliseconds

==> Some builds didn't complete successfully and had errors:
--> proxmox-iso.ubuntu-server: Error running boot command: Put "https://proxmox-02-01.upstream.domain.com:8006/api2/json/nodes/proxmox-02-01/qemu/102/sendkey": EOF

==> Builds finished but no artifacts were created.
2023/10/12 10:12:56 waiting for all plugin processes to complete...
2023/10/12 10:12:56 /home/user/.config/packer/plugins/github.com/hashicorp/proxmox/packer-plugin-proxmox_v1.1.5_x5.0_linux_amd64: plugin process exited

Since EOF usually happens due to connectivity issues, I'm wondering what is actually happening since both my and proxmox's connections are fine, and I'm the only one using it so there's no network saturation.

I have reported the issue also on :

• Random EOF Error while running boot command on sendkey API hashicorp/packer-plugin-proxmox#237
• https://forum.proxmox.com/threads/api-error-running-boot-command-sendkey-eof.134796/

[stmcginnis]

Same error here. Using packer 1.9.5, running commands from the latest CAPI image builder.

PACKER_FLAGS="--var 'kubernetes_rpm_version=1.28.3' --var 'kubernetes_semver=v1.28.3' --var 'kubernetes_series=v1.28' --var 'kubernetes_deb_version=1.28.3-1.1'" make build-proxmox-ubuntu-2204

# pveversion
pve-manager/8.1.4/ec5affc9e41f1d79 (running kernel: 6.5.11-7-pve)