Release Flatcar Container Linux LTS 4081.3.1 FAILED

[widovsky]

When trying to update from LTS release 3510.3.6 to 4081.3.1 we run into an issue.

All runs fine, as expected only when the PostinstallRunnerAction is staring the update is looking for the oem-vmware.gz and can not find it. He wants to download it from update.release.flatcar-linux.net, but we have not a connection to that url. Never has been in the past either, and we had no issues in the past with Flatcar updates.

Does anyone recognise this issue, or know what's the problem?

Log:
Mar 07 14:33:19 our_server update_engine[1318952]: Args { output_dir: "/var/lib/update_engine/ue-rs/", target_filename: Some("oem-vmware.raw"), input_xml: Some("/var/lib/update_engine/prefs/full-response"), payload_url: None, pubkey_file: "/usr/share/update_engine/update-payload-key.pub.pem", image_match: ["*oem-vmware.gz"], take_first_match: true }
Mar 07 14:33:19 our_server update_engine[1318952]: mv: cannot stat '/var/lib/update_engine/ue-rs//oem-vmware.raw': No such file or directory
Mar 07 14:33:19 our_server update_engine[1318952]: Falling back to release-server for extension 'oem-vmware'
Mar 07 14:33:19 our_server update_engine[1318952]: Args { output_dir: "/var/lib/update_engine/ue-rs/", target_filename: Some("oem-vmware.raw"), input_xml: None, payload_url: Some("https://update.release.flatcar-linux.net/amd64-usr/4081.3.1/oem-vmware.gz"), pubkey_file: "/usr/share/update_engine/update-payload-key.pub.pem", image_match: [], take_first_match: false }
Mar 07 14:33:19 our_server update_engine[1318952]: Error: unable to download data(url Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("update.release.flatcar-linux.net")), port: None, path: "/amd64-usr/4081.3.1/oem-vmware.gz", query: None, fragment: None })
Mar 07 14:33:19 our_server update_engine[1318952]: Caused by:
Mar 07 14:33:19 our_server update_engine[1318952]: 0: client get & send"https://update.release.flatcar-linux.net/amd64-usr/4081.3.1/oem-vmware.gz" failed
Mar 07 14:33:19 our_server update_engine[1318952]: 1: error sending request for url (https://update.release.flatcar-linux.net/amd64-usr/4081.3.1/oem-vmware.gz): error trying to connect: tcp connect error: Network is unreachable (os error 101)
Mar 07 14:33:19 our_server update_engine[1318952]: 2: error trying to connect: tcp connect error: Network is unreachable (os error 101)
Mar 07 14:33:19 our_server update_engine[1318952]: 3: tcp connect error: Network is unreachable (os error 101)
Mar 07 14:33:19 our_server update_engine[1318952]: 4: Network is unreachable (os error 101)
Mar 07 14:33:19 our_server update_engine[1318952]: Failed to download required OEM update payload
Mar 07 14:33:19 our_server update_engine[1318952]: E0307 14:33:19.024019 1318952 postinstall_runner_action.cc:94] Postinst command failed with code: 1
Mar 07 14:33:19 our_server update_engine[1318952]: I0307 14:33:19.039006 1318952 action_processor.cc:68] ActionProcessor::ActionComplete: PostinstallRunnerAction action failed. Aborting processing.
Mar 07 14:33:19 our_server update_engine[1318952]: I0307 14:33:19.039233 1318952 action_processor.cc:73] ActionProcessor::ActionComplete: finished last action of type PostinstallRunnerAction
Mar 07 14:33:19 our_server update_engine[1318952]: I0307 14:33:19.039366 1318952 update_attempter.cc:306] Processing Done.
Mar 07 14:33:19 our_server update_engine[1318952]: E0307 14:33:19.039642 1318952 update_attempter.cc:619] Update failed.

[t-lo]

This looks like a connection error:

Mar 07 14:33:19 our_server update_engine[1318952]: 1: error sending request for url (https://update.release.flatcar-linux.net/amd64-usr/4081.3.1/oem-vmware.gz): error trying to connect: tcp connect error: Network is unreachable (os error 101)
Mar 07 14:33:19 our_server update_engine[1318952]: 3: tcp connect error: Network is unreachable (os error 101)

Is the node connected to the internet? I can download https://update.release.flatcar-linux.net/amd64-usr/4081.3.1/oem-vmware.gz from here just fine...

[jepio]

What method are you using to update @widovsky? Are you self-hosting a nebraska server? Do you have a proxy?

[widovsky]

We use an airgapped environment. We have our own Nebraska server and proxy.

The workaround mentioned in (Flatcar#1326) and Flatcar#1332), do not work for us, our Flatcar servers have no connection directly with internet.

[jepio]

Can you paste the contents of /var/lib/update_engine/prefs/full-response?

Do you use nebraska to host/mirror update packages or do you manage your own location?

You will definitely need to start mirroring oem-vmware.gz files together with the update payloads, as they are required parts of updates since some time.