Prelude branch not building Elasticsearch #20

coreymhudson · 2025-01-06T15:50:46Z

coreymhudson
Jan 6, 2025

When building prelude branch of the conductor repo on Docker Desktop - it generates the following error:

Gracefully stopping... (press Ctrl+C again to force)
dependency failed to start: container conductor is unhealthy
make: *** [platform] Error 1 View Config   w Enable Watch

The Elasticsearch logs produce the following:

2025-01-06 09:17:20 {"type": "server", "timestamp": "2025-01-06T15:17:20,527Z", "level": "WARN", "component": "o.e.b.JNANatives", "cluster.name": "workflow.elasticsearch", "node.name": "d88844abd3d3", "message": "unable to install syscall filter: ", 
2025-01-06 09:17:20 "stacktrace": ["java.lang.UnsupportedOperationException: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed",
2025-01-06 09:17:20 "at org.elasticsearch.bootstrap.SystemCallFilter.linuxImpl(SystemCallFilter.java:347) ~[elasticsearch-7.17.1.jar:7.17.1]",
2025-01-06 09:17:20 "at org.elasticsearch.bootstrap.SystemCallFilter.init(SystemCallFilter.java:638) ~[elasticsearch-7.17.1.jar:7.17.1]",
2025-01-06 09:17:20 "at org.elasticsearch.bootstrap.JNANatives.tryInstallSystemCallFilter(JNANatives.java:255) [elasticsearch-7.17.1.jar:7.17.1]",
2025-01-06 09:17:20 "at org.elasticsearch.bootstrap.Natives.tryInstallSystemCallFilter(Natives.java:102) [elasticsearch-7.17.1.jar:7.17.1]",
2025-01-06 09:17:20 "at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:112) [elasticsearch-7.17.1.jar:7.17.1]",
2025-01-06 09:17:20 "at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:183) [elasticsearch-7.17.1.jar:7.17.1]",
2025-01-06 09:17:20 "at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) [elasticsearch-7.17.1.jar:7.17.1]",
2025-01-06 09:17:20 "at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) [elasticsearch-7.17.1.jar:7.17.1]",
2025-01-06 09:17:20 "at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) [elasticsearch-7.17.1.jar:7.17.1]",
2025-01-06 09:17:20 "at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) [elasticsearch-7.17.1.jar:7.17.1]",
2025-01-06 09:17:20 "at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) [elasticsearch-cli-7.17.1.jar:7.17.1]",
2025-01-06 09:17:20 "at org.elasticsearch.cli.Command.main(Command.java:77) [elasticsearch-cli-7.17.1.jar:7.17.1]",
2025-01-06 09:17:20 "at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) [elasticsearch-7.17.1.jar:7.17.1]",
2025-01-06 09:17:20 "at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) [elasticsearch-7.17.1.jar:7.17.1]"] }

The inspect panel produces the following:

{
	"Id": "d88844abd3d30da8a85fdf8945894cc380ab57597b2ef054b0a84ddbd88d3c03",
	"Created": "2025-01-06T15:17:02.334621797Z",
	"Path": "/bin/tini",
	"Args": [
		"--",
		"/usr/local/bin/docker-entrypoint.sh",
		"eswrapper"
	],
	"State": {
		"Status": "exited",
		"Running": false,
		"Paused": false,
		"Restarting": false,
		"OOMKilled": true,
		"Dead": false,
		"Pid": 0,
		"ExitCode": 137,
		"Error": "",
		"StartedAt": "2025-01-06T15:17:02.950963922Z",
		"FinishedAt": "2025-01-06T15:17:25.968456919Z",
		"Health": {
			"Status": "unhealthy",
			"FailingStreak": 0,
			"Log": [
				{
					"Start": "2025-01-06T15:17:08.074475258Z",
					"End": "2025-01-06T15:17:08.206250592Z",
					"ExitCode": 0,
					"Output": ""
				},
				{
					"Start": "2025-01-06T15:17:18.207842763Z",
					"End": "2025-01-06T15:17:18.568589221Z",
					"ExitCode": 0,
					"Output": ""
				}
			]
		}
	},
	"Image": "sha256:35f81ab02dda48d6f81ad9be52b25cdbd3a832e8660b48218af805d6de3ddba8",
	"ResolvConfPath": "/var/lib/docker/containers/d88844abd3d30da8a85fdf8945894cc380ab57597b2ef054b0a84ddbd88d3c03/resolv.conf",
	"HostnamePath": "/var/lib/docker/containers/d88844abd3d30da8a85fdf8945894cc380ab57597b2ef054b0a84ddbd88d3c03/hostname",
	"HostsPath": "/var/lib/docker/containers/d88844abd3d30da8a85fdf8945894cc380ab57597b2ef054b0a84ddbd88d3c03/hosts",
	"LogPath": "/var/lib/docker/containers/d88844abd3d30da8a85fdf8945894cc380ab57597b2ef054b0a84ddbd88d3c03/d88844abd3d30da8a85fdf8945894cc380ab57597b2ef054b0a84ddbd88d3c03-json.log",
	"Name": "/elasticsearch",
	"RestartCount": 0,
	"Driver": "overlayfs",
	"Platform": "linux",
	"MountLabel": "",
	"ProcessLabel": "",
	"AppArmorProfile": "",
	"ExecIDs": null,
	"HostConfig": {
		"Binds": null,
		"ContainerIDFile": "",
		"LogConfig": {
			"Type": "json-file",
			"Config": {
				"max-file": "10",
				"max-size": "50m"
			}
		},
		"NetworkMode": "conductor_conductor-network",
		"PortBindings": {
			"9200/tcp": [
				{
					"HostIp": "",
					"HostPort": "9200"
				}
			]
		},
		"RestartPolicy": {
			"Name": "no",
			"MaximumRetryCount": 0
		},
		"AutoRemove": false,
		"VolumeDriver": "",
		"VolumesFrom": null,
		"ConsoleSize": [
			0,
			0
		],
		"CapAdd": null,
		"CapDrop": null,
		"CgroupnsMode": "private",
		"Dns": null,
		"DnsOptions": null,
		"DnsSearch": null,
		"ExtraHosts": [],
		"GroupAdd": null,
		"IpcMode": "private",
		"Cgroup": "",
		"Links": null,
		"OomScoreAdj": 0,
		"PidMode": "",
		"Privileged": false,
		"PublishAllPorts": false,
		"ReadonlyRootfs": false,
		"SecurityOpt": null,
		"UTSMode": "",
		"UsernsMode": "",
		"ShmSize": 67108864,
		"Runtime": "runc",
		"Isolation": "",
		"CpuShares": 0,
		"Memory": 17179869184,
		"NanoCpus": 6000000000,
		"CgroupParent": "",
		"BlkioWeight": 0,
		"BlkioWeightDevice": null,
		"BlkioDeviceReadBps": null,
		"BlkioDeviceWriteBps": null,
		"BlkioDeviceReadIOps": null,
		"BlkioDeviceWriteIOps": null,
		"CpuPeriod": 0,
		"CpuQuota": 0,
		"CpuRealtimePeriod": 0,
		"CpuRealtimeRuntime": 0,
		"CpusetCpus": "",
		"CpusetMems": "",
		"Devices": null,
		"DeviceCgroupRules": null,
		"DeviceRequests": null,
		"MemoryReservation": 8589934592,
		"MemorySwap": 34359738368,
		"MemorySwappiness": null,
		"OomKillDisable": null,
		"PidsLimit": null,
		"Ulimits": [
			{
				"Name": "memlock",
				"Hard": -1,
				"Soft": -1
			},
			{
				"Name": "nofile",
				"Hard": 65536,
				"Soft": 65536
			}
		],
		"CpuCount": 0,
		"CpuPercent": 0,
		"IOMaximumIOps": 0,
		"IOMaximumBandwidth": 0,
		"Mounts": [
			{
				"Type": "volume",
				"Source": "conductor_elasticsearch-data",
				"Target": "/usr/share/elasticsearch/data",
				"VolumeOptions": {}
			},
			{
				"Type": "volume",
				"Source": "conductor_elasticsearch-logs",
				"Target": "/usr/share/elasticsearch/logs",
				"VolumeOptions": {}
			}
		],
		"MaskedPaths": [
			"/proc/asound",
			"/proc/acpi",
			"/proc/kcore",
			"/proc/keys",
			"/proc/latency_stats",
			"/proc/timer_list",
			"/proc/timer_stats",
			"/proc/sched_debug",
			"/proc/scsi",
			"/sys/firmware",
			"/sys/devices/virtual/powercap"
		],
		"ReadonlyPaths": [
			"/proc/bus",
			"/proc/fs",
			"/proc/irq",
			"/proc/sys",
			"/proc/sysrq-trigger"
		]
	},
	"GraphDriver": {
		"Data": null,
		"Name": "overlayfs"
	},
	"Mounts": [
		{
			"Type": "volume",
			"Name": "conductor_elasticsearch-data",
			"Source": "/var/lib/docker/volumes/conductor_elasticsearch-data/_data",
			"Destination": "/usr/share/elasticsearch/data",
			"Driver": "local",
			"Mode": "z",
			"RW": true,
			"Propagation": ""
		},
		{
			"Type": "volume",
			"Name": "conductor_elasticsearch-logs",
			"Source": "/var/lib/docker/volumes/conductor_elasticsearch-logs/_data",
			"Destination": "/usr/share/elasticsearch/logs",
			"Driver": "local",
			"Mode": "z",
			"RW": true,
			"Propagation": ""
		}
	],
	"Config": {
		"Hostname": "d88844abd3d3",
		"Domainname": "",
		"User": "",
		"AttachStdin": false,
		"AttachStdout": true,
		"AttachStderr": true,
		"ExposedPorts": {
			"9200/tcp": {},
			"9300/tcp": {}
		},
		"Tty": false,
		"OpenStdin": false,
		"StdinOnce": false,
		"Env": [
			"xpack.security.enabled=true",
			"PGID=100",
			"cluster.name=workflow.elasticsearch",
			"ELASTIC_PASSWORD=myelasticpassword",
			"MANAGE_INDEX_TEMPLATES=true",
			"discovery.type=single-node",
			"bootstrap.memory_lock=true",
			"NETWORK_HOST=http://localhost:9200",
			"PUID=3381",
			"ES_JAVA_OPTS=-Xms8g -Xmx8g",
			"PATH=/usr/share/elasticsearch/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
			"ELASTIC_CONTAINER=true"
		],
		"Cmd": [
			"eswrapper"
		],
		"Healthcheck": {
			"Test": [
				"CMD-SHELL",
				"curl --silent --fail localhost:9200/_cluster/health?wait_for_status=yellow&timeout=50s || exit 1"
			],
			"Interval": 10000000000,
			"Timeout": 10000000000,
			"StartPeriod": 25000000000,
			"Retries": 5
		},
		"Image": "docker.elastic.co/elasticsearch/elasticsearch:7.17.1",
		"Volumes": null,
		"WorkingDir": "/usr/share/elasticsearch",
		"Entrypoint": [
			"/bin/tini",
			"--",
			"/usr/local/bin/docker-entrypoint.sh"
		],
		"OnBuild": null,
		"Labels": {
			"com.docker.compose.config-hash": "599fd7af34384a8dd3620e82b5420f08b21f13c533d999a583f54431bed26a04",
			"com.docker.compose.container-number": "1",
			"com.docker.compose.depends_on": "",
			"com.docker.compose.image": "sha256:35f81ab02dda48d6f81ad9be52b25cdbd3a832e8660b48218af805d6de3ddba8",
			"com.docker.compose.oneoff": "False",
			"com.docker.compose.project": "conductor",
			"com.docker.compose.project.config_files": "/Users/coreyhudson/Desktop/Building Data Pipeline/conductor/docker-compose.yml",
			"com.docker.compose.project.working_dir": "/Users/coreyhudson/Desktop/Building Data Pipeline/conductor",
			"com.docker.compose.service": "elasticsearch",
			"com.docker.compose.version": "2.31.0",
			"org.label-schema.build-date": "2022-02-24T09:29:39.346656497Z",
			"org.label-schema.license": "Elastic-License-2.0",
			"org.label-schema.name": "Elasticsearch",
			"org.label-schema.schema-version": "1.0",
			"org.label-schema.url": "https://www.elastic.co/products/elasticsearch",
			"org.label-schema.usage": "https://www.elastic.co/guide/en/elasticsearch/reference/index.html",
			"org.label-schema.vcs-ref": "e5acb99f822233d62d6444ce45a4543dc1c8059a",
			"org.label-schema.vcs-url": "https://github.com/elastic/elasticsearch",
			"org.label-schema.vendor": "Elastic",
			"org.label-schema.version": "7.17.1",
			"org.opencontainers.image.created": "2022-02-24T09:29:39.346656497Z",
			"org.opencontainers.image.documentation": "https://www.elastic.co/guide/en/elasticsearch/reference/index.html",
			"org.opencontainers.image.licenses": "Elastic-License-2.0",
			"org.opencontainers.image.revision": "e5acb99f822233d62d6444ce45a4543dc1c8059a",
			"org.opencontainers.image.source": "https://github.com/elastic/elasticsearch",
			"org.opencontainers.image.title": "Elasticsearch",
			"org.opencontainers.image.url": "https://www.elastic.co/products/elasticsearch",
			"org.opencontainers.image.vendor": "Elastic",
			"org.opencontainers.image.version": "7.17.1"
		}
	},
	"NetworkSettings": {
		"Bridge": "",
		"SandboxID": "",
		"SandboxKey": "",
		"Ports": {},
		"HairpinMode": false,
		"LinkLocalIPv6Address": "",
		"LinkLocalIPv6PrefixLen": 0,
		"SecondaryIPAddresses": null,
		"SecondaryIPv6Addresses": null,
		"EndpointID": "",
		"Gateway": "",
		"GlobalIPv6Address": "",
		"GlobalIPv6PrefixLen": 0,
		"IPAddress": "",
		"IPPrefixLen": 0,
		"IPv6Gateway": "",
		"MacAddress": "",
		"Networks": {
			"conductor_conductor-network": {
				"IPAMConfig": null,
				"Links": null,
				"Aliases": [
					"elasticsearch",
					"elasticsearch"
				],
				"MacAddress": "",
				"DriverOpts": null,
				"NetworkID": "c04ad3505161b27d2031dd2d8d17c9c69d6d1309c503ca504cbcaac25daa1d55",
				"EndpointID": "",
				"Gateway": "",
				"IPAddress": "",
				"IPPrefixLen": 0,
				"IPv6Gateway": "",
				"GlobalIPv6Address": "",
				"GlobalIPv6PrefixLen": 0,
				"DNSNames": [
					"elasticsearch",
					"d88844abd3d3"
				]
			}
		}
	}
}

This stops things immediately after the "Spinning up the Prelude Development Portal" message is triggered.

Answered by MitchellShiell

Jan 6, 2025

Hey Corey, thanks for the detailed log, I was unable to reproduce the issue however it looks like Elasticsearch is running out of memory when trying to deploy, you can see this from the state objected in the ES log:

"OOMKilled": true, ..."ExitCode": 137, where OOM is Out of Memory and 137 is saying docker does not have enough RAM

Two things I'd like you to try/confirm:

Make sure docker has enough resources allocated to, I'll copy paste the recommended configs below

Minimum CPU: 8 cores
Memory: 8 GB
Swap: 2 GB
Virtual disk: 64 GB

Adjust these in Docker Desktop settings under "Resources". If you already have these configured appropriately or are still having issues then try bumping up t…

View full answer

MitchellShiell · 2025-01-06T16:13:54Z

MitchellShiell
Jan 6, 2025
Collaborator

Hey Corey, thanks for the detailed log, I was unable to reproduce the issue however it looks like Elasticsearch is running out of memory when trying to deploy, you can see this from the state objected in the ES log:

"OOMKilled": true, ..."ExitCode": 137, where OOM is Out of Memory and 137 is saying docker does not have enough RAM

Two things I'd like you to try/confirm:

Make sure docker has enough resources allocated to, I'll copy paste the recommended configs below

Minimum CPU: 8 cores
Memory: 8 GB
Swap: 2 GB
Virtual disk: 64 GB

Adjust these in Docker Desktop settings under "Resources". If you already have these configured appropriately or are still having issues then try bumping up these values, I personally am running a cpu limit of 8, memory limit of 16gb, swap of 4gb and virtual disk limit of 192 GB (likely overkill).

If this fails we can try lowering the JVM heap size for elasticsearch from 8gb to 4gb in the docker-compose (config linked here)

Let me know if this works! if not I'll ask some more questions to troubleshoot further.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prelude branch not building Elasticsearch #20

{{title}}

Replies: 1 comment

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Prelude branch not building Elasticsearch #20

coreymhudson Jan 6, 2025

Replies: 1 comment

MitchellShiell Jan 6, 2025 Collaborator

coreymhudson
Jan 6, 2025

MitchellShiell
Jan 6, 2025
Collaborator