Generate production packages with Wazuh changes and config files #4

AlexRuiz7 · 2023-08-02T15:49:24Z

Description

We need to generate production packages of the indexer with our changes and default configuration in it.

These packages will contain our security configuration.

Tasks

Generate tarball
Generate Docker image
Generate DEB & RPM packages

Related issues

The text was updated successfully, but these errors were encountered:

AlexRuiz7 · 2023-08-07T09:51:29Z

The necessary tooling to build packages is located under the distribution folder, which is a Gradle subproject.

https://github.com/wazuh/wazuh-indexer/tree/4.8.0/distribution

I'll continue investigating how to use these tools.

AlexRuiz7 · 2023-08-07T13:23:48Z

Successfully run OpenSearch locally:

Details

./gradlew run

Welcome to Gradle 8.2.1!

Here are the highlights of this release:
 - Kotlin DSL: new reference documentation, assignment syntax by default
 - Kotlin DSL is now the default with Gradle init
 - Improved suggestions to resolve errors in console output
 - Reduced sync memory consumption

For more details see https://docs.gradle.org/8.2.1/release-notes.html

=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 8.2.1
  OS Info               : Linux 5.19.0-50-generic (amd64)
  JDK Version           : 11 (Ubuntu JDK)
  JAVA_HOME             : /usr/lib/jvm/java-11-openjdk-amd64
  Random Testing Seed   : B998A4CA8A75E61
  In FIPS 140 mode      : false
=======================================

> Task :libs:opensearch-common:compileJava
Note: /wazuh-indexer/libs/common/src/main/java/org/opensearch/common/util/set/Sets.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :libs:opensearch-core:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :libs:opensearch-x-content:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :server:compileJava
Note: Processing Log4j annotations
Note: Annotations processed
Note: Processing Log4j annotations
Note: No elements to process
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :modules:systemd:compileJava
Note: /wazuh-indexer/modules/systemd/src/main/java/org/opensearch/systemd/SystemdPlugin.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :modules:aggs-matrix-stats:compileJava
Note: /wazuh-indexer/modules/aggs-matrix-stats/src/main/java/org/opensearch/search/aggregations/matrix/MatrixAggregationPlugin.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :modules:parent-join:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :modules:lang-expression:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :modules:percolator:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :modules:lang-mustache:compileJava
Note: /wazuh-indexer/modules/lang-mustache/src/main/java/org/opensearch/script/mustache/MustacheScriptEngine.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :modules:geo:compileJava
Note: /wazuh-indexer/modules/geo/src/main/java/org/opensearch/geo/GeoModulePlugin.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :modules:transport-netty4:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :modules:reindex:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :modules:lang-painless:compileJava
Note: /wazuh-indexer/modules/lang-painless/src/main/java/org/opensearch/painless/antlr/Walker.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :modules:analysis-common:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :run
[2023-08-07T13:00:47.166255Z] [BUILD] Copying additional config files from distro [/wazuh-indexer/distribution/archives/linux-tar/build/install/opensearch-2.10.0-SNAPSHOT/config/jvm.options.d, /wazuh-indexer/distribution/archives/linux-tar/build/install/opensearch-2.10.0-SNAPSHOT/config/opensearch.prod.yml, /wazuh-indexer/distribution/archives/linux-tar/build/install/opensearch-2.10.0-SNAPSHOT/config/jvm.prod.options, /wazuh-indexer/distribution/archives/linux-tar/build/install/opensearch-2.10.0-SNAPSHOT/config/security, /wazuh-indexer/distribution/archives/linux-tar/build/install/opensearch-2.10.0-SNAPSHOT/config/log4j2.properties, /wazuh-indexer/distribution/archives/linux-tar/build/install/opensearch-2.10.0-SNAPSHOT/config/opensearch.yml, /wazuh-indexer/distribution/archives/linux-tar/build/install/opensearch-2.10.0-SNAPSHOT/config/jvm.options]
[2023-08-07T13:00:47.167671Z] [BUILD] Creating opensearch keystore with password set to []
[2023-08-07T13:00:47.777952Z] [BUILD] Starting OpenSearch process
[2023-08-07T15:00:49,094][INFO ][o.o.n.Node               ] [runTask-0] version[2.10.0-SNAPSHOT], pid[63392], build[tar/0fcdd5647c613fd8e653eb68a3086dcaea314e06/2023-08-07T12:56:56.904455Z], OS[Linux/5.19.0-50-generic/amd64], JVM[Ubuntu/OpenJDK 64-Bit Server VM/11.0.20/11.0.20+8-post-Ubuntu-1ubuntu122.04]
[2023-08-07T15:00:49,095][INFO ][o.o.n.Node               ] [runTask-0] JVM home [/usr/lib/jvm/java-11-openjdk-amd64], using bundled JDK [false]
[2023-08-07T15:00:49,095][INFO ][o.o.n.Node               ] [runTask-0] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/wazuh-indexer/build/testclusters/runTask-0/tmp, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=logs, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -Xms512m, -Xmx512m, -ea, -esa, -XX:TieredStopAtLevel=1, -XX:ReservedCodeCacheSize=64m, -XX:MaxDirectMemorySize=268435456, -Dopensearch.path.home=/wazuh-indexer/distribution/archives/linux-tar/build/install/opensearch-2.10.0-SNAPSHOT, -Dopensearch.path.conf=/wazuh-indexer/build/testclusters/runTask-0/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
[2023-08-07T15:00:49,096][WARN ][o.o.n.Node               ] [runTask-0] version [2.10.0-SNAPSHOT] is a pre-release version of OpenSearch and is not suitable for production
[2023-08-07T15:00:49,736][INFO ][o.o.i.r.ReindexPlugin    ] [runTask-0] ReindexPlugin reloadSPI called
[2023-08-07T15:00:49,737][INFO ][o.o.i.r.ReindexPlugin    ] [runTask-0] Unable to find any implementation for RemoteReindexExtension
[2023-08-07T15:00:49,743][INFO ][o.o.p.PluginsService     ] [runTask-0] loaded module [aggs-matrix-stats]
[2023-08-07T15:00:49,743][INFO ][o.o.p.PluginsService     ] [runTask-0] loaded module [analysis-common]
[2023-08-07T15:00:49,744][INFO ][o.o.p.PluginsService     ] [runTask-0] loaded module [geo]
[2023-08-07T15:00:49,744][INFO ][o.o.p.PluginsService     ] [runTask-0] loaded module [ingest-common]
[2023-08-07T15:00:49,744][INFO ][o.o.p.PluginsService     ] [runTask-0] loaded module [ingest-geoip]
[2023-08-07T15:00:49,744][INFO ][o.o.p.PluginsService     ] [runTask-0] loaded module [ingest-user-agent]
[2023-08-07T15:00:49,744][INFO ][o.o.p.PluginsService     ] [runTask-0] loaded module [lang-expression]
[2023-08-07T15:00:49,744][INFO ][o.o.p.PluginsService     ] [runTask-0] loaded module [lang-mustache]
[2023-08-07T15:00:49,744][INFO ][o.o.p.PluginsService     ] [runTask-0] loaded module [lang-painless]
[2023-08-07T15:00:49,744][INFO ][o.o.p.PluginsService     ] [runTask-0] loaded module [mapper-extras]
[2023-08-07T15:00:49,744][INFO ][o.o.p.PluginsService     ] [runTask-0] loaded module [opensearch-dashboards]
[2023-08-07T15:00:49,745][INFO ][o.o.p.PluginsService     ] [runTask-0] loaded module [parent-join]
[2023-08-07T15:00:49,745][INFO ][o.o.p.PluginsService     ] [runTask-0] loaded module [percolator]
[2023-08-07T15:00:49,745][INFO ][o.o.p.PluginsService     ] [runTask-0] loaded module [rank-eval]
[2023-08-07T15:00:49,745][INFO ][o.o.p.PluginsService     ] [runTask-0] loaded module [reindex]
[2023-08-07T15:00:49,745][INFO ][o.o.p.PluginsService     ] [runTask-0] loaded module [repository-url]
[2023-08-07T15:00:49,745][INFO ][o.o.p.PluginsService     ] [runTask-0] loaded module [search-pipeline-common]
[2023-08-07T15:00:49,745][INFO ][o.o.p.PluginsService     ] [runTask-0] loaded module [systemd]
[2023-08-07T15:00:49,745][INFO ][o.o.p.PluginsService     ] [runTask-0] loaded module [test-delayed-aggs]
[2023-08-07T15:00:49,745][INFO ][o.o.p.PluginsService     ] [runTask-0] loaded module [transport-netty4]
[2023-08-07T15:00:49,746][INFO ][o.o.p.PluginsService     ] [runTask-0] no plugins loaded
[2023-08-07T15:00:49,755][INFO ][o.o.e.ExtensionsManager  ] [runTask-0] ExtensionsManager initialized
[2023-08-07T15:00:49,767][INFO ][o.o.e.NodeEnvironment    ] [runTask-0] using [1] data paths, mounts [[/ (/dev/nvme0n1p6)]], net usable_space [209.7gb], net total_space [366.3gb], types [ext4]
[2023-08-07T15:00:49,768][INFO ][o.o.e.NodeEnvironment    ] [runTask-0] heap size [512mb], compressed ordinary object pointers [true]
[2023-08-07T15:00:49,784][INFO ][o.o.n.Node               ] [runTask-0] node name [runTask-0], node ID [avktelhUT36J9QzyRBZaiA], cluster name [runTask], roles [ingest, remote_cluster_client, data, cluster_manager]
[2023-08-07T15:00:51,348][INFO ][o.o.t.NettyAllocator     ] [runTask-0] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=512mb}]
[2023-08-07T15:00:51,397][INFO ][o.o.d.DiscoveryModule    ] [runTask-0] using discovery type [zen] and seed hosts providers [settings, file]
[2023-08-07T15:00:51,559][WARN ][o.o.g.DanglingIndicesState] [runTask-0] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2023-08-07T15:00:51,690][INFO ][o.o.n.Node               ] [runTask-0] initialized
[2023-08-07T15:00:51,690][INFO ][o.o.n.Node               ] [runTask-0] starting ...
[2023-08-07T15:00:51,789][INFO ][o.o.t.TransportService   ] [runTask-0] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2023-08-07T15:00:51,790][INFO ][o.o.t.TransportService   ] [runTask-0] Remote clusters initialized successfully.
[2023-08-07T15:00:51,955][DEBUG][o.o.c.c.Coordinator      ] [runTask-0] startInitialJoin: coordinator becoming CANDIDATE in term 0 (was null, lastKnownLeader was [Optional.empty])
[2023-08-07T15:00:51,958][INFO ][o.o.c.c.Coordinator      ] [runTask-0] setting initial configuration to VotingConfiguration{avktelhUT36J9QzyRBZaiA}
[2023-08-07T15:00:51,959][WARN ][o.o.d.FileBasedSeedHostsProvider] [runTask-0] expected, but did not find, a dynamic hosts list at [/wazuh-indexer/build/testclusters/runTask-0/config/unicast_hosts.txt]
[2023-08-07T15:00:51,992][DEBUG][o.o.c.c.ElectionSchedulerFactory] [runTask-0] scheduling scheduleNextElection{gracePeriod=0s, thisAttempt=0, maxDelayMillis=100, delayMillis=10, ElectionScheduler{attempt=1, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2023-08-07T15:00:52,002][INFO ][o.o.h.AbstractHttpServerTransport] [runTask-0] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2023-08-07T15:00:52,003][INFO ][o.o.n.Node               ] [runTask-0] started
[2023-08-07T15:00:52,003][DEBUG][o.o.c.c.ElectionSchedulerFactory] [runTask-0] scheduleNextElection{gracePeriod=0s, thisAttempt=0, maxDelayMillis=100, delayMillis=10, ElectionScheduler{attempt=1, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2023-08-07T15:00:52,004][DEBUG][o.o.c.c.ElectionSchedulerFactory] [runTask-0] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=1, maxDelayMillis=200, delayMillis=563, ElectionScheduler{attempt=2, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2023-08-07T15:00:52,007][DEBUG][o.o.c.c.PreVoteCollector ] [runTask-0] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}, currentTerm=0}, isClosed=false} requesting pre-votes from [{runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}]
[2023-08-07T15:00:52,012][DEBUG][o.o.c.c.PreVoteCollector ] [runTask-0] PreVotingRound{preVotesReceived={{runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}=PreVoteResponse{currentTerm=0, lastAcceptedTerm=0, lastAcceptedVersion=0}}, electionStarted=true, preVoteRequest=PreVoteRequest{sourceNode={runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}, currentTerm=0}, isClosed=false} added PreVoteResponse{currentTerm=0, lastAcceptedTerm=0, lastAcceptedVersion=0} from {runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}, starting election
[2023-08-07T15:00:52,014][DEBUG][o.o.c.c.Coordinator      ] [runTask-0] starting election with StartJoinRequest{term=1,node={runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}}
[2023-08-07T15:00:52,014][DEBUG][o.o.c.c.Coordinator      ] [runTask-0] joinLeaderInTerm: for [{runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}] with term 1
[2023-08-07T15:00:52,014][DEBUG][o.o.c.c.CoordinationState] [runTask-0] handleStartJoin: leaving term [0] due to StartJoinRequest{term=1,node={runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}}
[2023-08-07T15:00:52,027][DEBUG][o.o.c.c.JoinHelper       ] [runTask-0] attempting to join {runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true} with JoinRequest{sourceNode={runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}, minimumTerm=0, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}, targetNode={runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}}]}
[2023-08-07T15:00:52,027][DEBUG][o.o.c.c.JoinHelper       ] [runTask-0] successful response to StartJoinRequest{term=1,node={runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}} from {runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}
[2023-08-07T15:00:52,028][DEBUG][o.o.c.c.CoordinationState] [runTask-0] handleJoin: added join Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}, targetNode={runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}} from [{runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}] for election, electionWon=true lastAcceptedTerm=0 lastAcceptedVersion=0
[2023-08-07T15:00:52,028][DEBUG][o.o.c.c.CoordinationState] [runTask-0] handleJoin: election won in term [1] with VoteCollection{votes=[avktelhUT36J9QzyRBZaiA], joins=[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}, targetNode={runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}}]}
[2023-08-07T15:00:52,028][DEBUG][o.o.c.c.Coordinator      ] [runTask-0] handleJoinRequest: coordinator becoming LEADER in term 1 (was CANDIDATE, lastKnownLeader was [Optional.empty])
[2023-08-07T15:00:52,039][DEBUG][o.o.c.c.JoinHelper       ] [runTask-0] received a join request for an existing node [{runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}]
[2023-08-07T15:00:52,041][INFO ][o.o.c.s.MasterService    ] [runTask-0] elected-as-cluster-manager ([1] nodes joined)[{runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true} elect leader, _BECOME_CLUSTER_MANAGER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, delta: cluster-manager node changed {previous [], current [{runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}]}
[2023-08-07T15:00:52,053][DEBUG][o.o.c.c.PublicationTransportHandler] [runTask-0] received full cluster state version [1] with size [329]
[2023-08-07T15:00:52,072][INFO ][o.o.c.c.CoordinationState] [runTask-0] cluster UUID set to [DXmrrJsuQH-F9z4FgDmppg]
[2023-08-07T15:00:52,099][INFO ][o.o.c.s.ClusterApplierService] [runTask-0] cluster-manager node changed {previous [], current [{runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}]}, term: 1, version: 1, reason: Publication{term=1, version=1}
[2023-08-07T15:00:52,109][DEBUG][o.o.c.c.C.CoordinatorPublication] [runTask-0] publication ended successfully: Publication{term=1, version=1}
[2023-08-07T15:00:52,113][DEBUG][o.o.c.c.JoinHelper       ] [runTask-0] successfully joined {runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true} with JoinRequest{sourceNode={runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}, minimumTerm=0, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}, targetNode={runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}}]}
[2023-08-07T15:00:52,113][INFO ][o.o.d.PeerFinder         ] [runTask-0] setting findPeersInterval to [1s] as node commission status = [true] for local node [{runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}]
[2023-08-07T15:00:52,122][DEBUG][o.o.c.c.PublicationTransportHandler] [runTask-0] received full cluster state version [2] with size [282]
[2023-08-07T15:00:52,131][DEBUG][o.o.c.c.C.CoordinatorPublication] [runTask-0] publication ended successfully: Publication{term=1, version=2}
[2023-08-07T15:00:52,132][INFO ][o.o.g.GatewayService     ] [runTask-0] recovered [0] indices into cluster_state
[2023-08-07T15:00:52,568][DEBUG][o.o.c.c.ElectionSchedulerFactory] [runTask-0] scheduleNextElection{gracePeriod=500ms, thisAttempt=1, maxDelayMillis=200, delayMillis=563, ElectionScheduler{attempt=2, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} not starting election
<============-> 99% EXECUTING [24m 32s]
> IDLE
> :run
> IDLE
> IDLE
> IDLE
> IDLE
> IDLE
> IDLE

API up and running

{
  "name" : "runTask-0",
  "cluster_name" : "runTask",
  "cluster_uuid" : "DXmrrJsuQH-F9z4FgDmppg",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.10.0-SNAPSHOT",
    "build_type" : "tar",
    "build_hash" : "0fcdd5647c613fd8e653eb68a3086dcaea314e06",
    "build_date" : "2023-08-07T12:56:56.904455Z",
    "build_snapshot" : true,
    "lucene_version" : "9.7.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

AlexRuiz7 · 2023-08-07T14:04:54Z

Successfully generated distribution packages from branch 4.8.0.

The following distributions are being built:
* Archives (`*.zip`, `*.tar`): these form the basis for all other OpenSearch distributions
* Packages (`*.deb`, `*.rpm`): specific package formats for some Linux distributions
* Docker images
* Backwards compatibility tests: used internally for version compatibility testing, not for public consumption

Details

./gradlew assemble   
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 8.2.1
  OS Info               : Linux 5.19.0-50-generic (amd64)
  JDK Version           : 11 (Ubuntu JDK)
  JAVA_HOME             : /usr/lib/jvm/java-11-openjdk-amd64
  Random Testing Seed   : D3FDD69024240051
  In FIPS 140 mode      : false
=======================================

> Task :example-plugins:custom-suggester:compileJava
Note: /home/alex/wazuh/wazuh-indexer/plugins/examples/custom-suggester/src/main/java/org/opensearch/example/customsuggester/CustomSuggestion.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :client:rest-high-level:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :benchmarks:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :build-tools:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :plugins:analysis-icu:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :plugins:discovery-ec2:compileJava
Note: /home/alex/wazuh/wazuh-indexer/plugins/discovery-ec2/src/main/java/org/opensearch/discovery/ec2/Ec2DiscoveryPlugin.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :plugins:transport-nio:compileJava
Note: /home/alex/wazuh/wazuh-indexer/plugins/transport-nio/src/main/java/org/opensearch/http/nio/HttpReadWriteHandler.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :plugins:repository-azure:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :build-tools:compileGroovy
Note: /home/alex/wazuh/wazuh-indexer/buildSrc/src/main/groovy/org/opensearch/gradle/test/TestWithSslPlugin.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :test:framework:compileJava
Note: Processing Log4j annotations
Note: Annotations processed
Note: Processing Log4j annotations
Note: No elements to process
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :qa:full-cluster-restart:compileTestJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :qa:repository-multi-version:compileTestJava
Note: /home/alex/wazuh/wazuh-indexer/qa/repository-multi-version/src/test/java/org/opensearch/upgrades/MultiVersionRepositoryAccessIT.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :plugins:repository-gcs:compileJava
Note: /home/alex/wazuh/wazuh-indexer/plugins/repository-gcs/src/main/java/org/opensearch/repositories/gcs/GoogleCloudStorageBlobStore.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

BUILD SUCCESSFUL in 14m 50s
883 actionable tasks: 709 executed, 1 from cache, 173 up-to-date

AlexRuiz7 · 2023-08-08T08:34:09Z

As the generated packages do not include any plugins, I'm looking for the process they call assembly in the https://github.com/opensearch-project/opensearch-build repo.

AlexRuiz7 · 2023-08-08T13:16:44Z

I've been reading the documentation of the opensearch-build repo, but it's outdated. The mentioned files in the README do not match the existing files.

I've been figuring out which files do correspond to the mentioned ones in the documentation.

AlexRuiz7 · 2023-08-22T12:19:09Z

I managed to generate a "fat" deb package using OpenSearch's tooling:

There are dedicated runners (docker images) to run these tools:

# Start the runner attaching the tools as a volume
docker run -ti --rm -v .:/wazuh-build opensearchstaging/ci-runner:ci-runner-ubuntu2004-opensearch-build-v2 bash

# Build
./build.sh manifests/2.9.0/opensearch-2.9.0.yml --component OpenSearch security -d deb -p linux -a x64

# Assemble (install plugins)
./assemble.sh deb/builds/opensearch/manifest.yml

AlexRuiz7 · 2023-09-18T15:29:31Z

Included plugins (v4.6.0)

wazuh-indexer:

[root@rhel7 vagrant]# /usr/share/wazuh-indexer/bin/opensearch-plugin list
opensearch-alerting
opensearch-anomaly-detection
opensearch-asynchronous-search
opensearch-cross-cluster-replication
opensearch-geospatial
opensearch-index-management
opensearch-job-scheduler
opensearch-knn
opensearch-ml
opensearch-neural-search
opensearch-notifications
opensearch-notifications-core
opensearch-observability
opensearch-performance-analyzer
opensearch-reports-scheduler
opensearch-security
opensearch-security-analytics
opensearch-sql

wazuh-dashboard:

[root@rhel7 vagrant]# /usr/share/wazuh-dashboard/bin/opensearch-dashboards-plugin list --allow-root
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]

Plugins are mostly 1:1, which means each plugin has an indexer and a dashboard component. We
have discarded some plugins in dashboard, but haven't done the same with their indexer component.

Some plugins, as opensearch-neural-search, opensearch-knn and opensearch-ml (in this case mlCommonsDashboards too) could be skipped, as we do not have any reason to include them.

AlexRuiz7 · 2023-09-21T11:18:15Z

Process documented. Also in GDrive

Wazuh indexer packages generation

Compatibility matrix

Current compatibility is in the documentation
Desired compatibility matrix is in this table.

Pre-requisites

Docker
OpenSearch-build repo (wazuh-build)

Start the staging docker container:

docker run -ti --rm \
    -e JAVA_HOME=/opt/java/openjdk-17 \
    -v .:/usr/share/opensearch \
    opensearchstaging/ci-runner:ci-runner-centos7-opensearch-build-v3 \
    bash

Build minified packages

By using the build.sh script.

Usage:

usage: run_build.py [-h] [-l] [-s] [-c [COMPONENTS ...]] [--keep] [-p {linux,darwin,windows}] [-a {x64,arm64}] [-v] [-d {tar,zip,rpm,deb}]
                    [--continue-on-error]
                    manifest

Build an OpenSearch Distribution

positional arguments:
  manifest              Manifest file.

optional arguments:
  -h, --help            show this help message and exit
  -l, --lock            Generate a stable reference manifest.
  -s, --snapshot        Build snapshot.
  -c [COMPONENTS ...], --component [COMPONENTS ...]
                        Rebuild one or more components.
  --keep                Do not delete the working temporary directory.
  -p {linux,darwin,windows}, --platform {linux,darwin,windows}
                        Platform to build.
  -a {x64,arm64}, --architecture {x64,arm64}
                        Architecture to build.
  -v, --verbose         Show more verbose output.
  -d {tar,zip,rpm,deb}, --distribution {tar,zip,rpm,deb}
                        Distribution to build.
  --continue-on-error   Do not fail the distribution build on any plugin component failure.

Build RPM and DEB packages for Linux x64:

./build.sh manifests/2.9.0/opensearch-2.9.0.yml -d rpm -p linux -a x64
./build.sh manifests/2.9.0/opensearch-2.9.0.yml -d deb -p linux -a x64

Assemble packages

By using the assemble.sh script.

Usage:

usage: run_assemble.py [-h] [-b BASE_URL] [--keep] [-v] manifest

Assemble an OpenSearch Distribution

positional arguments:
  manifest              Manifest file.

optional arguments:
  -h, --help            show this help message and exit
  -b BASE_URL, --base-url BASE_URL
                        The base url to download the artifacts.
  --keep                Do not delete the working temporary directory.
  -v, --verbose         Show more verbose output.

Assemble RPM and DEB packages from the previous step:

./assemble.sh rpm/builds/opensearch/manifest.yml
./assemble.sh deb/builds/opensearch/manifest.yml

AlexRuiz7 · 2023-11-23T15:21:05Z

Continued in wazuh/wazuh-packages#54

AlexRuiz7 added level/task Task issue type/research Research issue labels Aug 2, 2023

AlexRuiz7 self-assigned this Aug 2, 2023

gdiazlo added level/epic Epic issue type/enhancement Enhancement issue and removed level/task Task issue type/research Research issue labels Aug 2, 2023

wazuhci added this to Release 4.8.0 Aug 2, 2023

gdiazlo mentioned this issue Aug 14, 2023

Wazuh indexer fork first steps #1

Closed

7 tasks

wazuhci moved this to In progress in Release 4.8.0 Aug 16, 2023

wazuhci moved this from In progress to On hold in Release 4.8.0 Aug 29, 2023

wazuhci removed this from Release 4.8.0 Sep 7, 2023

gdiazlo added the request/operational Operational requests label Sep 18, 2023

AlexRuiz7 mentioned this issue Nov 14, 2023

Wazuh indexer fork update #54

Closed

7 tasks

AlexRuiz7 added level/task Task issue and removed level/epic Epic issue labels Nov 14, 2023

wazuhci added this to Release 4.9.0 Nov 14, 2023

wazuhci moved this to Backlog in Release 4.9.0 Nov 14, 2023

wazuhci moved this from Backlog to In progress in Release 4.9.0 Nov 17, 2023

wazuhci moved this from In progress to On hold in Release 4.9.0 Nov 23, 2023

AlexRuiz7 closed this as completed Nov 23, 2023

wazuhci moved this from On hold to Done in Release 4.9.0 Nov 23, 2023

AlexRuiz7 mentioned this issue Nov 23, 2023

Phase I: Build - Automate the generation of min packages #61

Closed

4 tasks

AlexRuiz7 mentioned this issue May 9, 2024

Wazuh indexer startup errors #225

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generate production packages with Wazuh changes and config files #4

Generate production packages with Wazuh changes and config files #4

AlexRuiz7 commented Aug 2, 2023 •

edited

Loading

AlexRuiz7 commented Aug 7, 2023 •

edited

Loading

AlexRuiz7 commented Aug 7, 2023 •

edited

Loading

AlexRuiz7 commented Aug 7, 2023

AlexRuiz7 commented Aug 8, 2023

AlexRuiz7 commented Aug 8, 2023

AlexRuiz7 commented Aug 22, 2023 •

edited

Loading

AlexRuiz7 commented Sep 18, 2023

AlexRuiz7 commented Sep 21, 2023

AlexRuiz7 commented Nov 23, 2023

Generate production packages with Wazuh changes and config files #4

Generate production packages with Wazuh changes and config files #4

Comments

AlexRuiz7 commented Aug 2, 2023 • edited Loading

Description

Tasks

Related issues

AlexRuiz7 commented Aug 7, 2023 • edited Loading

AlexRuiz7 commented Aug 7, 2023 • edited Loading

AlexRuiz7 commented Aug 7, 2023

AlexRuiz7 commented Aug 8, 2023

AlexRuiz7 commented Aug 8, 2023

AlexRuiz7 commented Aug 22, 2023 • edited Loading

AlexRuiz7 commented Sep 18, 2023

Included plugins (v4.6.0)

AlexRuiz7 commented Sep 21, 2023

Wazuh indexer packages generation

Compatibility matrix

Pre-requisites

Build minified packages

Assemble packages

AlexRuiz7 commented Nov 23, 2023

AlexRuiz7 commented Aug 2, 2023 •

edited

Loading

AlexRuiz7 commented Aug 7, 2023 •

edited

Loading

AlexRuiz7 commented Aug 7, 2023 •

edited

Loading

AlexRuiz7 commented Aug 22, 2023 •

edited

Loading