Add filter and foreach sub commands for bulk operations #3027

vinit-chauhan · 2025-10-27T15:26:38Z

Elastic-package has been extensively used for integrations related tasks. However, most of the commands now are targeted to be run on one package at a time. We don’t have an option if we want to repeat a certain operation across multiple integrations.

This pull request adds two subcommands to the elastic-package to allow bulk operations from the elastic-package.

filter
foreach

Note: Both commands are expected to be run from the integration repository.

Filter subcommand

Filter command adds the ability to filter and return a list of integrations based on specified criteria..

Available Filters:

Category
Codeowner
Input
Integration Type ( eg., integration, input )
Package Name ( supports glob patterns )
Spec Version

You can chain multiple filters and each filter can have multiple comma-separated values
Matching:
All filters must match.
At least one of the values match

Currently, the filter runs sequentially and reads all the manifest files at once and keeps it in memory. We can update the code to use a buffered channel and have a producer and consumer to reduce memory footprint Additionally we can use a worker pool to spread out filtering.

The current elastic-package and spec does not enforce the same package_name and directory name in the repo. Which leads to some integrations having different package_name and directory name.

The filter command by default returns the name of the directory in the integration repo. However, it also provides a flag --output-package-name / -p to make it return the package name.

If no filter flag is provided, the command will return a list of all the integrations.

elastic-package filter --input tcp,udp --code-owner elastic/integration-experience --package-name cisco_*

Foreach subcommand

The foreach command leverages the filter registry. Therefore all the flags available in the filter are directly available to foreach commands without any code changes.

Additionally, foreach has 1 flag --parallel which allows the user to run commands parallelly using worker pool.

default is 1 ( runs sequentially )

The elastic-package command you want to run goes after -- with all of its flags.

Note: You are only allowed to run allowed elastic-package subcommands ["build","check","clean","format","install","lint","test","uninstall"] (cmd/foreach.go)

elastic-package foreach --input tcp,udp --code-owner elastic/integration-experience --parallel 5 -- test pipeline –generate

File changes:

internal/packages/packages.go: Added function to find the integrations repo root dir and read all manifests.
cmd/filter.go: Filter command implementation
cmd/foreach.go: Foreach command implementation
internal/filter/*: Filter interface and implementation for each filter flag.

Related Issues

AI Tools used

Cursor With Claude-4.5-Sonnet

…lter

…n manifest and dir_name in integrations repo.

teresaromero · 2025-10-28T08:37:48Z

cmd/foreach.go

+
+func executeCommand(args []string, path string) error {
+	// Look up the elastic-package binary in PATH
+	execPath, err := exec.LookPath("elastic-package")


as this is a command run on top of other commands of this binary, could we target directly the commands by its functions, instead of looking for the path binary?
what if the binary is not updated or needs to run specific version? just having some thoughts here on how would be a better aproach without depending of having the binary installed as its own dependency.

I think it should be possible to do something like this:

cmd := cmd.RootCmd() cmd.SetArgs(args) return cmd.Execute()

We could also create our own command that is like cmd.RootCmd(), but where we only add the commands with AddCommand that we allow.

cmd/foreach.go

teresaromero · 2025-10-28T08:43:32Z

internal/packages/packages.go


+// MustFindIntegrationRoot finds and returns the path to the root folder of a package from the working directory.
+// It fails with an error if the package root can't be found.
+func MustFindIntegrationRoot() (string, error) {


I think all this code related to detect the repository root can be reused from https://github.com/vinit-chauhan/elastic-package/blob/caaff9abcc9abb60ec525e40eab441ffd99b7828/internal/files/repository.go#L16 wdyt?

There the "integrations root" is considered the git repository of the integrations repository.

teresaromero · 2025-10-28T08:47:10Z

internal/packages/packages.go

 			(m.Type == dataStreamTypeLogs || m.Type == dataStreamTypeMetrics || m.Type == dataStreamTypeSynthetics || m.Type == dataStreamTypeTraces),
 		nil
 }
+func isIntegrationRepo(path string) (bool, error) {


same as above, i think we can use the reference of the git repository instead of the go.mod file. @jsoriano wdyt?

jsoriano

Thanks for working on this! This will be helpful. Added some questions and suggestions.

jsoriano · 2025-10-28T15:09:47Z

internal/cobraext/flags.go

+	FilterIntegrationTypeFlagName        = "integration-type"
+	FilterIntegrationTypeFlagDescription = "integration types to filter by (comma-separated values)"


Integration is a type of package, this should be probably called package type.

Suggested change

FilterIntegrationTypeFlagName = "integration-type"

FilterIntegrationTypeFlagDescription = "integration types to filter by (comma-separated values)"

FilterPackageTypeFlagName = "package-type"

FilterPackageTypeFlagDescription = "package types to filter by (comma-separated values)"

jsoriano · 2025-10-28T15:10:53Z

internal/cobraext/flags.go

+	FilterPackagesFlagName        = "package-name"
+	FilterPackagesFlagDescription = "package names to filter by (comma-separated values)"


Maybe we can call this flag directly packages, or package-path to avoid confusion between package paths and their names.

jsoriano · 2025-10-28T15:12:04Z

internal/cobraext/flags.go

+	FilterSpecVersionFlagDescription = "Package spec version to filter by (semver)"
+
+	ForeachPoolSizeFlagName        = "parallel"
+	ForeachPoolSizeFlagDescription = "number of packages to execute in parallel (defaults to serial execution)"


Suggested change

ForeachPoolSizeFlagDescription = "number of packages to execute in parallel (defaults to serial execution)"

ForeachPoolSizeFlagDescription = "Number of subcommands to execute in parallel (defaults to serial execution)"

jsoriano · 2025-10-28T15:18:02Z

internal/filter/packagename.go

+
+func (f *PackageNameFlag) Matches(dirName string, manifest *packages.PackageManifest) bool {
+	for _, pattern := range f.patterns {
+		if pattern.Match(dirName) {


Yeah, here it is confusing to have a filter package name, that is actually matching paths.

jsoriano · 2025-10-28T15:21:47Z

internal/packages/packages.go


+// ReadAllPackageManifests reads all the package manifests in the given root directory.
+func ReadAllPackageManifests(root string) ([]PackageDirNameAndManifest, error) {
+	files, err := filepath.Glob(filepath.Join(root, "packages", "*", PackageManifestFile))


The packages directory is a convention in the integrations repository, but is not formalized. Actually it is intended that developers can place their packages wherever they want. For example in package-spec and elastic-package repositories there are packages under ./test/packages, and it would be nice to be able to use this tool there too 🙂

The idea of the --packages flag proposed in #2327 was to allow to customize this, even if the default is still ./packages.

In #2327 I also proposed an --auto flag, that was intended to look for packages in a repository, by looking for manifest files and so on. There is no need to implement it, just in case.

Hmm - That makes sense. I overlooked the fact that we allow the integrations in places other than integration repo.

I'll update the code to allow different paths in --packages.

for the --auto; if no filter is provided, it will work return all packages and foreach would perform action on all of those.

jsoriano · 2025-10-28T15:30:34Z

internal/filter/category.go

Nit. Underscores in Go file or package names should be avoided when possible. Specially on file names where some suffixes may have meaning, such as _test.go, or _linux.go.

Yeah, when I added _flag it felt weird to me as well. but I did it to make the flag files distinct. But I'll remove that.

jsoriano · 2025-10-28T15:38:04Z

cmd/foreach.go

+		Path:   execPath,
+		Args:   append([]string{execPath}, args...),
+		Dir:    path,
+		Stdout: io.Discard,


Why discarding stdout?

jsoriano · 2025-10-28T15:40:34Z

internal/filter/type.go

+func (f *FilterFlagBase) Name() string {
+	return f.name
+}
+
+func (f *FilterFlagBase) Description() string {
+	return f.description
+}
+
+func (f *FilterFlagBase) Shorthand() string {
+	return f.shorthand
+}
+
+func (f *FilterFlagBase) DefaultValue() string {
+	return f.defaultValue
+}
+
+func (f *FilterFlagBase) Register(cmd *cobra.Command) {
+	cmd.Flags().StringP(f.Name(), f.Shorthand(), f.DefaultValue(), f.Description())
+}


Nit. No need to define all these functions as this is only used here, right?

Suggested change

func (f *FilterFlagBase) Name() string {

return f.name

}

func (f *FilterFlagBase) Description() string {

return f.description

}

func (f *FilterFlagBase) Shorthand() string {

return f.shorthand

}

func (f *FilterFlagBase) DefaultValue() string {

return f.defaultValue

}

func (f *FilterFlagBase) Register(cmd *cobra.Command) {

cmd.Flags().StringP(f.Name(), f.Shorthand(), f.DefaultValue(), f.Description())

}

func (f *FilterFlagBase) Register(cmd *cobra.Command) {

cmd.Flags().StringP(f.name, f.shorthand, f.defaultValue, f.description)

}

jsoriano · 2025-10-28T15:44:45Z

internal/filter/type.go

+// FilterFlag defines the basic interface for filter flags.
+type FilterFlag interface {
+	Name() string
+	Description() string
+	Shorthand() string
+	DefaultValue() string
+
+	Register(cmd *cobra.Command)
+	IsApplied() bool
+}
+
+// Filter extends FilterFlag with filtering capabilities.
+// It defines the interface for filtering packages based on specific criteria.
+type Filter interface {
+	FilterFlag
+
+	Parse(cmd *cobra.Command) error
+	Validate() error
+	ApplyTo(pkgs []packages.PackageDirNameAndManifest) ([]packages.PackageDirNameAndManifest, error)
+	// Matches checks if a package matches the filter criteria.
+	// dirName is the directory name of the package in package root.
+	Matches(dirName string, manifest *packages.PackageManifest) bool
+}


Nit. Is FilterFlag interface ever used? I think these interfaces can be reduced to one, specially if we remove the methods to access attributes.

Suggested change

// FilterFlag defines the basic interface for filter flags.

type FilterFlag interface {

Name() string

Description() string

Shorthand() string

DefaultValue() string

Register(cmd *cobra.Command)

IsApplied() bool

}

// Filter extends FilterFlag with filtering capabilities.

// It defines the interface for filtering packages based on specific criteria.

type Filter interface {

FilterFlag

Parse(cmd *cobra.Command) error

Validate() error

ApplyTo(pkgs []packages.PackageDirNameAndManifest) ([]packages.PackageDirNameAndManifest, error)

// Matches checks if a package matches the filter criteria.

// dirName is the directory name of the package in package root.

Matches(dirName string, manifest *packages.PackageManifest) bool

}

// Filter defines the interface for filtering packages based on specific criteria.

type Filter interface {

Register(cmd *cobra.Command)

IsApplied() bool

Parse(cmd *cobra.Command) error

Validate() error

ApplyTo(pkgs []packages.PackageDirNameAndManifest) ([]packages.PackageDirNameAndManifest, error)

// Matches checks if a package matches the filter criteria.

// dirName is the directory name of the package in package root.

Matches(dirName string, manifest *packages.PackageManifest) bool

}

internal/filter/utils.go

- update --packages to use only package name; added another filter to filter by package dirs. - Added auto discovery of package for configurable depth. - Added flag to exclude dirs from filter process.

jrmolin · 2025-11-10T16:35:36Z

are you going to pull this out of draft?

vinit-chauhan · 2025-11-12T19:19:56Z

Removed code for parallel execution from the PR for now.
Once the required changes in elastic-package merged will raise a new PR.

mjwolf · 2025-11-12T20:07:33Z

It could be nice to add some additional output formats to the filter command. For example, one result per line output would make working with shell scripts easier. But, it's also easy to convert the json output to do this with jq, so adding multiple formats might not be critical to add.

mjwolf · 2025-11-12T20:15:12Z

internal/packages/packages.go

+		// Validate it's a package manifest
+		ok, err := isPackageManifest(path)
+		if err != nil {
+			// Log the error but continue searching


This doesn't actually log the error here. You should either remove the comment or add logging

jrmolin

there's a startling lack of testing, but it looks reasonable otherwise

jsoriano · 2025-11-12T20:22:12Z

internal/cobraext/flags.go

+	FilterOutputFlagShorthand   = "o"
+
+	FilterPackageDirNameFlagName        = "package-dirs"
+	FilterPackageDirNameFlagDescription = "package directories to filter by (comma-separated values)"


How is this flag expected to work? If I use --package-dirs ./test/packages/parallel in the elastic-package repository it doesn't find any package.

$ elastic-package filter --package-dirs ./test/packages/parallel 2025/11/12 21:20:31 INFO Found 0 matching package(s) null

package-dirs would filter packages based on the name of the package's directory. This is used to search without needing to know the package name in the manifest file.

example:

go run main.go -C ../integrations filter --package-dirs sql_input -o pkgname 2025/11/13 09:29:56 INFO Found 1 matching package(s) ["sql"]

if you want to filter all packages in one directory you can use the following command.

❯ go run main.go -C test/packages/parallel filter 2025/11/13 09:11:51 INFO Found 19 matching package(s) ["apache","apache_basic_license","auditd_manager","auth0_logsdb","aws","awsfirehose","custom_entrypoint","httpcheck","mongodb","nginx","nginx_multiple_services","oracle","otel_http_server","sql_input","system","terraform_local","ti_anomali","ti_anomali_logsdb","ti_anomali_template"]

Oh ok. So it is not possible to search in multiple directories. I guess this is fine.

does it make sense to add the example to the description so it goes to the docs?

jsoriano · 2025-11-12T20:35:14Z

cmd/foreach.go

+	errors := multierror.Error{}
+
+	for _, pkg := range filtered {
+		rootCmd := cmd.Root()


We could define another root command with the specific commands that are supported and use it here, then we could have different sets of subcommands, and maybe we could even plug it directly as subcommands of foreach.

This could be left for a future refactor too.

Something like this:

import ( "github.com/spf13/cobra" "github.com/elastic/elastic-package/internal/cobraext" ) var forEachCommands = []*cobraext.Command{ setupBuildCommand(), setupCheckCommand(), setupCleanCommand(), setupFormatCommand(), setupInstallCommand(), setupLintCommand(), setupTestCommand(), setupUninstallCommand(), } // ForEachRootCmd creates and returns root cmd for elastic-package func ForEachRootCmd() *cobra.Command { forEachCmd := &cobra.Command{ SilenceUsage: true, PersistentPreRunE: func(cmd *cobra.Command, args []string) error { return cobraext.ComposeCommandActions(cmd, args, processPersistentFlags, checkVersionUpdate, ) }, } forEachCmd.PersistentFlags().CountP(cobraext.VerboseFlagName, cobraext.VerboseFlagShorthand, cobraext.VerboseFlagDescription) forEachCmd.PersistentFlags().StringP(cobraext.ChangeDirectoryFlagName, cobraext.ChangeDirectoryFlagShorthand, "", cobraext.ChangeDirectoryFlagDescription) for _, cmd := range forEachCommands { forEachCmd.AddCommand(cmd.Command) } return forEachCmd }

That could be used here like this:

Suggested change

rootCmd := cmd.Root()

rootCmd := ForEachRootCmd()

rootCmd.SetContext(cmd.Context())

This is something that we need to do to add parallel execution. That's why I left it out of this PR, and would like to revisit it while I work on the other PR.

With that said, we do have a list of allowed sub-commands. Here

Let me know if I missed any command. - I'll also add changelog command in allow list.

Yes, we can leave this for a future refactor.

I'll also add changelog command in allow list.

Yes please.

jsoriano · 2025-11-12T20:37:52Z

internal/cobraext/flags.go

+	FilterDepthFlagName        = "depth"
+	FilterDepthFlagDescription = "maximum depth to search for packages"
+	FilterDepthFlagDefault     = 2
+	FilterDepthFlagShorthand   = "d"


Maybe we need to filter out built packages by default, otherwise they are discovered, and it is confusing to see them listed as their versions:

$ elastic-package filter -d 10 --packages apache 2025/11/12 21:31:32 INFO Found 3 matching package(s) ["999.999.999","apache","apache_basic_license"]

999.999.999 there is a built package under ./build/packages/apache/999.999.999/.

I came across the same issue, so I've added a --exclude-dirs flag that can be use to explicitly exclude directories.
Also, .git is excluded by default.

❯ go run main.go filter -d 10 --packages apache --exclude-dirs build 2025/11/13 09:24:18 INFO Found 2 matching package(s) ["apache","apache_basic_license"]

Maybe build could be also excluded by default as I don't think it is ever intended to search into it.

jsoriano · 2025-11-13T18:20:26Z

It could be nice to add some additional output formats to the filter command. For example, one result per line output would make working with shell scripts easier.

I would also prefer to have this format by default, and make json an optional alternative format.

- add build to exclude list - default output to nsv. also allow json and yaml

vinit-chauhan · 2025-11-13T20:47:03Z

Updated the code to support multiple output formats.

- Newline separated list ( default )
- JSON
- YAML

…ent-filter-registry

elasticmachine · 2025-11-15T00:23:20Z

💚 Build Succeeded

Buildkite Build
Commit: 89ef933

History

💔 Build #6613 failed d93ee4f
💚 Build #6604 succeeded 12a7bea
💔 Build #6603 failed 6196985
💚 Build #6585 succeeded 689eb72
💔 Build #6582 failed 61bba87

cc @vinit-chauhan

vinit-chauhan added 30 commits October 9, 2025 14:23

initial structure of command

85d659a

fix: error skipped in edit dashboard command

17e4b39

fix: error skipped in edit dashboard command

01a681c

code refactors

010f1a4

Merge branch 'elastic:main' into cmd-filter

358f9d6

update response format.

c3fef19

move filter code to internal

3d84151

ignore .vscode dir

9a1c2aa

no args will return list of all packages

1336ff2

Merge branch 'main' of github.com:elastic/elastic-package into cmd-fi…

b588164

…lter

initial implementation for foreach.

53bd4f8

run multiple commands

2d827cc

add option to run commands in parallel

bc7a167

stash: filter-registry

ada16f7

working with subcommand flags

9123dae

Merge branch 'cmd-filter' into implement-filter-registry

423f035

created structure with working input filter

2e3721c

minor refactor; add code owner flag

8180ff4

minor refactor

f2abc11

add category flag

f72ea26

refactor code

5c4d63e

added spec version flag

d94e9b2

fix issue with unused filters applied

3cc0824

Merge branch 'main' into implement-filter-registry

78cce2b

Merge branch 'elastic:main' into implement-filter-registry

2656a22

run command in parallel

c36efd7

remove --exec to leverage -- to pass sub commands without string

0649c73

add subcommand allowlist

5e3b8b0

create map of dir_name and manifest to allow different package_name i…

df2c85b

…n manifest and dir_name in integrations repo.

add flag suffix to go files

b495ced

lint and build docs

caaff9a

teresaromero reviewed Oct 28, 2025

View reviewed changes

cmd/foreach.go Outdated Show resolved Hide resolved

teresaromero reviewed Oct 28, 2025

View reviewed changes

jsoriano reviewed Oct 28, 2025

View reviewed changes

vinit-chauhan self-assigned this Oct 30, 2025

vinit-chauhan marked this pull request as draft October 31, 2025 04:31

vinit-chauhan added 3 commits November 5, 2025 09:47

solve some review comments

f997640

- add way to filter packages in arbitrary directory.

a9699e4

- update --packages to use only package name; added another filter to filter by package dirs. - Added auto discovery of package for configurable depth. - Added flag to exclude dirs from filter process.

add a output type flag

90a93ad

jsoriano mentioned this pull request Nov 11, 2025

Stop using Chdir #3066

Draft

remove parallel execution code from PR.

689eb72

vinit-chauhan force-pushed the implement-filter-registry branch from 61bba87 to 689eb72 Compare November 12, 2025 17:49

vinit-chauhan marked this pull request as ready for review November 12, 2025 19:17

vinit-chauhan requested review from jsoriano and teresaromero November 12, 2025 19:55

mjwolf reviewed Nov 12, 2025

View reviewed changes

jrmolin reviewed Nov 12, 2025

View reviewed changes

jsoriano reviewed Nov 12, 2025

View reviewed changes

vinit-chauhan added 2 commits November 13, 2025 15:37

- add changelog to allowlist

6196985

- add build to exclude list - default output to nsv. also allow json and yaml

fix: check

12a7bea

vinit-chauhan added 2 commits November 14, 2025 11:13

add example in filter command.

d93ee4f

Merge branch 'main' of github.com:elastic/elastic-package into implem…

89ef933

…ent-filter-registry

		FilterIntegrationTypeFlagName = "integration-type"
		FilterIntegrationTypeFlagDescription = "integration types to filter by (comma-separated values)"

		FilterPackagesFlagName = "package-name"
		FilterPackagesFlagDescription = "package names to filter by (comma-separated values)"

	ForeachPoolSizeFlagDescription = "number of packages to execute in parallel (defaults to serial execution)"
	ForeachPoolSizeFlagDescription = "Number of subcommands to execute in parallel (defaults to serial execution)"

	rootCmd := cmd.Root()
	rootCmd := ForEachRootCmd()
	rootCmd.SetContext(cmd.Context())

Add filter and foreach sub commands for bulk operations #3027

Are you sure you want to change the base?

Add filter and foreach sub commands for bulk operations #3027

Uh oh!

Conversation

vinit-chauhan commented Oct 27, 2025

Filter subcommand

Foreach subcommand

File changes:

Related Issues

AI Tools used

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jsoriano left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jsoriano Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jrmolin commented Nov 10, 2025

Uh oh!

vinit-chauhan commented Nov 12, 2025

Uh oh!

mjwolf commented Nov 12, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jrmolin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vinit-chauhan Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jsoriano Oct 28, 2025 •

edited

Loading

vinit-chauhan Nov 13, 2025 •

edited

Loading