npmi-go

npmi-go caches the contents of node_modules directory in a tarball stored locally or in a Minio instance. The Node runtime environment and a hash of package-lock.json is used as the cache key.

The cache key is something like v12.16.3-darwin-x64-dev-78c49bbaba2e4002e313e55018716d9a673fa99f1e676afcb03df0a902f4883f.

pre-1.0 software

Note that npmi-go is still pre-1.0, which means that breaking changes MAY be present in point releases if unavoidable.

Signing keys

The artifacts are signed using both minisign and GPG for convenience.

Installation

Download a prebuilt binary from the releases page

Building

Quick & dirty

Just run go build in the root directory to build npmi-go binary.

Goreleaser

Dev builds for your system only may be built by using goreleaser:

goreleaser build --snapshot --clean --single-target

The output will appear under dist/.

Creating a symlink with a shorter name is recommended for testing: ln -s dist/npmi-go_linux_amd64/npmi-go

Releasing

Create a git tag with the desired version

git tag v0.4.0

Create a release build to see if everything is setup correctly

goreleaser release --clean --skip=publish

If everything seems OK, release to Github by running

goreleaser release --clean

Supported Caches

Local

The local cache is a directory containing tarballs.

It is enabled by default and caches to the system temp directory.

To disable the local cache use the flag -local=0.

See the -local* options in usage for more info.

Minio

A Minio instance may also be used as a cache backend.

Using Minio allows several clients to share the same cache. CI systems often install the same deps over and over again and using a shared cache will reduce bandwidth, IO and CPU usage significantly.

Security notice

Note that the contents of a tarball in the cache are not checked in any way and only trusted systems should be allowed to access the shared cache.

Testing with Minio

Start a temporary Minio instance using Docker:

docker run --rm --name minio \
-p 9000:9000 -p 9001:9001 \
-e "MINIO_ROOT_USER=minio" \
-e "MINIO_ROOT_PASSWORD=password" \
-v "$PWD/test/minio/certs:/root/.minio/certs" \
minio/minio \
server /data --console-address ":9001"

Edit test/minio/config.json and change the IP to match whatever Minio reports.
Create a bucket called npmi using Minio Client:

docker run -v "$PWD/test/minio/config.json:/root/.mc/config.json" minio/mc --insecure mb minio/npmi

Use the dummy project under testdata/ and run npmi-go using minio:

cd testdata
../npmi-go -loglevel trace \
-minio=1 \
-minio-endpoint=localhost:9000 \
-minio-bucket=npmi \
-minio-access-key-id=minio \
-minio-secret-access-key=password \
-minio-tls-insecure \
-local=0

Note that we disable the local cache for testing purposes.

2024-02-14T13:13:16.176+0200 [INFO]  npmi: Starting installation: version=dev
2024-02-14T13:13:16.176+0200 [TRACE] npmi.cache: start: cacheKey=v20.10.0-linux-x64-dev-b7782c38ef77fe9874c3c30e7a0ba49ce90c8a3e394df9b775db4e502ec19f26
2024-02-14T13:13:16.176+0200 [TRACE] npmi.cache.minio.lookup: start
2024-02-14T13:13:16.176+0200 [TRACE] npmi.cache.minio.has: start: key=v20.10.0-linux-x64-dev-b7782c38ef77fe9874c3c30e7a0ba49ce90c8a3e394df9b775db4e502ec19f26
2024-02-14T13:13:16.181+0200 [TRACE] npmi.cache.minio.has: complete: found=false
2024-02-14T13:13:16.181+0200 [TRACE] npmi.cache.minio.lookup: complete
2024-02-14T13:13:16.181+0200 [DEBUG] npmi.cache.minio.lookup: cache MISS
2024-02-14T13:13:16.181+0200 [TRACE] npmi.cache: complete
2024-02-14T13:13:16.181+0200 [TRACE] npmi.install: start
2024-02-14T13:13:16.181+0200 [TRACE] npmi.installPackages: start
2024-02-14T13:13:16.181+0200 [TRACE] npmi.npmInstaller: Running: npmBinary=/usr/bin/npm args=["ci", "--dev", "--loglevel", "error", "--progress", "false"]
2024-02-14T13:13:16.814+0200 [TRACE] npmi.installPackages: complete: stdout="added 2 packages, and audited 3 packages in 492ms\n\nfound 0 vulnerabilities"
2024-02-14T13:13:16.814+0200 [TRACE] npmi.installPackages: complete
2024-02-14T13:13:16.814+0200 [TRACE] npmi.createArchive: start
2024-02-14T13:13:16.814+0200 [DEBUG] npmi.createArchive: Creating archive: path=/tmp/modules-v20.10.0-linux-x64-dev-b7782c38ef77fe9874c3c30e7a0ba49ce90c8a3e394df9b775db4e502ec19f26.tar.gz
2024-02-14T13:13:16.821+0200 [TRACE] npmi.createArchive: complete
2024-02-14T13:13:16.821+0200 [TRACE] npmi.cacheArchive: start
2024-02-14T13:13:16.821+0200 [TRACE] npmi.cacheArchive.minio: start
2024-02-14T13:13:16.821+0200 [TRACE] npmi.cache.minio.put: start: key=v20.10.0-linux-x64-dev-b7782c38ef77fe9874c3c30e7a0ba49ce90c8a3e394df9b775db4e502ec19f26
2024-02-14T13:13:16.848+0200 [TRACE] npmi.cache.minio.put: complete
2024-02-14T13:13:16.848+0200 [TRACE] npmi.cacheArchive.minio: complete
2024-02-14T13:13:16.848+0200 [TRACE] npmi.cacheArchive: complete
2024-02-14T13:13:16.848+0200 [DEBUG] npmi.createArchive: Removed temporary archive: path=/tmp/modules-v20.10.0-linux-x64-dev-b7782c38ef77fe9874c3c30e7a0ba49ce90c8a3e394df9b775db4e502ec19f26.tar.gz
2024-02-14T13:13:16.848+0200 [TRACE] npmi: complete
2024-02-14T13:13:16.848+0200 [INFO]  npmi: Installation complete

Now run the same npmi-go command again. The output will be something like:

2024-02-14T13:14:37.014+0200 [INFO]  npmi: Starting installation: version=dev
2024-02-14T13:14:37.014+0200 [TRACE] npmi.cache: start: cacheKey=v20.10.0-linux-x64-dev-b7782c38ef77fe9874c3c30e7a0ba49ce90c8a3e394df9b775db4e502ec19f26
2024-02-14T13:14:37.014+0200 [TRACE] npmi.cache.minio.lookup: start
2024-02-14T13:14:37.014+0200 [TRACE] npmi.cache.minio.has: start: key=v20.10.0-linux-x64-dev-b7782c38ef77fe9874c3c30e7a0ba49ce90c8a3e394df9b775db4e502ec19f26
2024-02-14T13:14:37.021+0200 [TRACE] npmi.cache.minio.has: complete: found=true
2024-02-14T13:14:37.021+0200 [TRACE] npmi.cache.minio.lookup: complete
2024-02-14T13:14:37.021+0200 [DEBUG] npmi.cache.minio.lookup: cache HIT
2024-02-14T13:14:37.021+0200 [TRACE] npmi.cache.minio.fetch: start
2024-02-14T13:14:37.021+0200 [TRACE] npmi.cache.minio.put: start: key=v20.10.0-linux-x64-dev-b7782c38ef77fe9874c3c30e7a0ba49ce90c8a3e394df9b775db4e502ec19f26
2024-02-14T13:14:37.021+0200 [TRACE] npmi.cache.minio.fetch: complete
2024-02-14T13:14:37.021+0200 [TRACE] npmi.cache.minio.extract: start
2024-02-14T13:14:37.030+0200 [TRACE] npmi.cache.minio.extract.cleanup: start
2024-02-14T13:14:37.030+0200 [TRACE] npmi.cache.minio.extract.cleanup: complete: numFilesRemoved=0 filesRemoved=[]
2024-02-14T13:14:37.030+0200 [TRACE] npmi.cache.minio.extract: complete
2024-02-14T13:14:37.030+0200 [DEBUG] npmi.cache.minio: packages successfully installed from cache
2024-02-14T13:14:37.030+0200 [TRACE] npmi.cache: complete
2024-02-14T13:14:37.030+0200 [TRACE] npmi: complete
2024-02-14T13:14:37.030+0200 [INFO]  npmi: Installation complete

See the -minio* options in usage for more info.

Usage

npmi-go dev, commit none, built at unknown.
npmi-go installs NPM packages from a cache to speed up repeating installations.

Supported caches:
-  local          Data is cached locally in a directory.
-  minio          Data is cached to a (shared) Minio instance.

When using both caches, the local one is accessed first.

USAGE:
 npmi-go [OPTIONS]

ENVIRONMENT VARIABLES:
Use the following env variables to set default options.

  NPMI_LOGLEVEL  Log level. One of info|debug|trace (Default: "info")
  NPMI_JSON      Use JSON for log output (Default: false)
  NPMI_VERBOSE   Verbose output. DEPRECATED
                 Please use NPMI_LOGLEVEL with 'debug' or 'trace'
  NPMI_FORCE     Force (re)installation of deps
  NPMI_PRECACHE  Pre-cache command
  NPMI_TEMP_DIR  Use specified temp directory when creating archives (Default: system temp)

Tar file security hardening:
  NPMI_TAR_ABSOLUTE_PATHS           Allow absolute paths in tar archives (Default: true)
  NPMI_TAR_DOUBLE_DOT_PATHS         Allow double dot paths in tar archives (Default: true)
  NPMI_TAR_LINKS_OUTSIDE_CWD  Allow links outside of the current working directory (Default: true)

Local cache:
  NPMI_LOCAL      Use local cache
  NPMI_LOCAL_DIR  Local cache directory (Default: system temp)

Minio cache:
  NPMI_MINIO                    Use Minio cache
  NPMI_MINIO_ENDPOINT           Minio endpoint URL
  NPMI_MINIO_ACCESS_KEY_ID      Minio access key ID
  NPMI_MINIO_SECRET_ACCESS_KEY  Minio secret access key
  NPMI_MINIO_BUCKET             Minio bucket name
  NPMI_MINIO_TLS                Use TLS when connection to minio
  NPMI_MINIO_TLS_INSECURE       Disable TLS certificate checks

OPTIONS:
  -force
        Force (re)installation of NPM deps and update cache(s)
  -json
        Use JSON output
  -local
        Use local cache
  -local-dir string
        Local cache directory (default "/tmp")
  -loglevel string
        Log level. One of info|debug|trace (default "info")
  -minio
        Use Minio for caching (default true)
  -minio-access-key-id string
        Minio access key ID (default "EJjhQWkij3PlGwxVcN0n")
  -minio-bucket string
        Minio Bucket (default "npmi-go")
  -minio-endpoint string
        Minio endpoint (default "minio-npmi-go.ci.dev.verkkokauppa.com")
  -minio-secret-access-key string
        Minio secret access key (default "K1BlFkl1In8VsewpQzjX")
  -minio-tls
        Use TLS to access Minio cache (default true)
  -minio-tls-insecure
        Disable TLS certificate checks
  -precache string
        Run the following shell command before caching packages
  -tar-absolute-paths
        Allow absolute paths in tar archives (default true)
  -tar-double-dot-paths
        Allow double dot paths in tar archives (default true)
  -tar-links-outside-cwd
        Allow links outside of the current working directory (default true)
  -temp-dir string
        Temporary directory for archive creation (default "/tmp")
  -verbose
        Verbose output, DEPRECATED
        Please use -loglevel with 'debug' or 'trace'

Configuration with .npmirc

npmi-go does not currently support a config file.

Configuration with environment variables

The environment variables described before are used as defaults when present.

Known Issues

package-lock.json sync is not checked

npmi-go does not check is a package-lock.json file is in sync with package.lock.

post-installation side effects outside node_modules/ will be ignored

Any post-installation script of NPM will NOT get run when installing from cache. This includes at least the following:

install
postinstall

Name		Name	Last commit message	Last commit date
Latest commit History 197 Commits
.github		.github
bench		bench
cmd		cmd
pkg		pkg
test		test
testdata		testdata
.gitignore		.gitignore
.goreleaser.yml		.goreleaser.yml
LICENSE		LICENSE
README.md		README.md
dev.md		dev.md
go.mod		go.mod
go.sum		go.sum
gpg.asc		gpg.asc
main.go		main.go
minisign.pub		minisign.pub
npmi-go.svg		npmi-go.svg

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

npmi-go

pre-1.0 software

Signing keys

Installation

Building

Quick & dirty

Goreleaser

Releasing

Supported Caches

Local

Minio

Security notice

Testing with Minio

Usage

Configuration with .npmirc

Configuration with environment variables

Known Issues

package-lock.json sync is not checked

post-installation side effects outside node_modules/ will be ignored

About

Releases 12

Contributors 2

Languages

License

hermo/npmi-go

Folders and files

Latest commit

History

Repository files navigation

npmi-go

pre-1.0 software

Signing keys

Installation

Building

Quick & dirty

Goreleaser

Releasing

Supported Caches

Local

Minio

Security notice

Testing with Minio

Usage

Configuration with .npmirc

Configuration with environment variables

Known Issues

package-lock.json sync is not checked

post-installation side effects outside node_modules/ will be ignored

About

Resources

License

Stars

Watchers

Forks

Releases 12

Contributors 2

Languages