npmi-go

README

npmi-go

npmi-go caches the contents of node_modules directory in a tarball stored locally or in a Minio instance. The Node runtime environment and a hash of package-lock.json is used as the cache key.

The cache key is something like v12.16.3-darwin-x64-dev-78c49bbaba2e4002e313e55018716d9a673fa99f1e676afcb03df0a902f4883f.

Not for production (yet)

Note that npmi-go is work-in-progress and should not be used in production.

Installation

• Download a prebuilt binary from the releases page

Supported Caches

Local

The local cache is a directory containing tarballs.

It is enabled by default and caches to the system temp directory.

To disable the local cache use the flag -local=0.

See the -local* options in usage for more info.

Minio

A Minio instance may also be used as a cache backend.

Using Minio allows several clients to share the same cache. CI systems often install the same deps over and over again and using a shared cache will reduce bandwidth, IO and CPU usage significantly.

Security notice

Note that the contents of a tarball in the cache are not checked in any way and only trusted systems should be allowed to access the shared cache.

Testing with Minio

Start a temporary Minio instance using Docker:

docker run --rm --name minio -p 9000:9000 minio/minio server /data

See the output to determine the instance address and default credentials.

Log in to Minio and create a bucket (npmi in this example).

Create a dummy NPM project:

mkdir npmi-test && cd npmi-test
npm init -y
npm add is-odd

Now run npmi-go using minio:

npmi-go   -verbose \
          -minio=1 \
          -minio-endpoint=localhost:9000 \
          -minio-bucket=npmi \
          -minio-access-key-id=minioadmin \
          -minio-secret-access-key=minioadmin \
          -minio-tls=0 \
          -local=0

Note that we disable the local cache for testing purposes.

Lookup start, looking for key v12.16.3-darwin-x64-dev-xxxx
Lookup(minio).Has start
Lookup(minio).Has complete: MISS
Lookup complete
Install start
Install(npm).InstallPackages start
Install(npm).InstallPackages complete: success: added 2 packages in 0.04s
Install complete
Archive start
Archive creating modules-v12.16.3-darwin-x64-dev-xxxx.tar.gz
Archive complete
Cache start
Cache(minio).Open start
Cache(minio).Open complete
Cache(minio).Put start
Cache(minio).Put complete
Cache complete
npmi-go complete
Post-Archive: Removed temporary archive modules-v12.16.3-darwin-x64-dev-xxxx.tar.gz

Now run the same npmi-go command again. The output will be something like:

npmi-go start
Lookup start, looking for key v12.16.3-darwin-x64-dev-xxxx
Lookup(minio).Has start
Lookup(minio).Has complete: HIT
Lookup(minio).Get start
Lookup(minio).Get complete
Lookup(minio).Extract start
Cleanup start
Cleanup complete, 0 extraneous files removed
Lookup(minio).Extract complete
Lookup complete
npmi-go complete

See the -minio* options in usage for more info.

Usage

npmi-go v0.0.0-SNAPSHOT-xxxx, commit xxxx, built at XXXX.
npmi-go installs NPM packages from a cache to speed up repeating installations.

Supported caches:
-  local		Data is cached locally in a directory.
-  minio		Data is cached to a (shared) Minio instance.

When using both caches, the local one is accessed first.

USAGE:
 npmi-go [OPTIONS]

ENVIRONMENT VARIABLES:
Use the following env variables to set default options.

  NPMI_VERBOSE   Verbose output
  NPMI_FORCE     Force (re)installation of deps
  NPMI_PRECACHE  Pre-cache command

Local cache:
  NPMI_LOCAL      Use local cache
  NPMI_LOCAL_DIR  Local cache directory

Minio cache:
  NPMI_MINIO                    Use Minio cache
  NPMI_MINIO_ENDPOINT           Minio endpoint URL
  NPMI_MINIO_ACCESS_KEY_ID      Minio access key ID
  NPMI_MINIO_SECRET_ACCESS_KEY  Minio secret access key
  NPMI_MINIO_BUCKET             Minio bucket name
  NPMI_MINIO_TLS                Use TLS when connection to minio

OPTIONS:
  -force
    	Force (re)installation of NPM deps and update cache(s)
  -local
    	Use local cache (default true)
  -local-dir string
    	Local cache directory (default "$TEMP")
  -minio
    	Use Minio for caching
  -minio-access-key-id string
    	Minio access key ID
  -minio-bucket string
    	Minio Bucket
  -minio-endpoint string
    	Minio endpoint
  -minio-secret-access-key string
    	Minio secret access key
  -minio-tls
    	Use TLS to access Minio cache (default true)
  -precache string
    	Run the following shell command before caching packages
  -verbose
    	Verbose output

Configuration with .npmirc

npmi-go does not currently support a config file.

Configuration with environment variables

The environment variables described before are used as defaults when present.

Known Issues

package-lock.json sync is not checked

npmi-go does not check is a package-lock.json file is in sync with package.lock.

post-installation side effects outside node_modules/ will be ignored

Any post-installation script of NPM will NOT get run when installing from cache. This includes at least the following:

• install
• postinstall