@kilohealth/web-app-monitoring

The Idea

Package was created to abstract away underlying level of monitoring, to make it easy to setup monitoring for any env as well as to make it easy to migrate from DataDog to other monitoring solution.

The package consists of 3 parts:

• Browser / Client monitoring (browser logs)
• CLI (needed to upload sourcemaps for browser monitoring)
• Server monitoring (server logs, APM, tracing)

Getting Started

Note: If you are migrating from direct datadog integration - don’t forget to remove @datadog/... dependencies. Those are now dependencies of @kilohealth/web-app-monitoring.

npm uninstall @datadog/...

Install package

npm install @kilohealth/web-app-monitoring

Setup environment variables

Variable	Description	Upload source maps	Server (APM, tracing)	Browser / Client
MONITORING_TOOL__API_KEY	This key is needed in order to uploaded source maps for browser monitoring, send server side (APM) logs and tracing info. You can find API key here.	✔️	✔️
MONITORING_TOOL__SERVICE_NAME	The service name, for example: timely-hand-web-funnel-app.	✔️	✔️	✔️
MONITORING_TOOL__SERVICE_VERSION	The service version, for example: $CI_COMMIT_SHA.	✔️	✔️	✔️
MONITORING_TOOL__SERVICE_ENV	The service environment, for example: $CI_ENVIRONMENT_NAME.	✔️	✔️	✔️
MONITORING_TOOL__CLIENT_TOKEN	This token is needed in order to send browser monitoring logs. You can create or find client token here.	️		✔️

Note: Depending on the framework you are using, in order to expose environment variables to the client you may need to prefix the environment variables as mentioned below:

• For Next.js, add the prefix NEXT_PUBLIC_ to each variable. Refer to the documentation for more details.
• For Gatsby.js, add the prefix GATSBY_ to each variable. Refer to the documentation for more details.
• For Vite.js, add the prefix VITE_ to each variable. Refer to the documentation for more details.

Tip: By following Single Source of Truth principle you can reexport variables, needed for the client, in the build stage (Next.js example):

NEXT_PUBLIC_MONITORING_TOOL__SERVICE_NAME=$MONITORING_TOOL__SERVICE_NAME
NEXT_PUBLIC_MONITORING_TOOL__SERVICE_VERSION=$MONITORING_TOOL__SERVICE_VERSION
NEXT_PUBLIC_MONITORING_TOOL__SERVICE_ENV=$MONITORING_TOOL__SERVICE_ENV

Setup browser monitoring

Generate hidden source maps

In order to upload source maps into the monitoring service we need to include those source map files into our build. This can be done by slightly altering the build phase bundler configuration of our app:

Next.js (next.config.js)

module.exports = {
  webpack: (config, context) => {
    const isClient = !context.isServer;
    const isProd = !context.dev;
    const isSourcemapsUploadEnabled = Boolean(
      process.env.MONITORING_TOOL__API_KEY,
    );

    // Generate source maps only for the client side production build
    if (isClient && isProd && isSourcemapsUploadEnabled) {
      return {
        ...config,
        // No reference. No source maps exposure to the client (browser).
        // Hidden source maps generation only for error reporting purposes.
        devtool: 'hidden-source-map',
      };
    }

    return config;
  },
};

Refer to the documentation for more details.

Gatsby.js (gatsby-node.js)

module.exports = {
  onCreateWebpackConfig: ({ stage, actions }) => {
    const isSourcemapsUploadEnabled = Boolean(
      process.env.MONITORING_TOOL__API_KEY,
    );
    // build-javascript is prod build phase
    if (stage === 'build-javascript' && isSourcemapsUploadEnabled) {
      actions.setWebpackConfig({
        // No reference. No source maps exposure to the client (browser).
        // Hidden source maps generation only for error reporting purposes.
        devtool: 'hidden-source-map',
      });
    }
  },
};

Refer to the documentation for more details.

Vite.js (vite.config.js)

export default defineConfig({
  build: {
    // No reference. No source maps exposure to the client (browser).
    // Hidden source maps generation only for error reporting purposes.
    sourcemap: process.env.MONITORING_TOOL__API_KEY ? 'hidden' : false,
  },
});

Refer to the documentation for more details.

Note: We are using hidden source maps only for error reporting purposes. That means our source maps are not exposed to the client and there are no references to those source maps in our source code.

Upload generated source maps

In order to upload generated source maps into the monitoring service, you should use web-app-monitoring__upload-sourcemaps bin, provided by @kilohealth/web-app-monitoring package. To run the script you need to provide arguments:

Argument	Description	Vite	Next	Gatsby
--buildDir or -d	This should be RELATIVE path to your build directory. For example ./dist or ./build.	./dist	./.next/static/chunks	./public
--publicPath or -p	This is RELATIVE path, part of URL between domain (which can be different for different environments) and path to file itself. In other words - base path for all the assets within your application. You can think of this as kind of relative Public Path. For example it can be / or /static. In other words this is common relative prefix for all your static files or / if there is none.	/	/_next/static/chunks (!!! _ instead of . in file system)️	/

Script example for Next.js:

"scripts": {
  "upload:sourcemaps": "web-app-monitoring__upload-sourcemaps --buildDir ./.next/static/chunks --publicPath /_next/static/chunks",
  ...
},

And then your CI should run upload:sourcemaps script for the build that includes generated source maps.

Browser Monitoring Usage

Important note: There is no single entry point for package. You can't do something like import { BrowserMonitoringService } from '@kilohealth/web-app-monitoring'; Reason for that is to avoid bundling server-code into client bundle and vice versa. This structure will ensure effective tree shaking during build time.

In case your bundler supports package.json exports field - you can also omit dist in path folder import { BrowserMonitoringService } from '@kilohealth/web-app-monitoring/browser';

import { BrowserMonitoringService } from '@kilohealth/web-app-monitoring/dist/browser';

export const monitoring = new BrowserMonitoringService({
  authToken: NEXT_PUBLIC_MONITORING_TOOL__CLIENT_TOKEN,
  serviceName: NEXT_PUBLIC_MONITORING_TOOL__SERVICE_NAME,
  serviceVersion: NEXT_PUBLIC_MONITORING_TOOL__SERVICE_VERSION,
  serviceEnv: NEXT_PUBLIC_MONITORING_TOOL__SERVICE_ENV,
});

As you can see we are using here all our exposed variables. If any of these is not defined - the service will fall back to console.log and warn you there about it. Now you can just use it like

monitoring.info('Monitoring service initialized');

OPTIONAL: If you are using React you may benefit from utilizing Error Boundaries:

import React, { Component, PropsWithChildren } from 'react';
import { monitoring } from '../services/monitoring';

interface ErrorBoundaryProps {}
interface ErrorBoundaryState {
  hasError: boolean;
}

export class ErrorBoundary extends Component<
  PropsWithChildren<ErrorBoundaryProps>,
  ErrorBoundaryState
> {
  static getDerivedStateFromError(_: Error): ErrorBoundaryState {
    return { hasError: true };
  }

  state: ErrorBoundaryState = {
    hasError: false,
  };

  componentDidCatch(error: Error) {
    monitoring.reportError(error);
  }

  render() {
    if (this.state.hasError) {
      return <h1>Sorry... There was an error</h1>;
    }

    return this.props.children;
  }
}

Setup Server Monitoring (Next.js)

Approach with facade

initServerMonitoring is a facade over ServerMonitoringService. You can instantiate and use that service directly. This function basically does one thing - instantiate it and can do 3 more additional things:

• call overrideNativeConsole - method of the service to override native console, to log to datadog instead.
• cal catchProcessErrors - method of the service to subscribe to native errors, to log them to datadog.
• put service itself into global scope under defined name, so serverside code can use it.

You may wonder why we instantiate service here and not in server-side code. The reason for that is if we override the native console and catch native errors - we would like to set up this as soon as possible. If you don’t care too much about the very first seconds of next server - you can use alternative simpler server side logging solution.

Example for Next.js:

• update next.config.ts to include into start script of production server code next.config.ts:

const {
  initServerMonitoring,
} = require('@kilohealth/web-app-monitoring/dist/server');

module.exports = phase => {
  if (phase === PHASE_PRODUCTION_SERVER) {
    const remoteMonitoringServiceParams = {
      serviceName: process.env.MONITORING_TOOL__SERVICE_NAME,
      serviceVersion: process.env.MONITORING_TOOL__SERVICE_VERSION,
      serviceEnv: process.env.MONITORING_TOOL__SERVICE_ENV,
      authToken: process.env.MONITORING_TOOL__API_KEY,
    };
    const config = {
      shouldOverrideNativeConsole: true,
      shouldCatchProcessErrors: true,
      globalMonitoringInstanceName: 'kiloServerMonitoring',
    };
    initServerMonitoring(remoteMonitoringServiceParams, config);
  }
};

• update custom.d.ts file to declare that global scope now have monitoring service as a prop In order to use ServerMonitoringService instance in other parts of code via global we need to let TS know that we added new property to global object. In Next.js you can just create or add next code into custom.d.ts file in root of the project. Be aware that var name matches string that you provided in code above (kiloServerMonitoring in this case). custom.d.ts:

import { ServerMonitoringService } from '@kilohealth/web-app-monitoring/dist/server';

declare global {
  // eslint-disable-next-line no-var
  var kiloServerMonitoring: ServerMonitoringService;
}

• use it in code

export const getHomeServerSideProps = async context => {
  global.kiloServerMonitoring.info('getHomeServerSideProps called');
};

Approach with direct instantiation

If you don’t care too much about catching native errors or native logs in the early stages of your server app - you can avoid sharing logger via global scope and instead initialize it inside of app.

import { ServerMonitoringService } from '@kilohealth/web-app-monitoring/dist/server';

export const monitoring = new ServerMonitoringService({
  authToken: MONITORING_TOOL__API_KEY,
  serviceName: MONITORING_TOOL__SERVICE_NAME,
  serviceVersion: MONITORING_TOOL__SERVICE_VERSION,
  serviceEnv: MONITORING_TOOL__SERVICE_ENV,
});

As you can see we are using here all our env variables. If any of these is not defined - the service will fall back to console.log and warn you there about it. Now you can just use it like

monitoring.info('Monitoring service initialized');

Init Tracing

We need to connect tracing as soon as possible during code, so it can be injected into all base modules for APM monitoring. Tracing module is available via:

const {
  initTracing,
} = require('@kilohealth/web-app-monitoring/dist/server/initTracing');

initTracing({
  serviceName: process.env.MONITORING_TOOL__SERVICE_NAME,
  serviceVersion: process.env.MONITORING_TOOL__SERVICE_VERSION,
  serviceEnv: process.env.MONITORING_TOOL__SERVICE_ENV,
  authToken: process.env.MONITORING_TOOL__API_KEY,
});

Example for Next.js:

const { PHASE_PRODUCTION_SERVER } = require('next/constants');
const {
  initTracing,
} = require('@kilohealth/web-app-monitoring/dist/server/initTracing');

module.exports = phase => {
  if (phase === PHASE_PRODUCTION_SERVER) {
    initTracing({
      serviceName: process.env.MONITORING_TOOL__SERVICE_NAME,
      serviceVersion: process.env.MONITORING_TOOL__SERVICE_VERSION,
      serviceEnv: process.env.MONITORING_TOOL__SERVICE_ENV,
      authToken: process.env.MONITORING_TOOL__API_KEY,
    });
  }
};

Note: In newer versions of Next.js there is experimental feature called instrumentationHook We can opt out from using undocumented PHASE_PRODUCTION_SERVER to use instrumentationHook for tracing init. There is also possibility to use NODE_OPTIONS='-r ./prestart-script.js ' next start instead. But there is an issue with pino-datadog-transport, which for performance reason spawns separate thread for log sending to data-dog and it this option seems to be passed to that process as well which triggers an infinite loop of require and initialization.

API

MonitoringService (both BrowserMonitoringService and ServerMonitoringService have these methods)

debug, info, warn

debug(message: string, context?: object)
info(message: string, context?: object)
warn(message: string, context?: object)

• message - any message to be logged
• context - object with all needed and related to the log entrance data

error

error(message: string, context?: object, error?: Error)

Same as above, but you can also optionally pass error instance as third parameter

reportError

reportError(error: Error, context?: object)

Shortcut for service.error(), which uses error.message field as message param for error method.

BrowserMonitoringService

constructor

constructor(
  remoteMonitoringServiceParams?: RemoteMonitoringServiceParams,
  remoteMonitoringServiceConfig?: RemoteMonitoringServiceConfig,
)

interface RemoteMonitoringServiceParams {
  serviceName?: string;
  serviceVersion?: string;
  serviceEnv?: string;
  authToken?: string;
}

• RemoteMonitoringServiceConfig - datadog params passed to init function. More info in docs
• serviceName - name of the service
• serviceVersion - version of the service
• serviceEnv - environment where service is deployed
• authToken - client token

ServerMonitoringService

constructor

constructor(
  remoteMonitoringServiceParams?: RemoteMonitoringServiceParams,
  remoteMonitoringServiceConfig?: RemoteMonitoringServiceConfig,
)

interface RemoteMonitoringServiceConfig {
  transportOptions?: Partial<TransportBaseOptions>;
  loggerOptions?: Partial<LoggerOptions>;
}

• transportOptions - pino-datadog-transport options
• loggerOptions - pino logger options

overrideLogger

Overrides logger passed as argument with monitoring logger. All methods of this logger will be overridden with corresponding methods of server monitoring.

overrideLogger(unknownLogger: UnknownLogger)

interface UnknownLogger {
  log?(...parts: unknown[]): void;
  debug?(...parts: unknown[]): void;
  info(...parts: unknown[]): void;
  warn(...parts: unknown[]): void;
  error(...parts: unknown[]): void;
}

overrideNativeConsole

Calls overrideLogger for native console.

overrideNativeConsole()

catchProcessErrors

Subscribes to unhandledRejection and uncaughtException events of the process to report error in such cases.

catchProcessErrors();

ServerMonitoringService

initServerMonitoring

Instantiate ServerMonitoringService with provided params and may also do additional work, depending on provided variables.

initServerMonitoring = (
  remoteMonitoringServiceParams?: RemoteMonitoringServiceParams,
  monitoringOptions?: MonitoringOptions,
  remoteMonitoringServiceConfig?: RemoteMonitoringServiceConfig
): ServerMonitoringService

interface MonitoringOptions {
  shouldOverrideNativeConsole?: boolean;
  shouldCatchProcessErrors?: boolean;
  globalMonitoringInstanceName?: string;
}

• RemoteMonitoringServiceParams and RemoteMonitoringServiceConfig are same as in constructor api
• shouldOverrideNativeConsole - if true, will call serverMonitoringService.overrideNativeConsole() under the hood
• shouldCatchProcessErrors - if true, will call serverMonitoringService.catchProcessErrors() under the hood
• globalMonitoringInstanceName - if provided with non-empty string will put instantiated serverMonitoringService into global scope under provided name.