Skip to content

Latest commit

 

History

History
408 lines (328 loc) · 20.4 KB

README.md

File metadata and controls

408 lines (328 loc) · 20.4 KB
Raw

MAGE Web Service & Web App

The Mobile Awareness GEOINT Environment, or MAGE platform, provides mobile situational awareness and data collection capabilities. This project comprises the ReST API web service the MAGE client apps use to submit and fetch MAGE data, as well as the browser-based web app. The MAGE web app provides user interfaces to view and edit MAGE observations similar to the MAGE mobile apps, and additionally provides the administrative UI to manage the MAGE server settings, access control, events, data collection forms, etc.

MAGE was developed at the National Geospatial-Intelligence Agency (NGA) in collaboration with BIT Systems. The government has "unlimited rights" and is releasing this software to increase the impact of government investments by providing developers with the opportunity to take things in new directions. The software use, modification, and distribution rights are stipulated within the Apache license.

Technology stack

MAGE is built using the MEAN stack. The components of the MEAN stack are as follows:

  • MongoDB - a NoSQL JSON document database
  • Express.js - a web server framework for Node to handle ReST API requests
  • Angular - a JavaScript MVC framework for web app front-ends
  • Node.js - a software platform for scalable server-side and networking applications.

The MAGE server code is TypeScript for strong typing, which transpiles to JavaScript that runs on Node.js. At the time of this writing, the transition of legacy JavaScript to TypeScript is still in progress.

Project structure

The MAGE server project is essentially a monorepo with several NPM package components that assemble into a running server instance.

service

The service directory contains the @ngageoint/mage.service package. This is the backend ReST web service that the web and mobile apps consume.

web-app

The web-app directory contains the @ngageoint/mage.web-app package. The package is the bundled, Angular-based web app MAGE client that includes standard user functionality as well as access administrative functions.

core-lib

The core-lib directory is a descendant of the web-app directory that contains the @ngageoint/mage.web-core-lib package. The package is an Angular library that includes shared elements that both the web app uses, and that web plugins can use to add custom UI elements to the web app.

instance

The instance directory is a development instance of the MAGE server whose dependencies are the relative paths to the other packages in the project. This is useful as an example of how to assemble and configure a MAGE server instance, as well as to run and test the server during development.

plugins

The plugins directory contains various plugin packages that the MAGE team maintains as part of the MAGE server open source project. Some of these are automatically bundled with MAGE server releases, and some serve as examples and/or development utilities.

Running a MAGE server

The MAGE server Node.js app is generally intended to run on Unix-like platforms. The server should run on Windows, but be aware some path-separator related bugs may exist

Install Node.js

The MAGE server is a Node.js application, so of course you'll need to install Node on your platform of choice. Node Version Manager is a nice tool to use for installing and managing different versions of Node, as opposed to various package managers. At the time of this writing, MAGE requires Node > 18.x. Developers should use the latest LTS, 20.x at the time of this writing.

Install MongoDB

Before running a MAGE server, you'll need to install and start MongoDB. At the time of this writing, MAGE supports MongoDB version 4.x (4.4).

Install MAGE server packages

Starting with release 6.2.2, the MAGE server packages are available from the NPM registry.

mkdir mage
cd mage
npm install --omit dev \
@ngageoint/mage.service \
@ngageoint/mage.web-app \
@ngageoint/mage.image.service \

That will yield a package.json file that looks something like

{
  "dependencies": {
    "@ngageoint/mage.image.service": "^1.0.4",
    "@ngageoint/mage.service": "^6.2.12",
    "@ngageoint/mage.web-app": "^6.2.12"
  }
}

as well as a package-lock.json file and node_modules directory containing all of the MAGE server's dependencies.

Register plugins

As the example instance configuration demonstrates, you'll need to tell the MAGE service what plugins to load. See the plugins entry in the configuration object, as well as the plugins readme. Note that the @ngageoint/mage.image.service package in the dependency list above is a plugin package, and resides in this monorepo.

Run mage.service script

The @ngageoint/mage.service package includes a [mage.service bin script for starting the server process. From the instance directory, you can run npx @ngageoint/mage.service --help to see the configuration options. Also, the instance directory in this project has an example configuration script which the mage.service script would load with the -C flag, e.g., mage.service -C config.js.

On Windows Server installations, running the command node node_modules\@ngageoint\mage.service\bin\mage.service.js in the instance directory will execute the mage.service script.

Configuration merging

Because the mage.service script uses the Commander library, you can set most of the configuration options with environment variables and/or command line switches. You can also pass options from a JSON file or JSON string literal via the -C (or --config, or MAGE_CONFIG environment variable) command line switch. The mage.service script merges options from command line switches, environment variables, and JSON object, in descending order of precedence. This enables you to have a base configuration file, then override the file's entries with environment variables and/or command line switches as desired. The full configuration object for the -C/MAGE_CONFIG environment variable must have the following form.

{
  "mage": {
    "address": "0.0.0.0",
    "port": 1234,
    // etc.
  }
}

The -C option can be a path to a .json file that contains the configuration object, or a Node JavaScript module whose export is a configuration object,

module.exports = {
  mage: {
    // config options ...
  }
}

or a literal JSON value.

mage.service -C /mage/config.json
# or
mage.service -C /mage/config.js
# or
mage.service -C '{ "mage": { "mongo": { "connTimeout": 5000 }}}'

To see your full effective configuration, add the --show-config switch to the command, and the mage.service will print the configuration as a JSON string and exit without starting the server.

<MAGE_*=value...> npx @ngageoint/mage.service <options...> --show-config

By default, the MAGE server will attempt to create and use a directory at /var/lib/mage for storing data and media such as videos, photos, and icons. If the system user account that runs the MAGE server does not have permission to create that directory, you must create it before starting the server. Of course you can change the directories the server uses through the script command line switches and/or environment variables.

For convenience, the MAGE server project contains an environment script that you can copy and customize. You can configure the MAGE system user account to source the script at login.

The Node MAGE server runs on port 4242 by default. You can access the MAGE web app in your web browser at http://127.0.0.1:4242 if you are running MAGE locally.

Running with Docker

Refer to the Docker README for details on running the MAGE server using Docker.

Production notes

When running a publicly accessible production server, consider the following points.

Configuration location

If all of your MAGE server configuration options come from environment variables, as should be the case with most cloud server deployments, you will not need to worry about the location of a configuration file. If you are using a JSON or JavaScript module configuration file, be sure to the store the file in a location outside where you have install the MAGE server packages. For example, if you installed the MAGE server in /opt/mage, keep your configuration in some non-overlapping directory like /etc/mage.json. That way, if you decide to delete the contents of /opt/mage and start fresh, your configuration will remain intact.

Running with forever

Use a tool like forever to run the MAGE server process as a daemon in a production environment. forever will restart the MAGE server process if it happens to terminate unexpectedly. First, you'll need to install forever.

npm install -g forever

To run the MAGE server with forever, you can start the server like the following, assuming you are in the directory where you have installed the MAGE server packages.

forever start ./node_modules/.bin/mage.service <...options>

NOTE: The forever readme now indicates that the project no longer has a dedicated developer and is totally reliant on community updates. Try using the newer tools pm2 or nodemon for running mage.service persistently in production.

Running as a Windows Service

To continuously run mage.service on a windows environment, it is recommended to create a windows service using a tool such as node-windows.

First you'll need to install node-windows

npm install -g node-windows

Then a script is needed to configure and create the windows service.

var Service = require('node-windows').Service;

// Create a new service object
var svc = new Service({
  name:'Mage Service',
  description: 'To run MAGE Server as a windows service',
  script: '{MAGE Install Directory}\\node_modules\\@ngageoint\\mage.service\\bin\\mage.service.js',
  nodeOptions: [
  ]
  //, workingDirectory: '...'
  //, allowServiceLogon: true
});

// Listen for the "install" event, which indicates the
// process is available as a service.
svc.on('install',function(){
  svc.start();
});

svc.install();

Upon running the created script, a new Windows service will be created matching the name given. This service can then be configured to automatically start by the windows server.

HTTPS/TLS

When running a MAGE server in a publicly accessible production environment, as with any web application, you should use a reverse proxy such as Apache HTTP Server or NGINX to force external connections to use HTTPS and run the MAGE server Node process on a private subnet host. The MAGE server Node application itself does not support HTTPS/TLS connections.

IMPORTANT: Be sure your reverse proxy properly sets the X-Forwarded-Host and X-Forwarded-Proto headers properly. The MAGE Node.js app builds URLs, via Express.js, that MAGE clients use to request resources.

For Microsoft ISS installations, in addition to running a reverse proxy, you must configure ISS to preserve host headers. To do this, execute the following command:

\Windows\System32\inetsrv\appcmd.exe set config -section:system.webServer/proxy -preserveHostHeader:true /commit:apphost

Cloud Foundry deployment

MAGE uses the cfenv Node module to read settings from Cloud Foundry's environment variables. If Cloud Foundry's environment variables are present, they will take precedence over any of their counterparts derived from the magerc.sh file. This pertains mostly to defining the connection to the MongoDB server as a bound service in Cloud Foundry, for which Cloud Foundry should supply the connection string and credentials in the VCAP_SERVICES value.

Upgrading MAGE server

Upgrading the MAGE server essentially consists of the same process as installing for the first time.

  1. As above, install the desired versions of the packages.
  2. Stop your current MAGE server if it is running.
  3. BACK UP YOUR DATABASE! (You already do that regularly, right?)
  4. Start your new MAGE server, which will automatically run any database migrations present in the new version.

Building from source

First, clone the MAGE server GitHub repository, or download a release source tarball and extract the contents to an empty directory, such as mage-server. The project has a monorepo structure. The main packages to build are @ngageoint/mage.service and @ngageoint/mage.web-app. There are more optional packages in the plugins directory. The instance package is an example of assembling all the packages into a running MAGE server instance.

First, build the service package.

cd service
npm ci
npm run build

Then build the web-app package.

cd web-app
npm ci
npm run build

Build optional plugin packages similarly.:

cd plugins/nga-msi
npm ci
npm link ../../service # **IMPORTANT** see below
npm run build

After building the core packages, install them as dependencies in the instance package.

cd instance
npm i

In the case that the dev dependencies need to be over ridden (eg nga-msi plugin)

cd instance
npm i --omit dev ../service ../web-app/dist ../plugins/nga-msi

The project's root package.json provides some convenience script entries to install, build, and run the MAGE server components, however, those are deprecated and will likely go away after migrating to NPM 7+'s workspaces feature.

Running from source

To run the Mage server directly from your built source tree, build the service, web-app, and any plugin packages you want to run as described in the above section. Then, from the instance directory, run

npm run start:dev

That NPM script will run the mage.service script from your working tree using the configuration from the instance directory. You can modify that configuration to suit your needs.

Local runtime issues

You may run into some problems running the Mage instance from your working tree due to NPM's dependency installation behavior and/or Node's module resolution algorithm. For example, if you are working on a plugin within this core Mage repository, you may see errors as plugins initialize. This is usually a null reference error that looks something like

2024-08-23T02:42:52.783Z - [mage.image] intializing image plugin ...
...
/<...>/mage-server/plugins/image/service/lib/processor.js:53
            return yield stateRepo.get().then(x => !!x ? x : stateRepo.put(exports.defaultImagePluginConfig));
                                   ^

TypeError: Cannot read properties of undefined (reading 'get')
    at /<...>/mage-server/plugins/image/service/lib/processor.js:53:36

This is usually because the plugin package has a peer depedency on @ngageoint/mage.service, which NPM pulls from the public registry and installs into the plugin's node_modules directory. However, your local Mage instance references @ngageoint/mage.service package from the local relative path. This results in your instance having two copies of @ngageoint/mage.service - one from your local build linked in the top-level instance/node_modules directory, and one from the registry in the plugin's node_modules directory. In the case of the error above, this results in a discrepancy during dependency injection because the Mage service defines unique Symbol constants for plugins to indicate which elements they need from their host Mage service. In the plugin's modules, Node resolves these symbol constants and any other core @ngageoint/mage.service modules from the plugin's copy of the package, as opposed to the relative package installed at the instance level. This is why you must ensure that you link the working tree core Mage service package in your plugin working tree, as the above instructions state.

~/my_plugin % npm ci
~/my_plugin % npm link <relative path to mage server repo>/service

Be aware that NPM's dependency resolution will delete this symbolic link every time you run npm install, npm install <dependency>, or npm ci for the plugin, so always npm link your relative Mage service dependency again after those commands.

If you encounter other unexpected issues running locally with plugins, especially reference errors, or other discrepancies in values the core modules define, check that the core service package is still linked properly in your plugin working tree. You can check using the npm ls command as follows.

% npm ls @ngageoint/mage.service
@ngageoint/[email protected] /<...>/mage-server/plugins/image/service
└── @ngageoint/[email protected] -> ./../../../service

ReST API

The MAGE ReSTful API is documented using OpenAPI. A MAGE server instance includes a Swagger UI page that renders a web app from the MAGE OpenAPI document. The Swagger UI page is handy for testing the ReST API operations individually. The About page in the MAGE web app has a link to the Swagger UI. After logging in to the web app, the Swagger UI will automatically use the authentication token from your login to authenticate ReST API requests. Be mindful that the SwaggerUI is interacting with your server's data, so use caution when trying POST/PUT/DELETE operations that mutate data.

Code generation

You can use the MAGE server's OpenAPI document to generate an HTTP client that can consume the API. Swagger and many other tools exist to generate client stubs based on OpenAPI. OpenAPI.Tools is a good place to start.

Android & iOS client apps

The MAGE team develops Android and iOS apps that interact with the ReST API. The apps are open source and available under the Apache License for anyone to use. Check them out if you are considering mobile capabilities.

Pull requests

If you'd like to contribute to this project, please make a pull request. We'll review the pull request and discuss the changes. All pull request contributions to this project will be released under the Apache license.

Software source code previously released under an open source license and then modified by NGA staff is considered a "joint work" (see 17 USC § 101); it is partially copyrighted, partially public domain, and as a whole is protected by the copyrights of the non-government authors and must be released according to the terms of the original open source license.

License

Copyright 2015 BIT Systems

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.