Skip to content

Commit

Permalink
feat(telemetry): Adding Scarf based telemetry to Superset (apache#26011)
Browse files Browse the repository at this point in the history
Co-authored-by: Michael S. Molina <[email protected]>
  • Loading branch information
rusackas and michael-s-molina authored Dec 14, 2023
1 parent 67468c4 commit 8437a23
Show file tree
Hide file tree
Showing 12 changed files with 181 additions and 24 deletions.
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -622,7 +622,7 @@ cd superset-frontend
npm ci
```

Note that Superset uses [Scarf](https://docs.scarf.sh) to capture telemetry/analytics about versions being installed, including the `scarf-js` npm package. As noted elsewhere in this documentation, Scarf gathers aggregated stats for the sake of security/release strategy, and does not capture/retain PII. [You can read here](https://docs.scarf.sh/package-analytics/) about the package, and various means to opt out of it, but one easy way to opt out is to add this setting in `superset-frontent/package.json`:
Note that Superset uses [Scarf](https://docs.scarf.sh) to capture telemetry/analytics about versions being installed, including the `scarf-js` npm package and an analytics pixel. As noted elsewhere in this documentation, Scarf gathers aggregated stats for the sake of security/release strategy, and does not capture/retain PII. [You can read here](https://docs.scarf.sh/package-analytics/) about the `scarf-js` package, and various means to opt out of it, but you can opt out of the npm package _and_ the pixel by setting the `SCARF_ANALYTICS` envinronment variable to `false` or opt out of the pixel by adding this setting in `superset-frontent/package.json`:

```json
// your-package/package.json
Expand Down
2 changes: 2 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,8 @@ services:
command: ["/app/docker/docker-frontend.sh"]
env_file: docker/.env
depends_on: *superset-depends-on
environment:
SCARF_ANALYTICS: "${SCARF_ANALYTICS}"
volumes: *superset-volumes

superset-worker:
Expand Down
6 changes: 4 additions & 2 deletions docs/docs/frequently-asked-questions.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -268,8 +268,10 @@ This can be used, for example, to convert UTC time to local time.
### Does Superset collect any telemetry data?

Superset uses [Scarf](https://about.scarf.sh/) by default to collect basic telemetry data upon installing and/or running Superset. This data helps the maintainers of Superset better understand which versions of Superset are being used, in order to prioritize patch/minor releases and security fixes.
We use the [Scarf Gateway](https://docs.scarf.sh/gateway/) to sit in front of container registries, and the [scarf-js](https://about.scarf.sh/package-sdks) package to track `npm` installations.
Scarf purges PII and provides aggregated statistics. Superset users can easily opt out of analytics in various ways documented [here](https://docs.scarf.sh/gateway/#do-not-track) and [here](https://docs.scarf.sh/package-analytics/#as-a-user-of-a-package-using-scarf-js-how-can-i-opt-out-of-analytics). Additional opt-out instructions for Docker users are available on the [Docker Installation](https://superset.apache.org/docs/installation/installing-superset-using-docker-compose) page.
We use the [Scarf Gateway](https://docs.scarf.sh/gateway/) to sit in front of container registries, the [scarf-js](https://about.scarf.sh/package-sdks) package to track `npm` installations, and a Scarf pixel to gather anonymous analytics on Superset page views.
Scarf purges PII and provides aggregated statistics. Superset users can easily opt out of analytics in various ways documented [here](https://docs.scarf.sh/gateway/#do-not-track) and [here](https://docs.scarf.sh/package-analytics/#as-a-user-of-a-package-using-scarf-js-how-can-i-opt-out-of-analytics).
Superset maintainers can also opt out of telemetry data collection by setting the `SCARF_ANALYTICS` environment variable to `false` in the Superset container (or anywhere Superset/webpack are run).
Additional opt-out instructions for Docker users are available on the [Docker Installation](https://superset.apache.org/docs/installation/installing-superset-using-docker-compose) page.

### Does Superset have an archive panel or trash bin from which a user can recover deleted assets?

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,9 @@ Users often want to connect to other databases from Superset. Currently, the eas
:::note
Superset uses [Scarf Gateway](https://about.scarf.sh/scarf-gateway) to collect telemetry data. Knowing the installation counts for different Superset versions informs the project's decisions about patching and long-term support. Scarf purges personally identifiable information (PII) and provides only aggregated statistics.

To opt-out of this data collection in your docker compose based installation, edit the `x-superset-image:` line in your `docker-compose.yml` and `docker-compose-non-dev.yml` files, replacing `apachesuperset.docker.scarf.sh/apache/superset` with `apache/superset` to pull the image directly from Docker Hub.
To opt-out of this data collection for packages downloaded through the Scarf Gateway by your docker compose based installation, edit the `x-superset-image:` line in your `docker-compose.yml` and `docker-compose-non-dev.yml` files, replacing `apachesuperset.docker.scarf.sh/apache/superset` with `apache/superset` to pull the image directly from Docker Hub.

To disable the Scarf telemetry pixel, set the `SCARF_ANALYTICS` environment variable to `False` in your terminal and/or in your `docker/.env` and `docker/.env-non-dev` files.
:::

### 4. Log in to Superset
Expand Down
42 changes: 33 additions & 9 deletions superset-frontend/package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion superset-frontend/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -259,7 +259,7 @@
"@types/js-levenshtein": "^1.1.0",
"@types/json-bigint": "^1.0.1",
"@types/mousetrap": "^1.6.11",
"@types/react": "^16.9.43",
"@types/react": "^16.9.53",
"@types/react-dom": "^16.9.8",
"@types/react-gravatar": "^2.6.8",
"@types/react-json-tree": "^0.6.11",
Expand Down
16 changes: 8 additions & 8 deletions superset-frontend/spec/helpers/shim.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -48,14 +48,14 @@ if (defaultView != null) {
}

const g = global as any;
g.window = g.window || {};
g.window.location = { href: 'about:blank' };
g.window.performance = { now: () => new Date().getTime() };
g.window.Worker = Worker;
g.window.IntersectionObserver = IntersectionObserver;
g.window.ResizeObserver = ResizeObserver;
g.window.featureFlags = {};
g.URL.createObjectURL = () => '';
g.window ??= Object.create(window);
g.window.location ??= { href: 'about:blank' };
g.window.performance ??= { now: () => new Date().getTime() };
g.window.Worker ??= Worker;
g.window.IntersectionObserver ??= IntersectionObserver;
g.window.ResizeObserver ??= ResizeObserver;
g.window.featureFlags ??= {};
g.URL.createObjectURL ??= () => '';
g.caches = new CacheStorage();

Object.defineProperty(window, 'matchMedia', {
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
import React from 'react';
import { render } from 'spec/helpers/testing-library';
import TelemetryPixel from '.';

const OLD_ENV = process.env;

// restor the process after messing with it!
afterAll(() => {
process.env = OLD_ENV;
});

test('should render', () => {
const { container } = render(<TelemetryPixel />);
expect(container).toBeInTheDocument();
});

test('should render the pixel link when FF is on', () => {
process.env.SCARF_ANALYTICS = 'true';
render(<TelemetryPixel />);

const image = document.querySelector('img[src*="scarf.sh"]');
expect(image).toBeInTheDocument();
});

test('should NOT render the pixel link when FF is off', () => {
process.env.SCARF_ANALYTICS = 'false';
render(<TelemetryPixel />);

const image = document.querySelector('img[src*="scarf.sh"]');
expect(image).not.toBeInTheDocument();
});
59 changes: 59 additions & 0 deletions superset-frontend/src/components/TelemetryPixel/index.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

import React from 'react';

interface TelemetryPixelProps {
version?: string;
sha?: string;
build?: string;
}

/**
* Renders a telemetry pixel component to capture anonymous, aggregated telemetry via Scarf.
* This can be disabled by setting the SCARF_ANALYTICS environment variable to false.
*
* @component
* @param {TelemetryPixelProps} props - The props for the TelemetryPixel component.
* @param {string} props.version - The version of Superset that's currently in use.
* @param {string} props.sha - The SHA of Superset that's currently in use.
* @param {string} props.build - The build of Superset that's currently in use.
* @returns {JSX.Element | null} The rendered TelemetryPixel component.
*/

const PIXEL_ID = '0d3461e1-abb1-4691-a0aa-5ed50de66af0';

const TelemetryPixel = ({
version = 'unknownVersion',
sha = 'unknownSHA',
build = 'unknownBuild',
}: TelemetryPixelProps): React.ReactElement | null => {
const pixelPath = `https://apachesuperset.gateway.scarf.sh/pixel/${PIXEL_ID}/${version}/${sha}/${build}`;
return process.env.SCARF_ANALYTICS === 'false' ||
process.env.SCARF_ANALYTICS === 'false' ? null : (
<img
referrerPolicy="no-referrer-when-downgrade"
src={pixelPath}
width={0}
height={0}
alt=""
/>
);
};
export default TelemetryPixel;
6 changes: 6 additions & 0 deletions superset-frontend/src/features/home/RightMenu.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ import {
import { RootState } from 'src/dashboard/types';
import DatabaseModal from 'src/features/databases/DatabaseModal';
import { uploadUserPerms } from 'src/views/CRUD/utils';
import TelemetryPixel from 'src/components/TelemetryPixel';
import LanguagePicker from './LanguagePicker';
import {
ExtensionConfigs,
Expand Down Expand Up @@ -562,6 +563,11 @@ const RightMenu = ({
{t('Login')}
</StyledAnchor>
)}
<TelemetryPixel
version={navbarRight.version_string}
sha={navbarRight.version_sha}
build={navbarRight.build_number}
/>
</StyledDiv>
);
};
Expand Down
1 change: 1 addition & 0 deletions superset-frontend/webpack.config.js
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,7 @@ const plugins = [
'process.env.WEBPACK_MODE': JSON.stringify(mode),
'process.env.REDUX_DEFAULT_MIDDLEWARE':
process.env.REDUX_DEFAULT_MIDDLEWARE,
'process.env.SCARF_ANALYTICS': process.env.SCARF_ANALYTICS,
}),

new CopyPlugin({
Expand Down
16 changes: 14 additions & 2 deletions superset/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -1427,7 +1427,13 @@ def EMAIL_HEADER_MUTATOR( # pylint: disable=invalid-name,unused-argument
"content_security_policy": {
"base-uri": ["'self'"],
"default-src": ["'self'"],
"img-src": ["'self'", "blob:", "data:"],
"img-src": [
"'self'",
"blob:",
"data:",
"https://apachesuperset.gateway.scarf.sh",
"https://static.scarf.sh/",
],
"worker-src": ["'self'", "blob:"],
"connect-src": [
"'self'",
Expand All @@ -1450,7 +1456,13 @@ def EMAIL_HEADER_MUTATOR( # pylint: disable=invalid-name,unused-argument
"content_security_policy": {
"base-uri": ["'self'"],
"default-src": ["'self'"],
"img-src": ["'self'", "blob:", "data:"],
"img-src": [
"'self'",
"blob:",
"data:",
"https://apachesuperset.gateway.scarf.sh",
"https://static.scarf.sh/",
],
"worker-src": ["'self'", "blob:"],
"connect-src": [
"'self'",
Expand Down

0 comments on commit 8437a23

Please sign in to comment.