Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Schema: Parsing HEALPix Maps #1205

Closed
wants to merge 1 commit into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions app/routes/_gcn.docs.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,9 @@ export default function () {
<NavLink key="schema" to="notices/schema">
Unified Schema
</NavLink>,
<NavLink key="attachments" to="producers/attachments">
Attachments
</NavLink>,
<NavLink key="archive" to="notices/archive">
Archive
</NavLink>,
Expand Down
87 changes: 87 additions & 0 deletions app/routes/docs.producers._index.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
---
handle:
breadcrumb: New Notice Producers
---

import {
ProcessList,
ProcessListHeading,
ProcessListItem,
} from '@trussworks/react-uswds'

# New Notice Producers

The following steps guide new instrument, mission, or observatory producers
into setting up new notices streams that are distributed to the user community
via [Kafka](faq#what-is-kafka). This process requires interaction
with the [GCN Team](https://heasarc.gsfc.nasa.gov/cgi-bin/Feedback?selected=kafkagcn)
to enable accounts and Kafka topics creation on the GCN Kafka broker. The GCN Team is
also happy to work with the mission teams to help construct your alerts.

## Start Producing Alerts

<ProcessList>
<ProcessListItem>
<ProcessListHeading type="h3">
Sign in / Sign up
</ProcessListHeading>
Decide which of your team members will have programmatic access to produce your alerts.
Make sure that they have all signed in at least once to the [GCN website](https://gcn.nasa.gov/login)
and the [GCN test website](https://test.gcn.nasa.gov/login).
</ProcessListItem>
<ProcessListItem>
<ProcessListHeading type="h3">
Name Your Kafka Topics
</ProcessListHeading>
Names of Kafka topics follow the format <code>gcn.notices.<i>mission</i>.<i>notice_type</i></code>.
Pick a prefix for your Kafka topic names, <code><i>mission</i>.\*</code>.
</ProcessListItem>
<ProcessListItem>
<ProcessListHeading type="h3">
Contact the GCN Team
</ProcessListHeading>
Send the [GCN Team](/contact)
your list of team members from Step 1 and your chosen Kafka topic prefix from Step 2.
The GCN Team will reply after they have configured producer permissions for your team.
</ProcessListItem>
<ProcessListItem>
<ProcessListHeading type="h3">
Build Producer Code
</ProcessListHeading>
- Log out and log back in.
- Go through the [Start Streaming GCN Notices](/quickstart) process.
- On Step 2, choose the scope <code>gcn.nasa.gov/kafka-<i>mission</i>-producer</code>.
- Your producer code will look very similar to the [client example code](client) and
Step 4 of [Start Streaming GCN Notices](/quickstart). `client_id` and `client_secret`
can be found in Step 4 client example code.
- Start from this and adjust the `client_id`, `client_secret`, `topic` and `data` content:

```python
from gcn_kafka import Producer
# Connect as a producer.
# Warning: don't share the client secret with others.
producer = Producer(client_id='fill me in', client_secret='fill me in')
# any topic starting with 'mission.'
topic = 'gcn.notices.mission.example'
data = b'...' # any bytes
producer.produce(topic, data)
```

</ProcessListItem>
<ProcessListItem>
<ProcessListHeading type="h3">
Create or Update the Mission Page
</ProcessListHeading>
Create a new mission page by submitting a [pull request](https://github.com/nasa-gcn/gcn.nasa.gov/pulls)
or by sending text to the [GCN Team](https://heasarc.gsfc.nasa.gov/cgi-bin/Feedback?selected=kafkagcn).
</ProcessListItem>

<ProcessListItem>
<ProcessListHeading type="h3">
Announce New Notice Types
</ProcessListHeading>
Work with the
[GCN Team](/contact)
to draft a community announcement, which the GCN Team will circulate.
</ProcessListItem>
</ProcessList>
51 changes: 51 additions & 0 deletions app/routes/docs.producers.attachments.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
---
handle:
breadcrumb: Advanced Documentation
---

import { Highlight } from '~/components/Highlight'

# Attachments

## Parsing HEALPix maps

TODO: Leo/Israel?:Instructions on how to create multiresolution healpix maps.

## File Encoding and Decoding

The following code sample shows how to encode/decode a file in python. The `base64` package includes the methods `b64decode` and `b64encode` to make this task simple.

```python
import base64

# Parse the content of your file to a base64 encoded string:
with open("path/to/your/file", 'rb') as file:
encoded_string = base64.b64encode(file.read())

print(encoded_string)
# b'a1512dabc1b6adb3cd1b6dcb6d4c6......'

# Decode and write the content to a local file:
with open("path/to/destination/file", 'wb') as file:
file.write(base64.b64decode(encoded_string))

```

For example, if we wanted to include a FITS file in a notice, you would need to add a property to your schema definition and define the value to be the following:

<Highlight
language="json"
code={JSON.stringify(
{
type: 'string',
contentEncoding: 'base64',
contentMediaType: 'image/fits',
},
null,
2
)}
/>

Then in your data production pipeline, you can use the encoding steps to convert your file to a bytestring, and set the value of the property to said bytestring. See [non json data](https://json-schema.org/understanding-json-schema/reference/non_json_data.html) for more information

The following is a rough example of what that would look like in your producer code. We will assume for this example that the schema is defined such that the encoded data property is named `fits_data`:
Loading