Configuring datasets in wis2box

Learning outcomes

By the end of this practical session, you will be able to:

create a new dataset
create discovery metadata for a dataset
configure data mappings for a dataset
publish a WIS2 notification with a WCMP2 record
update and re-publish your dataset

Introduction

wis2box uses datasets that are associated with discovery metadata and data mappings.

Discovery metadata is used to create a WCMP2 (WMO Core Metadata Profile 2) record that is shared using a WIS2 notification published on your wis2box-broker.

The data mappings are used to associate a data plugin to your input data, allowing your data to be transformed prior to being published using the WIS2 notification.

This session will walk you through creating a new dataset, creating discovery metadata, and configuring data mappings. You will inspect your dataset in the wis2box-api and review the WIS2 notification for your discovery metadata.

Preparation

Connect to your broker using MQTT Explorer.

Instead of using your internal broker credentials, use the public credentials everyone/everyone:

MQTT Explorer: Connect to broker

Note

You never need to share the credentials of your internal broker with external users. The 'everyone' user is a public user to enable sharing of WIS2 notifications.

The everyone/everyone credentials has read-only access on the topic 'origin/a/wis2/#'. This is the topic where the WIS2 notifications are published. The Global Broker can subscribe with these public credentials to receive the notifications.

The 'everyone' user will not see internal topics or be able to publish messages.

Open a browser and open a page to http://<your-host>/wis2box-webapp. Make sure you are logged in and can access the 'dataset editor' page.

See the section on Initializing wis2box if you need to remember how to connect to the broker or access the wis2box-webapp.

Create an authorization token for processes/wis2box

You will need an authorization token for the 'processes/wis2box' endpoint to publish your dataset.

To create an authorization token, access your training VM over SSH and use the following commands to login to the wis2box-management container:

cd ~/wis2box-1.0b8
python3 wis2box-ctl.py login

Then run the following command to create a randomly generated authorization token for the 'processes/wis2box' endpoint:

wis2box auth add-token --path processes/wis2box

You can also create a token with a specific value by providing the token as an argument to the command:

wis2box auth add-token --path processes/wis2box MyS3cretToken

Make sure to copy the token value and store it on your local machine, as you will need it later.

Once you have your token, you can exit the wis2box-management container:

exit

Creating a new dataset in the wis2box-webapp

Navigate to the 'dataset editor' page in the wis2box-webapp of your wis2box instance by going to http://<your-host>/wis2box-webapp and selecting 'dataset editor' from the menu on the left hand side.

On the 'dataset editor' page, under the 'Datasets' tab, click on "Create New ...":

Create New Dataset

A pop-up window will appear, asking you to provide:

Centre ID : this is the agency acronym (in lower case and no spaces), as specified by the WMO Member, that identifies the data centre responsible for publishing the data.
Data Type: The type of data you are creating metadata for. You can choose between using a predefined template or selecting 'other'. If 'other' is selected, more fields will have to be manually filled.

Centre ID

Your centre-id should start with the TLD of your country, followed by a dash (-) and an abbreviated name of your organization (for example fr-meteofrance). The centre-id must be lowercase and use alphanumeric characters only. The dropdown list shows all currently registered centre-ids on WIS2 as well as any centre-id you have already created in wis2box.

Data Type Templates

The Data Type field allows you to select from a list of templates available in the wis2box-webapp dataset editor. A template will pre-populate the form with suggested default values appropriate for the data type. This includes suggested title and keywords for the metadata and pre-configured data plugins. The topic will be fixed to the default topic for the data type.

For the purpose of the training we will use the weather/surface-based-observations/synop data type which includes data plugins that ensure the data is transformed into BUFR format before being published.

If you want to publish CAP alerts using wis2box, use the template weather/advisories-warnings. This template includes a data plugin that verifies the input data is a valid CAP alert before publishing. To create CAP alerts and publish them via wis2box you can use CAP Composer.

Please choose a centre-id appropriate for your organization.

For Data Type, select weather/surface-based-observations/synop:

Create New Dataset Form: Initial information

Click continue to form to proceed, you will now be presented with the Dataset Editor Form.

Since you selected the weather/surface-based-observations/synop data type, the form will be pre-populated with some initial values related to this data type.

Creating discovery metadata

The Dataset Editor Form allows you to provide the Discovery Metadata for your dataset that the wis2box-management container will use to publish a WCMP2 record.

Since you have selected the 'weather/surface-based-observations/synop' data type, the form will be pre-populated with some default values.

Review the title and keywords, and update them as necessary, and provide a description for your dataset:

Metadata Editor: title, description, keywords

Note there are options to change the 'WMO Data Policy' from 'core' to 'recommended' or to modify your default Metadata Identifier, please keep data-policy as 'core' and use the default Metadata Identifier.

Next, review the section defining your 'Temporal Properties' and 'Spatial Properties'. You can adjust the bounding box by updating the 'North Latitude', 'South Latitude', 'East Longitude', and 'West Longitude' fields:

Metadata Editor: temporal properties, spatial properties

Next, fill out the section defining the 'Contact Information of the Data Provider':

Metadata Editor: contact information

Finally, fill out the section defining the 'Data Quality Information':

Once you are done filling out all the sections, click 'VALIDATE FORM' and check the form for any errors:

Metadata Editor: validation

If there are any errors, correct them and click 'VALIDATE FORM' again.

Making sure you have no errors and that you get a pop-up indication your form has been validated:

Metadata Editor: validation success

Next, before submitting your dataset, review the data mappings for your dataset.

Configuring data mappings

Since you used a template to create your dataset, the dataset mappings have been pre-populated with the defaults plugins for the 'weather/surface-based-observations/synop' data type. Data plugins are used in the wis2box to transform data before it is published using the WIS2 notification.

Data Mappings: update plugin

Note that you can click on the "update"-button to change settings for the plugin such as file-extension and the file-pattern, you can leave the default settings for now. In a later session, you will learn more about BUFR and the transformation of data into BUFR format.

Submitting your dataset

Finally, you can click 'submit' to publish your dataset.

You will need to provide the authorization token for 'processes/wis2box' that you created earlier. If you have not done so, you can create a new token by following the instructions in the preparation section.

Check that you get the following message after submitting your dataset, indicating that the dataset was successfully submitted:

Submit Dataset Success

After you click 'OK', you are redirected to the Dataset Editor home page. Now if you click on the 'Dataset' tab, you should see your new dataset listed:

Dataset Editor: new dataset

Reviewing the WIS2-notification for your discovery metadata

Go to MQTT Explorer, if you were connected to the broker, you should see a new WIS2 notification published on the topic origin/a/wis2/<your-centre-id>/metadata:

MQTT Explorer: WIS2 notification

Inspect the content of the WIS2 notification you published. You should see a JSON with a structure corresponding to the WIS Notification Message (WNM) format.

Question

On what topic is the WIS2 notification published?

Click to reveal answer

The WIS2 notification is published on the topic origin/a/wis2/<your-centre-id>/metadata.

Question

Try to find the title, description and keywords you provided in the discovery metadata in the WIS2 notification. Can you find them?

Click to reveal answer

Note that the title, description, and keywords you provided in the discovery metadata are not present in the WIS2 notification payload!

Instead, try to look for the canonical link in the "links"-section in the WIS2 notification:

WIS2 notification for metadata, links sections

The WIS2 notification contains a canonical link to the WCMP2 record that was published. If you copy-paste this link into a browser, you will download the WCMP2 record and see the title, description, and keywords you provided.

Conclusion

Congratulations!

In this practical session, you learned how to:

create a new dataset
define your discovery metadata
review your data mappings
publish discovery metadata
review the WIS2 notification for your discovery metadata