Ingestion
You are able to add, edit, delete, save and run Ingestion Processes depending on the settings of the Project.
An Ingestion Process forms the connection between a Linked Data Resource and your Data Model in nodegoat. The URI, Label, and Values can be mapped to Object Descriptions, Sub-Objects, or Sub-Object Descriptions in your Data Model. Running an Ingestion Process will write URIs, Labels, and Values to the mapped Object Descriptions, Sub-Objects, or Sub-Object Descriptions.
Add / Edit an Ingestion Process
Specify the name of the Ingestion Process. This name is used to identify the Ingestion Process.
Select a Linked Data Resource as the 'Source' for the Ingestion Process.
Select an Object Type or Classification as the 'Target' of the Ingestion Process. If the mode'Add New Objects (If Not Exists)' or 'Update Existing Objects' is selected: click the blue 'filter' button to specify a Filter. Only filtered Objects or Categories will be used.
Specify the 'Mode' of the Ingestion Process. The mode 'Add New Objects' allows you to enter a new Object for every returned element. The mode 'Add New Objects (If Not Exists)' allows you to enter a new Object for every returned element only if the specified identifier is not present yet. The mode 'Update Existing Objects' allows you to update Objects that are present in your nodegoat domain.
Query / Filter External Resource By Value
Limit the returned results by specifying a Query using variables defined in the Linked Data Resource Query or a Filter based on the values defined in the Linked Data Resource.
| Query | Use the dropdown menu to select a variable that has been defined the Linked Data Resource Query. Enter a value in the input field that will be used in the query. Click 'add' to add additional variables. Click 'del' to remove empty variables. |
| Filter | Use the dropdown menu to select a value that has been defined the Linked Data Resource. Enter a value in the input field that will be used in the query. Click 'add' to add additional values. Click 'del' to remove empty values. |
Link External Resource To Object
This section is only relevant for the mode 'Add New Objects (If Not Exists)' and 'Update Existing Objects'. When adding or updating Objects in nodegoat by means of ingesting data, the Ingestion Process needs to be configured to identify the Objects that will be added or updated. Objects can be identified by an Object ID, by a filter, or by a query.
| Object URI | Use the dropdown menu to select an Object Description that is designated as Object identifier. This option is only available if the target Type contains at least one Object Descriptions that is designated as Object identifier. |
| Filter | Use the dropdown menu to select a value that has been defined the Linked Data Resource. Use the next dropdown menu to select the target Object Description or Sub-Object Description. Click 'add' to add additional values. Click 'del' to remove empty values. |
| Query | Use the dropdown menu to select a variable that has been defined the Linked Data Resource Query. Use the next dropdown menu to select the target Object Description or Sub-Object Description. Click 'add' to add additional values. Click 'del' to remove empty variables. |
Query External Resource By Object Value
This section is only relevant for the mode 'Add New Objects' and 'Add New Objects (If Not Exists)'. Use this setting to run the query for every (filtered) Object or Category. A value stored in an Object Description or Sub-Object Description can be used as a variable in the Linked Data Resource Query.
| Use | Use the dropdown menu to select an Object Type or Classification. Click the blue 'filter' button to specify a Filter. Only filtered Objects or Categories will be used. |
| Query | Use the dropdown menu to select a variable that has been defined the Linked Data Resource Query. Use the next dropdown menu to select an Object Description or Sub-Object Description. Click 'add' to add additional values. Click 'del' to remove empty variables. |
Map External Resource To Data Model
The URI, Label and every Value of the Linked Data Resource can be mapped to one ore more Object Descriptions, Sub-Objects, or Sub-Object Descriptions. Click 'add' to add additional values. Click 'del' to remove empty values.
Use the first dropdown menu to select the URI, Label, or a value.
Optional: Use input field to specify a character to split the value into multiple values. Once a character has been entered you can use the dropdown menu that appears to either select a fixed position (1 to 5) of the multiple values, or select the 'multiple' option to store all the values.
Use the next dropdown menu to select the target Object Description, Sub-Object, or Sub-Object Description.
Two dropdown menus will appear if the target Object Description or Sub-Object Description is a reference. The first (disabled) dropdown menu indicates the referenced Object Type or Classification. The second dropdown menu allows you to select an element of the referenced Object Type or Classification in which the value is stored that is to be used to make the reference. Use this option when ingesting a relational dataset from an external source.
Three dropdown menus will appear if the target is a Location Reference. The first dropdown menu allows you to select Object Type of the Location Reference. The second dropdown menu allows you to select the Sub-Object of the Location Reference. The third dropdown menu allows you to select an element of the referenced Object Type in which the value is stored that is to be used to make the reference. Use this option when ingesting a relational dataset from an external source.
If the mode of the Ingestion Process is 'Update Existing Objects', you can click 'more' to specify additional options.
| Overwrite | Existing data will be overwritten. |
| Append | New data will be appended to existing data. This option is only available for Descriptions that have been configured as strings, integers, or texts, or Descriptions that have the 'Multiple' option enable (see the section on Object Descriptions). |
| Ignore when Empty | Select this option in combination with 'Overwrite' to only overwrite existing data with new data and take no action when a returned value is empty. This ensures that existing data is not overwritten by blanks. |
| Ignore when Identical | Select this option to take no action when new data is identical to existing data. |
Run an Ingestion Process
To run an Ingestion Process, click the green 'run' button at the relevant Ingestion Process. In the screen that follows you will be asked to select a Linked Data Resource as the source of the Ingestion Process. The Linked Data Resource you used to create the Ingestion Process will be selected by default.
If an Ingestion Process has been intermitted or has been completed, the amount of processed Objects will be displayed. This number can be reset to 0 by clicking the red 'reset' button.
Click 'Run Ingestion' to run the Ingestion Process.
If the Ingestion Process does not contain references, you will be shown the result of the run.
If the Ingestion Process contains references, these references will be checked. Unambiguous references will be stored as String to Object Pairs. References that produce no or more than one result will be returned. You are asked to resolve any unmatched references. Once all unmatched references have been resolved, you can click 'Next' to continue. Matched references will be stored as String to Object Pairs.