Ingest (Upload) Data Files
Last updated
Was this helpful?
Last updated
Was this helpful?
Currently, only csv files can be ingested by TDO. These files must meet some simple rules:
· Each file must contain column headers in the first row, and at least one row of data.
· The column header field names must be lower case only, and use underscores instead of spaces.
· Files cannot contain commas. This is due to rules governing csv files, which interpret a comma as a field delimiter. Including a comma in a field will cause it to be split into two fields upon ingestion and all subsequent fields in that row will be out of sync with the column headers.
Data upload is sent as two separate API’s, first a POST request to load the data and create an object ID in TDO and then a PUT request to provide the object names in TDO.
There are two ways to send the data file to TDO - by pasting it in the body of the POST request, or by identifying the path/file in the body of the post request.
Regardless of the method used to move the data into TDO, the POST request must always be followed by a PUT request that links the uploaded data to a specific project, file name, and data block name.
The file contents are the body of the message for the POST request, and for the PUT request the body must contain the file name and the data block name. If this is a new upload, that is all that is needed. If this is a refresh of an existing data block then you may also include a third line in the body, with a version number for the existing data block that is to be updated. If you do not include a version number, TDO will assign the the next sequence number available (i.e., 1.0 for a new data block name, 1.n for an existing data block).
The POST request format is http://<server:port>/core/1.0/API/upload.
The return message contains the UUID for the object as well as the sample for the second API.
Identifying the path/file in the POST request
The format of the request remains the same (URL). In your automation tool (Postman is used here to demonstrate) select 'binary' as the format for the body of the request. A 'select file' box will be displayed. Click on the '+' sign next to 'New file from local machine' and a file explorer/file selection window will open. Navigate to the desired file, highlight it, and click 'Open' and the file name will populate in the window.
Note that any automation program that supports binary as an input option in the body (e.g., Insomnia, Postman, etc.) will support this process.
Click 'Send' to run this call, and you will receive the same return message as shown above.
The second API URL is http://<server:port>/core/1.0/API/project/<project-name>/<project-version>/upload/<file-id>.
In Postman, the header line, body, and response for a new file put will look like the below example. TDO will automatically version the upload - if this data block ('test3_block') already exists as version 1.0, TDO will assign version 1.1 to your uploaded data. If this data block name is new in the project, TDO will assign version 1.0.
After this is run, you can see the resulting data block in TDO. You will not be able to see the data block in TDO until this call is completed; this is what associates the data to a specific project and data block name.
The contents of the data block can be viewed in TDO.
