HTTP REST Collection Source

This connector is by far the most used, it is a generic REST api connector. Built to be as flexible as possible to ensure you require to get the data you are looking for. It is configured using a JSON file which consists of these sections:

“PreRequests” & “Request”: These processes all require basic configurations – “Url” endpoints, “Method” (Post, Get, Put, Patch, Delete etc), “Headers” section – this would include and header information “Content-Type”, “Authorization” maybe, Accept or any other custom field you require. Parameters are added into the end point and if you need variable parameters, you can tell Utopia which fields to save in the Parameter section of the configuration. You can then add a “Body” if required. This will always be in a string format, so if for example you need to post a JSON file you will need to convert that to a string format, for example where a “ gets noted as \”. A “URLEncoding” is also by default set to true, however you can override this in the request setting it to false. You can link up as many Pre-requests and Requests that you need to in order to get the data you require. Example you can use a pre-request to get an access Token, then use a second pre-request to fetch a userid off a second endpoint and lastly run the request with all the data gathered. You can daisy-chain as many as you need together.

Sometimes you may need to apply a calculation or two on some parameters or even a pre-request in order to get the data you are looking for. In this case you can use our "PreRequestScript" node. This can be used in both PreRequests and Request function and will run prior to the actual request. Currently we only allow javascript notation in the Code node. Note: if you want to save a parameter for future calls, you need to add the node in the parameter section (Path is not required if it is not being referenced out the document). To reference it in you JS script simply use this.yourParameterName and to save it just return it at the end of the call with the correct yourParameterName.

“Paths”: this section explains to utopia how to deal with the data coming in, here you can set: “Root” – specify the root field of the document, this is specifically handy if you are receiving a batch document and you would like to split the document at the root field or have other data you would like to ignore when pulling the document into Utopia. “Id” - specify a field which you would like to save in the document field to identify this document (will be saved as _header.Id) “CreatedDateTime” – identify a date field on the incoming data set which you would like to save as the created date/time.

“Authorization”: We currently have 2 types loaded for use here

"Authorization": {
   "Type": "oauth2",
   "GrantType" : "password",
   "TokenUrl": "https://tokenendpoint.com/services/oauth2/token",   
   "ClientId": "1234clientid",
   "ClientSecret": "1234clientsecret",
   "Username": "[email protected]",
   "Password": "passwordforuser"
}

"Authorization": {
    "Type": "NTLM",
    "Username": "username",
    "Password": "userpassword",
    "Domain": "DomainIfApplicable"
}    

For Basic Authentication please use do not use this Authorization section and rather all “Authorization” to the “Headers” section, then for the field value you take the username and password and concatenate them with a colon in the middle and run it through a Base64 encoder. Then add “Basic “ as a suffix, e.g.: user:pass -> Basic dXNlcjpwYXNz Alternatively you can add the details into a API testing program such as Postman and copy the Header details they convert for the specific Auth you require. We are continually looking for test cases for this, if you have an authentication type that you would like to add please get in contact with us and we can look at adding it for you.

Call Back URL’s -> Utopia currently doesn’t have functionality to implement authorization processes which require a call back URL. While we understand that this is becoming a requirement on a regular basis, we are still developing this functionality. In the meantime, as a work around we would suggest creating an initial connection outside of Utopia and loaded Utopia with a request using the refresh tokens.

“Parameters”: In order to use variable parameters, you will need to tell the connector which fields it needs to save in the request process. Each parameter will need to get assigned a name (try use a unique field name here), it will then get assigned a “Path” to the field on the incoming API response and an “InitialValue” to indicate what the initial call needs to use. EG:

"Parameters": {
 "LastDate": {
      "Path": "Order.Posting_Date",
      "InitialValue": "2023-02-14"
   }
}

SOAP - You can use this connector as a generic SOAP connector by including the SOAP header information “soapAction”

Here is an example of the Basic JSON file of a request with basic REST api parameters

{
  "Request": {
    "Url": "https://api.yourdomain.com/listener/934801c8-1a2b-4796-35g5-88d61c3d66cd",
    "Method": "GET",
    "Headers": {
      "Authorization": [
        "Basic 123456base64encoded"
      ],
      "Accept": [
        "application/json"
      ]
    }
  }
}

Here is an example of the full JSON file of a request with all parameters, you can pick and choose which you want.

{
  "PreRequests": [
    {
      "Url": "https://gettoken.com/tokens/OAuth/2",
      "Method": "POST",
      "Headers": {
        "Content-Type": [
          "application/x-www-form-urlencoded"
        ]
      },
      "Body": "grant_type=client_credentials&client_id=1234ClientID&client_secret=1234ClientSecret&resource=000012344"
    }
  ],
  "Request": {
    "Url": "https://endpointexample.com/orders/details?$filter=Posting_Date > ${LastDate}&$orderby=Posting_Date asc",
    "Method": "POST",
    "Headers": {
      "Accept": [
        "application/json"
      ],
      "Content-Type": [
        "application/json"
      ],
      "Authorization": [
        "Bearer ${token}"
      ],
      "customerheader1": [
        "HappyDays"
      ]
    },
    "PreRequestScript": {
      "Code": "let LastDateCalc = this.LastDateCalc; if (orders.length === 0) { PageNo = 0; LastDate = this.LastDateCalc;} else { PageNo = Number(this.PageNo) + 1; let LastUpdate = new Date(orders[0].orderSubmittedDate);  orders.forEach(order => { let orderDate = new Date(order.orderSubmittedDate); if (orderDate > LastUpdate) { LastUpdate = orderDate;}}); if (LastUpdate > new Date(this.LastDateCalc)) { LastDateCalc = LastUpdate.toISOString(); } } return { PageNo, LastDateCalc, LastDate: this.LastDate };",
      "Language": "javascript"
    },
    "UrlEncode": false,
    "Body": "{ \"IncludeHeader\": true, \"FilterDefinition\":  \"{_id: { $gt: ObjectId('${LastChange}')}}\"}"
  },
  "Paths": {
    "Root": "value"
  },
  "Authorization": {
    "Type": "NTLM",
    "Username": "user",
    "Password": "P@ssword",
    "Domain": "groupdomain"
  },
  "Parameters": {
    "LastDate": {
      "Path": "Order.Posting_Date",
      "InitialValue": "2023-02-14"
    },
    "LastChange": {
      "Path": "_Id",
      "InitialValue": "abcd12344id"
    },
    "LastDateCalc": {
      "InitialValue": "2024-04-02T11:05:28.187+00:00"
    },
    "PageNo": {
      "InitialValue": "0"
    }
  }
}