How Can I Use the hasMore Object?
Use Case
You have read the documentation on our cloud function connectors but don’t understand how best to utilize the hasMore object. You are looking for an example of how to use this object in your function's response.
Recommendations
Fivetran recommends the following:
Ensure your function is maintaining an internal variable that determines whether or not the data returned from your source API is paginated.
Ensure your function utilizes an
if/elsestatement to determine how to return thehasMoreobject:trueorfalse.
For more information, see the following example request and response format:
Request
var paginationCounter = 0
if (req.body.hasMore = false) {
hostname = endpoint.com
path = /giveMeEverything
}
else {
hostname = endpoint.com
path = /paginationCounter
}
if (res.moreData = true) {
function add() {
paginationCounter += 1;
}
else {
paginationCounter = 0
}
Response
if (paginationCounter > 0) {
hasMore:true
}
else {
hasMore:false
}
Context
The hasMore object is the mechanism that Fivetran provides to allow a function to paginate through data that is returned in a paginated format.
For example, if the source API provides paginated data:
On the initial run of a function, the
hasMoreobject is set asfalse:{“hasMore”:false}. Fivetran expects the response (from the source API) to be returned with an indicator. For example,{“nextCursor”:”yes”},{“page”:”1/10”}, or{“moreData”:true}.During the initial run of the function, if a response from the source API indicates there is more data to be retrieved, then you must pass
{“hasMore”:true}to Fivetran. Fivetran doesn't proceed to the next sync. Fivetran writes the returned data and asynchronously within the context of the original sync passes the same value ({“hasMore”:true}) back to the function. This indicates to the function to paginate the request and continuously work through the paginated data provided in the response.When there is no more data in the paginated response, the function passes
{“hasMore”:false}to Fivetran. When thehasMoreobject isfalse, the function moves to the next sync and a new invocation occurs.
Considerations
If the source API that you’re interacting with doesn't provide paginated data, then you can simply pass this back in the same manner to Fivetran: {“hasMore”:false}.