Pagination

Many times, it is required to paginate the results that the server returns in order to retrieve extra older/newer items because the server mostly does not return how many items you want.

Pagination specification

The pagination collection is placed in the root level of Request Specification, along with url, method , body and response.

Below is the list of all pagination directives. They should be placed inside the pagination collection.

Key	Type	Description
mergeWithParent	Boolean	Specifies the condition if to execute pagination or not
url	IML String	Specifies the URL that should be called.
method	IML String	Specifies the HTTP method, that should be used when issuing a request.
headers	IML Flat Object	A single level (flat) collection, that specifies request headers.
qs	IML Flat Object	A single level (flat) collection that specifies request query string parameters.
body	Any IML Type	Specifies a request body.
condition	IML String or Boolean	Determines if to execute current request or never.

mergeWithParent

This directive specifies if to merge pagination parameters with the original request parameters, or not. Default value is true, which means that all of your pagination request parameters will be merged with the original request parameters.

If the value is set to false, then no parameters will be merged, but directly used to make a pagination request.

url

This directive specifies the URL that will be called when the pagination request is executed. It will override the original request URL no matter the value of mergeWithParent.

method

This directive specifies the HTTP method to be used when executing the pagination request. It will override the original request method no matter the value of mergeWithParent.

headers

This directive specifies the request headers to be used when executing the pagination request. It will merge with headers of the original request, overriding headers with the same name when mergeWithParent is set to true(default).

qs

This directive specifies the request query string parameters to be used when executing the pagination request. It will merge with query string parameters of the original request, overriding ones with the same name when mergeWithParent is set to true (default).

body

This directive specifies the request body when the request method is anything but GET to be used when executing the pagination request. It will override the original request body no matter the value ofmergeWithParent.

condition

This directive specifies whether to execute the pagination request or not. It has all context variables available to it as response collection, such as body, headers, temp etc. Use this to stop paginating if the server is sending you info about the number of pages or items, or if the next page is available or not.

Limits

As we cannot paginate to infinity and beyond, there are four limits on the paginated requests.

Name	Total limit of ...	Value
Max Request Count	... all requests performed by a single module	100
Max Pagination Request Count	... pagination requests performed by a single module	50
Max Past Record	... paginated records	3200
Max Call Timeout	... seconds	40

Examples

Total pages

Common way of pagination is using total pages or pages count . Basically, the service gives you information about how many pages of content (the total count) are available and you have to iterate over those pages.

Response from a service

{
  "Data": [
    {...},
    {...},
    {...},
    {...},
    {...}
  ]
  "TotalItems": 42,
  "TotalPages": 10
}

This is a sample response from the service. As you can see, the service returns some items in the Data array, TotalItems, and TotalPages metadata. The TotalPages is the one we will use for setting up pagination in Make.

Pagination code

"pagination": {
	"qs": {
		"Page": "{{pagination.page}}"
	},
	"condition": "{{body.TotalPages >= pagination.page}}"
}

The service accepts Page in query string as a parameter which tells the service which page you want to retrieve. There's no need for setting some temporary variables. In this case, just send {{pagination.page}} in correct query parameter and set the condition so the pagination directive knows when to stop requesting more pages.

Some services index pages from 0, and some index from 1. The pagination.page directive indexes pages from 1. You'll need to check that to set up the pagination correctly.

Limit - offset

This is another common way how pagination is being implemented. You can set the paging cursor using two parameters:

• limit - sets the maximum amount of records being retrieved per page

• offset - sets the number of skipped records from the beginning of the data set

Using these two parameters, you're able to control the pagination.

Response from a service

[
    {...},
    {...},
    {...},
    {...},
    {...}
]

The service doesn't even have to return you any pagination info. Sometimes the service returns something like PageSize , but that's not important for us now.

Pagination code

"qs": {
    "limit": 100
},
"response": {
    "limit": "{{parameters.limit}}",
    "output": "{{item}}",
    "iterate": "{{body.users}}"
},
"pagination": {
    "qs": {
        "offset": "{{(pagination.page - 1) * 100}}"
    }
}

Notice that the limit parameter is set statically in the query string for a request. It represents the size of a page.

Inside the pagination directive, we'll set the offset parameter.

After the first request, the value of pagination.page will be 2, as it's prepared to retrieve a second page. We'll transform it to the offset we want. pagination.page - 1 = 1 and 1 * 100 is 100. Notice that the value for computing the offset is the same as in the limit parameter.

So the second request will have limit: 100 and offset: 100 so it will skip first 100 records and retrieve records 101-200. The third request will skip first 200 (as (3-1)*100 = 200) and retrieve records 201-300. And so on.

The pagination will stop automatically after the response with 0 records or once the condition response.limit is met. If that behavior does not fit your needs, you can specify the additional condition.

Next Page Token

Next page token is basically a unique identifier of a next page, which is provided in the response. You can get the next page without counting some pages or offsets using this token only. Sometimes the "Next page URL" is provided instead of the token only, but the flow is quite similar.

Response from a service

{
    "data": [
        {...},
        {...},
        {...},
        {...},
        {...}
    ],
    "nextPageToken": "ABCDEFGHHGFEDCBA"
}

The service returns a nextPageToken for retrieving the next page.

Pagination code

"pagination": {
	"qs": {
		"nextPageToken": "{{body.nextPageToken}}"
	},
	"condition": "{{body.nextPageToken}}"
}

In every following request, we'll use the nextPageToken provided with the previous one. We'll stop the pagination when no other nextPageToken was received.

Next Page URL

The next page URL is a link to the next page which is provided in the response.

If the next page URL is provided, you can map it directly in the pagination directive.

Response from a service

{
    "items": [
        {...},        },
        {...}
    ],
    "_links": {
        "next": {
            "href": "https://api.abc.com/locations?page=ZmFsc2V8aHR...."
        },
        "previous": null
    }
}

Pagination code

"pagination": {
        "condition": "{{body._links.next.href}}",
        "url": "{{body._links.next.href}}"
    }

Has More Items

When using this method, you will get information on whether the data set contains more records or not.

Response from a service

{
    "data": [
        {...},
        {...},
        {...},
        {...},
        {...}
    ],
    "has_more_items": true
}

The service returns a has_more_items boolean which tells you if you can get more pages or not.

Pagination code

"pagination": {
	"qs": {
		"page": "{{pagination.page}}"
	},
	"condition": "{{body.has_more_items}}"
}

We'll simply keep asking for the next page until the has_more_items becomes false. When that happens, we can be sure that no records can be retrieved.

Page Only

This type of pagination isn't the best, but many services use this. Basically, you keep asking for the next page and when you get an empty response, you know you're done.

Response from a service

[
    {...},
    {...},
    {...},
    {...},
    {...}
]

No additional info about pagination is provided.

Pagination code

"pagination": {
	"qs": {
		"page": "{{pagination.page}}"
	}
}

We'll keep asking for the next page until the body (or some key in the body) is not empty.

The pagination should stop after an empty response or empty data in the body. But some services think that it's a good practice to keep returning the last page after the maximum amount of pages has been reached. Always check the behavior of the service when using this type of pagination.

Debugging pagination

Debugging of pagination in list/search modules