We do the deduplication automatically for you, but you must be mindful of how it works. The important bits are outlined below!
- Provide a unique
- Sort reverse-chronologically by time created.
An unfortunate artifact of polling for new data is that we must deduplicate the results we get back from your API. After all, we don’t want to trigger an action multiple times when a newly created item in your API exists in multiple distinct polls.
For example, say your endpoint returns a list of to-dos:
Every time a new Zap is created or turned on, we make a call to your API to populate our deduplication mechanism. We will cache and store each
id field in our database.
After this, we will poll at an interval (based on customer’s plan) looking for changes against this cached list of
Now let’s say we created a new to-do:
Only the first to-do with
id equal to
8 will be seen as a new item. That particular JSON object will then be passed through to the action in the user’s Zap. The others will be ignored since we already have seen their ids.
Your API must be able to return results in reverse-chronological order to make sure new/updated items can be found on the first page of results, as we don’t fetch additional pages. Wufoo is a great example of an API that supports sorting on any field and direction.
One other requirement is the
id field should always be supplied and unique among all items in the result.
Custom or multiple ID fields
What if the items your API returns do not have an
id field? Or how would you go about adding a Updated to-do trigger to our to-do app? In both cases you’ll use scripting to modify the API response.
Let’s assume your to-do API has an endpoint to return to-dos sorted by
updatedAt in descending direction. This is how you would use the scripting post poll trigger method:
Notice how we preserve the original value before setting
id to a new combined value that is unique for every update of a to-do.
What if your API cannot order its results in reverse-chronological order? How would you fetch the last page if you don’t know how many pages there might be? Again scripting can help you deal with this.
Depending on your API it is most likely you will need to use the scripting poll trigger method. This will let you handle the request(s) to your API.
One possible scenario could be:
- Fetch the first page, containing the oldest items, but also the total number of pages.
- Fetch the last page and reverse the order of the items before returning it.