I am trying to consume an API which returns structure below.

{
  data: [{...},{...},{...},{...},...],
  nextUrl: "url_goes_here";
}

The pages end when the nextUrl is null. I want to collect all the elements in data into one array while going through the paginated responses. I tried the following code segment.

const getUserData = async (url) => {
    const result = await fetch(url)
    .then(res => res.json())
    .then(res => {
        dataList= [...dataList, ...res.data];
        console.log(res.data)
        if (res.nextUrl !== null) {
          getUserData(res.nextUrl);
        } else {
          console.log("else ", dataList);
        }
    })
    .catch(err => {});
  return result;
}

I am trying to consume an API which returns structure below.

{
  data: [{...},{...},{...},{...},...],
  nextUrl: "url_goes_here";
}

The pages end when the nextUrl is null. I want to collect all the elements in data into one array while going through the paginated responses. I tried the following code segment.

const getUserData = async (url) => {
    const result = await fetch(url)
    .then(res => res.json())
    .then(res => {
        dataList= [...dataList, ...res.data];
        console.log(res.data)
        if (res.nextUrl !== null) {
          getUserData(res.nextUrl);
        } else {
          console.log("else ", dataList);
        }
    })
    .catch(err => {});
  return result;
}

The console.log can print the result. but I want to get all the data to get into a variable that can be used for further processing later.

• javascript
• asynchronous
• async-await

1 Answer 1

Your approach using recursion isn't bad at all, but you're not returning the chunk of data (there's no value returned by your second .then handler). (You're also falling into the fetch pitfall of not checking ok. It's a flaw IMHO in the fetch API design.)

There's no need for recursion though. I'd be tempted to just use a loop:

const getUserData = async (url) => {
    const result = [];
    while (url) {
        const response = await fetch(url);
        if (!response.ok) {
            throw new Error("HTTP error " + response.status);
        }
        const {data, nextUrl} = await response.json();
        result.push(...data);
        url = nextUrl;
    }
    return result;
};

But if you want to use recursion:

const getUserData = async (url) => {
    const response = await fetch(url);
    if (!response.ok) {
        throw new Error("HTTP error " + response.status);
    }
    const {data, nextUrl} = await response.json();
    if (nextUrl) {
        data.push(...await getUserData(nextUrl));
    }
    return data;
};

or

const getUserData = async (url, result = null) => {
    const response = await fetch(url);
    if (!response.ok) {
        throw new Error("HTTP error " + response.status);
    }
    const {data, nextUrl} = await response.json();
    if (!result) {
        result = data;
    } else {
        result.push(...data);
    }
    if (nextUrl) {
        result = await getUserData(nextUrl, result);
    }
    return result;
};