Very large update requests trigger failure to sync due to revsDiff failure

[stevecassidy]

I've just diagnosed a problem we were having where sometimes our local pouchDB database did not sync to the remote CouchDB. The source of the problem turned out to be due to our use of a very large batch size (1000) combined with a number of large attachments on documents.

The problem occurs in the implementation of revsDiff:

 api.revsDiff = adapterFun('revsDiff', async function (req, opts, callback) {
    // If no options were given, set the callback to be the second parameter
    if (typeof opts === 'function') {
      callback = opts;
      opts = {};
    }
 
    try {
      // Get the missing document/revision IDs
      const result = await fetchJSON(genDBUrl(host, '_revs_diff'), {
        method: 'POST',
        body: JSON.stringify(req)
      });
      callback(null, result.data);
    } catch (error) {
      callback(error);
    }
  });

We were seeing the failure of JSON.stringify with our large payload (RangeError: Invalid string length). This meant that the POST request was not sent. PouchDB then went on to try again with the same payload and failed again.

There is most likely an error reported via the appropriate handler but we missed it.

The solution was to reduce the batch size so that the payload of one request was much smaller.

However, I feel that this is something that could be worked around inside PouchDB - perhaps backing off to smaller batches in the case of a RangeError in this function.

Posting here for reference in case others see this issue and in case this can be prioritised for a fix.

[alxndrsn]

Do you have a minimal reproduction you can share?

[stevecassidy]

No, I can try to put something together, this was hard to reproduce because it relied on having a large amount of data backed up for sync. We worked it out from crash logs in the end rather than being able to reproduce it.

[alxndrsn]

I can create data which will throw RangeError: Invalid string length like so:

> a = []; for(let i=0; i<10000; ++i) { const v = []; a.push(v); for(j=0; j<10000; ++j) v.push(Math.random()); } JSON.stringify(a)
Uncaught RangeError: Invalid string length
    at JSON.stringify (<anonymous>)

Knowing exactly where/how to feed that into PouchDB would be helpful 🙂

The solution was to reduce the batch size so that the payload of one request was much smaller.

There will probably be an edge case where the batch size is 1, and a single document can trigger RangeError: Invalid string length.

[stevecassidy]

Interesting. I can create a string up to about 500M with that line in browser (first loop to 2600). The limit will depend on the browser and environment as it's a memory limit, but yes, a single 500M object would also trigger the same error.

To reproduce in pouch I would create documents in the local db and then trigger sync with a very large block size.

[alxndrsn]

To reproduce in pouch I would create documents in the local db and then trigger sync with a very large block size.

Can the issue be recreated syncing between 2 local pouch DBs?

[stevecassidy]

Maybe not because this code is in the HTTP adaptor as I recall.

[alxndrsn]

Maybe a repro in a test case would be most useful then.