Option 1 for sure. Some guards got added at one point but I thought they'd all been removed by now.

Ok, I am working on this. It's not a simple as removing the setImmediates -- some methods relying on it to make things easier.

I use nexTick, then setImmediate later on with all my "next" callbacks when using async because I ran into issues a very long time ago. Does Async now have these built in? I'm assuming there is a performance hit if the library does it and I do it as well. If they are leaving, then I have to change nothing, but if they are staying (if they exist) then I should probably remove mine.

As for my suggestion it would making it a switch. I've always hated having to call my callbacks wrapped up, it makes for ugly code. If it were a switch to turn on hidden setImmediate calls that would be slick.

I've added an ensureAsync function wrapper to my acomb helper library that will use a setImmediate call with your callback if (and only if) it needs to:

https://github.com/aearly/acomb#ensureAsync

Something like this, http://dbaron.org/log/20100309-faster-timeouts, would be a good strategy for browsers.

I want to incorporate some postMessage tricks like that, but getting a cross browser/cross platform implementation that is fast in the most places is quite complicated: https://github.com/YuzuJS/setImmediate/blob/master/setImmediate.js

Also, even the fastest implementation (process.nextTick) is still slower than a simple function call, so it's not something we'd want to do by default everywhere.

Implementing a postMessage/nextTick function based on a variable MAX_STACK_SIZE option would be nice.

I'd suggest not guarding against stack overflows by default.

There could be an opt-in option that can either be turned on per method call or with an overall switch that wraps every async function in ensureAsync.

async.each(arr, iterator[, options][, callback])

if options.safe (or something else) is true, then wrap iterator in ensureAsync.

Agreed, it'd be nice to be able to detect the SO and provide a console warn to use a sync version (window.onerror&process.on('uncaughtException') hack :P)

With each, auto, and waterfall now without a deferral, I think this issue is safe to close. The only functions that have deferrals are ensureAsync (its core functionality), memoize, forever, and queue (which need it to work properly).

I ran into this issue as well recently. I am trying to parse a certain file format using node.js streams. As a consequence, I'm unable to parse the entire file at once, because a node.js stream will only give me buffers of which I don't know the size beforehand. As a result, I also don't know if sufficient bytes are present to do the parsing of a specific entry in the file. This is why I used the async module which uses callbacks that get called once sufficient bytes become available. If sufficient bytes are already available in the current buffer, then the callback is fired synchronously.

This works fine and provides a neat way of abstracting away whether I have to wait for another buffer chunk to be received or not. However, I if receive very large buffer chunks at once from the stream; it is possible that certain loops that I created using the async module are completely synchronous. Consider a trivial example where I have the entire file to be parsed as 1 buffer chunk, in that case all my async-stuff will be synchronous under the hood because all bytes are available - which is what I want because I would like to use my parser for synchronously parsing a file as well. As a result, I sometimes receive stack overflow errors.

I managed however to solve the issue by implementing my own "whilst" method. The philosophy is that I detect whether a callback is fired synchronously or asynchronously. If the callback is fired synchronously, I use a while loop which doesn't increase the call stack. If the callback is fired asynchronously, I use a recursive function call as normal. Stack overflow errors aren't to be expected here because the stack is cleared between two process ticks. This looks as follows:

const whilst = function(test, it, cb) {
  
  let ctx = this, main;
  let strategy = main = function() {
    return test.call(ctx);
  };
  let quit = function() {return false;};
  let results = [];
  const go = function() {

    // Loop synchronously as much as we can in a while loop instead of 
    // going deeper down the call stack.
    while (strategy()) {
      let sync = true;
      let pre = false;
      it.call(ctx, function(err, ...rest) {

        // This is where the magic happens: in case we're still 
        // synchronous, the "sync" variable will _still_ be set to 
        // true. Don't do anything now, the loop will continue out of 
        // itself by calling the test again. However, if the callback 
        // was fired asynchronously, the "sync" variable will already 
        // be false. In that case we need to re-enter the
        // while(strategy()) loop with the test function being called.
        results = rest;
        if (sync) {
          pre = true;
        }
        else {
          strategy = main;
          go();
        }

      });

      // In case we're async, "pre" will stil be false. This means that 
      // the loop needs to be quit, which can be done by changing 
      // strategies to quit.
      if (!pre) {
        strategy = quit;
      }
      sync = false;

    }

    // If we've reached this point, but the strategy was not set to 
    // "quit", this means that the synchronous test simply returned false. 
    // Call the final callback to exit the loop.
    if (strategy === main) {
      cb(null, ...results);
    }

  };
  go();

};

I can imagine that this concept would be possible to apply to other methods vulnerable to stack overflows as well. As such I think that in theory it is possible to adapt the entire async library to be able to handle synchronous callbacks without stack overflows. Whether this is desirable in a library whose purpose is for working with "true" asynchronous functions is an open question.

What you described is very similar to ensureAsync -- a wrapper for functions that always ensures the callback is called on the next tick.

@aearly, perhaps my explanation was not entirely clear, but what I tried to do was the exact opposite of ensureAsync: if all callbacks are called synchronously, then the entire whilst loop executes synchronously. This means that

let i = 0;
let order = [];
whilst(function() {
  return i < 1e6;
}, function(cb) {
  i++;
  cb(null);
}, function(err) {
  order.push(1);
});
order.push(2);

// Right now, order === [1, 2];, because the callbacks are called synchronously.

This is already the default behavior for the whilst function in async, but my approach above avoids stackoverflow errors for synchronous while loops with more iterations than the JS call stack can handle.