form.parse's Promise never resolves in tests

[Mitame]

Support plan

• Which support plan is this issue covered by? (Community, Sponsor, Enterprise): Community
• Currently blocking your project/work? (yes/no): no
• Affecting a production system? (yes/no): no

Context

• Node.js version: 18.18.2
• Release Line of Formidable (Legacy, Current, Next): next
• Formidable exact version: v3.5.1
• Environment (node, browser, native, OS): Node 18.18.2, Linux (though likely all runtimes are affected)
• Used with (popular names of modules): node-mocks-https, form-data

Note this issue appears to only be present in tests as it is dependent on how promises get queued, but it could lead to a bug in live code.

What are you trying to achieve or the steps to reproduce?

A (somewhat) minimal reproducible test.

import FormData from "form-data";
import formidable from "formidable";
import { createMocks } from "node-mocks-http";

test("Can parse files from node-mocks-http", async () => {
    let formData = new FormData();
    formData.append("file", "File data", "data.txt");

    const getLengthAsync: () => Promise<number> = async () => {
        return new Promise((res, rej) => {
            formData.getLength((err, length) => {
                if (err) {
                    rej(err);
                } else {
                    res(length);
                }
            });
        });
    };

    const { req, res } = createMocks({
        headers: {
          'content-type': `multipart/form-data;boundary="${formData.getBoundary()}"`,
          'content-length': (await getLengthAsync()).toString(),
        },
    });

    let form = formidable({});
    let promise = form.parse(req); 
    req.send(formData.getBuffer());
    let [fields, files] = await promise;
    console.log({fields, files});
})

What was the result you got?

The test times out at 5000ms because the await form.parse(req) never resolves.

What result did you expect?

The test passes.

Cause

The issue is caused as the callback handlers are set after an await of a promise.

formidable/src/Formidable.js

Line 232 in 1699ec6

await this.writeHeaders(req.headers);

This means the function returns, then data is sent which formidable is not listening for, then the await within parse is resolved and the handlers are set, but the events have already been missed, so the handlers just do nothing until Jest stops the test.

Workaround

It's possible to call req.send in a promise with a slight delay, e.g.

let promiseParse = form.parse(req);
let promiseSend = new Promise((resolve, reject) => {
    setTimeout(() => {
        req.send(formData.getBuffer());
        resolve(undefined);
    }, 50)
}
Promise.all([promiseParse, promiseSend]);

Breaking change

This is also a breaking change as it seems to have been introduced since v3.2.4. While the Promise style of parse didn't exist in that version, this bug also affects the callback style in the same manor.

[allthesignals]

This is happening to me too, not even in test environment. 3.5.1. The promise just never resolves

[Angryman18]

If you have body-parser package just remove or comment it. Hope it will resolve

[GrosSacASac]

(reminder for me: try req.on('data') earlier)

are you testing formidable, or a project using formidable ?

[george-i]

If you have body-parser package just remove or comment it. Hope it will resolve

I tried commenting out busboy and busboy-body-parser and the parsing completed successfully.

[egargale]

If you have body-parser package just remove or comment it. Hope it will resolve

I had been banging my head against the wall for two days.... Thank you

[tunnckoCore]

@george-i @egargale yeah, that's a known thing that things don't work well if someone else is interacting with the body too.

Formidable can be simplified A LOT, but the problem to move directly the newest stuff is that people are relying on old versions, there's still people on v1.. I had the code ready for over a year, a dream of mine for v4-5, but yeah ;d

[Ray0907]

I encounter the same issue. Is there any solution available to resolve the problem?

[maxlath]

I migrated an Express app using body-parser and formidable v1.2.6 to formidable v3.5.1, and bumped into that issue. I was able to keep body-parser on, by calling await form.parse(req) from a middleware running before body-parser. See commit inventaire/inventaire@3b3c556. Hope that that can help others

[maxlath]

I bumped into this issue again, but found a more practical way to get around it:

// Middleware function
export async function prepareFormParse (req, res, next) {
  if (!req.headers['content-type']?.startsWith('multipart/form-data')) return next()

  // Pause req stream to call req.resume only oncee formidable req event listeners are initialized
  req.pause()
  next()
}

then later from an express controller

const form = formidable({})
// Leave form.parse the time to setup req event listeners, after `await this.writeHeaders(req.headers)`
setImmediate(() => req.resume())
const [ fields, files ] = await form.parse(req)