最新消息:Welcome to the puzzle paradise for programmers! Here, a well-designed puzzle awaits you. From code logic puzzles to algorithmic challenges, each level is closely centered on the programmer's expertise and skills. Whether you're a novice programmer or an experienced tech guru, you'll find your own challenges on this site. In the process of solving puzzles, you can not only exercise your thinking skills, but also deepen your understanding and application of programming knowledge. Come to start this puzzle journey full of wisdom and challenges, with many programmers to compete with each other and show your programming wisdom! Translated with DeepL.com (free version)

javascript - Insane amount of RAM usage with BullMQ + fetch - Stack Overflow

matteradmin7PV0评论

I have a small app that runs several requests(node-fetch) with in a queue with bullMQ.

And i have 3 worker threads (if i dont use this main event loop gets blocked and my whole app freezes.) (This is an another issue why main loop gets blocked with a few fetch requests ?)

queue A , queue B and queue C

all queues have same configuration

new Queue('a-queue', {
connection: {
    url: process.env.REDIS_URL,
    enableReadyCheck: true,
},
defaultJobOptions: {
    attempts: 5,
    removeOnComplete: true,
    removeOnFail: false,
},

worker for the queue (a,b,c all same):

const processor = pathToFileURL(__dirname + '/a_processor.js')
new Worker('a-queue', processor, {
    connection: {
        url: process.env.REDIS_URL,
        enableReadyCheck: true,
    },
    concurrency: 1,
    useWorkerThreads: true,
})

And a_processor.js is sending some requests with fetch and returns the values.

Also i have a setting for the global concurrency limit in index.js (start of the express app):

await queue.setGlobalConcurrency(5)

Upto this points it seems like it works as i expected (bullmq dashboard blue ones are active yellow ones are waiting):

But after like a minute redis instance is crashing due high memory consumption:

Memory usage is like skyrocket when i run this app.And what is doing is just sending fetch requests (10-15 in paralel) and these requests are not like responsing mbs of data its only few kb of json each time. There is nothing much but this consumes so much memory idk how.Or someting else causing that idk.

Here's the request method:

const headerGenerator = new HeaderGenerator()
const headers = headerGenerator.getHeaders()

const response = await fetch(`${domain}`, {
    headers: headers,
    agent: agentDcIp,
    signal: AbortSignal.timeout(15000) as any,
})

const body = await response.text()

return {
    body: body,
    ok: response.ok,
    status: response.status,
}

Any idea why this is happening and how to prevent it.

Post a comment

comment list (0)

  1. No comments so far