$conf, $runtime; function_exists('chdir') AND chdir(APP_PATH); $r = 'mysql' == $conf['cache']['type'] ? website_set('runtime', $runtime) : cache_set('runtime', $runtime); } function runtime_truncate() { global $conf; 'mysql' == $conf['cache']['type'] ? website_set('runtime', '') : cache_delete('runtime'); } register_shutdown_function('runtime_save'); ?>javascript - Getting error while installing dependency manually on render environment - Stack Overflow|Programmer puzzle solving

Programmer puzzle solving

    最新消息：Welcome to the puzzle paradise for programmers! Here, a well-designed puzzle awaits you. From code logic puzzles to algorithmic challenges, each level is closely centered on the programmer's expertise and skills. Whether you're a novice programmer or an experienced tech guru, you'll find your own challenges on this site. In the process of solving puzzles, you can not only exercise your thinking skills, but also deepen your understanding and application of programming knowledge. Come to start this puzzle journey full of wisdom and challenges, with many programmers to compete with each other and show your programming wisdom! Translated with DeepL.com (free version)
    你的位置： HOME>matter>javascript - Getting error while installing dependency manually on render environment - Stack Overflow

    javascript - Getting error while installing dependency manually on render environment - Stack Overflow

    matteradmin14PV0评论

    I am unable to deploy my dockerized application on render I am using puppeteer-real-browser library which requires xvfb to be installed if ran on linux OS.

    I am using Chromium Version 130.0.6723.58 on my Linux (ubuntu)

    It works fine locally as i installed xvfb but during deployment on render its throwing error.

    Here's the Dockerfile:

    FROM ghcr.io/puppeteer/puppeteer:19.7.2
ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true \
    PUPPETEER_EXECUTABLE_PATH=/usr/bin/google-chrome-stable
RUN apt-get update && apt-get install -y xvfb
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm ci
COPY . .
CMD ["xvfb-run", "--server-args=-screen 0 1024x768x24", "node", "index.js"]

    Here's the error i see on render's logs:

    #6 [2/6] RUN apt-get update && apt-get install -y xvfb
#6 0.050 Reading package lists...
#6 0.061 E: List directory /var/lib/apt/lists/partial is missing. - Acquire (13: Permission denied)
#6 0.061 W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/google-chrome.list:3 and /etc/apt/sources.list.d/google.list:1
#6 0.061 W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/google-chrome.list:3 and /etc/apt/sources.list.d/google.list:1
#6 ERROR: process "/bin/sh -c apt-get update && apt-get install -y xvfb" did not complete successfully: exit code: 100

 > [2/6] RUN apt-get update && apt-get install -y xvfb:
0.061 E: List directory /var/lib/apt/lists/partial is missing. - Acquire (13: Permission denied)
0.061 W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/google-chrome.list:3 and /etc/apt/sources.list.d/google.list:1
0.061 W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/google-chrome.list:3 and /etc/apt/sources.list.d/google.list:1

Dockerfile:9

   7 |     
   8 |     # Install xvfb
   9 | >>> RUN apt-get update && apt-get install -y xvfb
  10 |     
  11 |     # Set working directory
error: failed to solve: process "/bin/sh -c apt-get update && apt-get install -y xvfb" did not complete successfully: exit code: 100
error: exit status 1

    Here's my index.js file

    const express = require("express");
const puppeteer = require("puppeteer");
const { connect } = require("puppeteer-real-browser");
const cors = require("cors");
require("dotenv").config();
const app = express();
app.use(cors());

const base = process.env.BASE_URL;
const PORT = process.env.PORT;

app.get("/wakeup", (req, res) => res.send("Server is awake!"));

app.get("/search", async (req, res) => {
  const query = req.query.q;
  if (!query) return res.status(400).json({ Error: "Missing query param 'q'" });

  let books = null;
  query.length >= 2 ? (books = await searchBooks(query)) : null;
  if (!books || books.length === 0)
    return res
      .status(404)
      .json({ Error: "Book not found, try different one!" });

  res.json(books);
});

app.get("/ebook", async (req, res) => {
  const { url } = req.query;
  if (!url) return res.status(400).json({ error: "Missing query 'url'" });

  const downloadUrl = await getDownloadLink(url);
  if (!downloadUrl.ebook)
    return res
      .status(404)
      .json({ Error: "Ebook not found, try different one!" });
  res.json(downloadUrl);
});

const searchBooks = async (title) => {
  const browser = await puppeteer.launch({
    headless: true,
    args: ["--no-sandbox", "--disable-setuid-sandbox"],
  });
  const page = await browser.newPage();
  await page.setUserAgent(
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
  );

  try {
    await page.goto(`${base}/?s=${title}&page=1`, {
      waitUntil: "networkidle2",
    });
    // await page.type('input[name="q"]', title);
    // await page.keyboard.press('Enter');
    await page.waitForSelector("#genesis-content", { timeout: 20000 });

    const books = await page.evaluate((baseUrl) => {
      return Array.from(
        document.querySelectorAll("#genesis-content .post")
      ).map((book) => ({
        title: book.querySelector(".entry-title a")?.textContent.trim(),
        year: book.querySelector(".entry-meta .entry-time")?.textContent.trim(),
        thumbnail: book
          .querySelector(".entry-image-link img")
          ?.getAttribute("src"),
        url: `${book.querySelector(".entry-image-link")?.getAttribute("href")}`,
      }));
    }, base);

    return books;
  } catch (error) {
    handleErrors(error, "searchBooks");
  } finally {
    await browser.close();
  }
};

const getDownloadLink = async (bookUrl) => {
  const { page, browser } = await connect({
    headless: false,
    args: ["--no-sandbox", "--disable-setuid-sandbox"],
  });

  // const page = await browser.newPage();
  await page.setUserAgent(
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
  );

  try {
    await page.goto(bookUrl, { waitUntil: "networkidle2" });
    await new Promise((resolve) => setTimeout(resolve, 5000));

    await page.mouse.move(100, 100);
    await new Promise((resolve) => setTimeout(resolve, 1000)); // Short delay

    const buttonSelector = `form[action="${base}/Fetching_Resource.php"]`;
    await page.waitForSelector(buttonSelector);
    await page.mouse.move(120, 120);
    await page.click(buttonSelector);

    const newTarget = await browser.waitForTarget(
      (target) => target.opener() === page.target(),
      { timeout: 10000 }
    );
    const newPage = await newTarget.page();

    let lastValidLink = null;

    newPage.on("response", async (response) => {
      const url = response.url();
      //   console.log("Response URL:", url);

      if (url.includes("fs4.oceanofpdf") && url.includes(".pdf")) {
        lastValidLink = url;
      }
    });

    const maxWaitTime = 30000;
    const interval = 1000;
    let elapsedTime = 0;

    while (!lastValidLink && elapsedTime < maxWaitTime) {
      await new Promise((resolve) => setTimeout(resolve, interval));
      elapsedTime += interval;
    }

    if (lastValidLink) return { ebook: lastValidLink };
    if (!lastValidLink) return { Error: "Sorry ebook not found!" };
  } catch (error) {
    // console.error("Error in downloadBookOne function:", error);
    return { Error: "Sorry ebook not found!" };
  } finally {
    await browser.close();
  }
};

const handleErrors = (error, functionName) => {
  console.error(`Error in ${functionName}:`, error.message);
};

app.listen(PORT, () => {
  console.log(`Server running on port ${PORT}`);
});

    here's package.json

    {
  "name": "backend",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "type": "commonjs",
  "scripts": {
    "start": "node index.js"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "cors": "^2.8.5",
    "dotenv": "^16.4.5",
    "express": "^4.21.1",
    "puppeteer": "^19.7.2",
    "puppeteer-real-browser": "^1.3.17"
  }
}

    The Base_URL will be

    I am unable to deploy my dockerized application on render I am using puppeteer-real-browser library which requires xvfb to be installed if ran on linux OS.

    I am using Chromium Version 130.0.6723.58 on my Linux (ubuntu)

    It works fine locally as i installed xvfb but during deployment on render its throwing error.

    Here's the Dockerfile:

    FROM ghcr.io/puppeteer/puppeteer:19.7.2
ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true \
    PUPPETEER_EXECUTABLE_PATH=/usr/bin/google-chrome-stable
RUN apt-get update && apt-get install -y xvfb
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm ci
COPY . .
CMD ["xvfb-run", "--server-args=-screen 0 1024x768x24", "node", "index.js"]

    Here's the error i see on render's logs:

    #6 [2/6] RUN apt-get update && apt-get install -y xvfb
#6 0.050 Reading package lists...
#6 0.061 E: List directory /var/lib/apt/lists/partial is missing. - Acquire (13: Permission denied)
#6 0.061 W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/google-chrome.list:3 and /etc/apt/sources.list.d/google.list:1
#6 0.061 W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/google-chrome.list:3 and /etc/apt/sources.list.d/google.list:1
#6 ERROR: process "/bin/sh -c apt-get update && apt-get install -y xvfb" did not complete successfully: exit code: 100

 > [2/6] RUN apt-get update && apt-get install -y xvfb:
0.061 E: List directory /var/lib/apt/lists/partial is missing. - Acquire (13: Permission denied)
0.061 W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/google-chrome.list:3 and /etc/apt/sources.list.d/google.list:1
0.061 W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/google-chrome.list:3 and /etc/apt/sources.list.d/google.list:1

Dockerfile:9

   7 |     
   8 |     # Install xvfb
   9 | >>> RUN apt-get update && apt-get install -y xvfb
  10 |     
  11 |     # Set working directory
error: failed to solve: process "/bin/sh -c apt-get update && apt-get install -y xvfb" did not complete successfully: exit code: 100
error: exit status 1

    Here's my index.js file

    const express = require("express");
const puppeteer = require("puppeteer");
const { connect } = require("puppeteer-real-browser");
const cors = require("cors");
require("dotenv").config();
const app = express();
app.use(cors());

const base = process.env.BASE_URL;
const PORT = process.env.PORT;

app.get("/wakeup", (req, res) => res.send("Server is awake!"));

app.get("/search", async (req, res) => {
  const query = req.query.q;
  if (!query) return res.status(400).json({ Error: "Missing query param 'q'" });

  let books = null;
  query.length >= 2 ? (books = await searchBooks(query)) : null;
  if (!books || books.length === 0)
    return res
      .status(404)
      .json({ Error: "Book not found, try different one!" });

  res.json(books);
});

app.get("/ebook", async (req, res) => {
  const { url } = req.query;
  if (!url) return res.status(400).json({ error: "Missing query 'url'" });

  const downloadUrl = await getDownloadLink(url);
  if (!downloadUrl.ebook)
    return res
      .status(404)
      .json({ Error: "Ebook not found, try different one!" });
  res.json(downloadUrl);
});

const searchBooks = async (title) => {
  const browser = await puppeteer.launch({
    headless: true,
    args: ["--no-sandbox", "--disable-setuid-sandbox"],
  });
  const page = await browser.newPage();
  await page.setUserAgent(
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
  );

  try {
    await page.goto(`${base}/?s=${title}&page=1`, {
      waitUntil: "networkidle2",
    });
    // await page.type('input[name="q"]', title);
    // await page.keyboard.press('Enter');
    await page.waitForSelector("#genesis-content", { timeout: 20000 });

    const books = await page.evaluate((baseUrl) => {
      return Array.from(
        document.querySelectorAll("#genesis-content .post")
      ).map((book) => ({
        title: book.querySelector(".entry-title a")?.textContent.trim(),
        year: book.querySelector(".entry-meta .entry-time")?.textContent.trim(),
        thumbnail: book
          .querySelector(".entry-image-link img")
          ?.getAttribute("src"),
        url: `${book.querySelector(".entry-image-link")?.getAttribute("href")}`,
      }));
    }, base);

    return books;
  } catch (error) {
    handleErrors(error, "searchBooks");
  } finally {
    await browser.close();
  }
};

const getDownloadLink = async (bookUrl) => {
  const { page, browser } = await connect({
    headless: false,
    args: ["--no-sandbox", "--disable-setuid-sandbox"],
  });

  // const page = await browser.newPage();
  await page.setUserAgent(
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
  );

  try {
    await page.goto(bookUrl, { waitUntil: "networkidle2" });
    await new Promise((resolve) => setTimeout(resolve, 5000));

    await page.mouse.move(100, 100);
    await new Promise((resolve) => setTimeout(resolve, 1000)); // Short delay

    const buttonSelector = `form[action="${base}/Fetching_Resource.php"]`;
    await page.waitForSelector(buttonSelector);
    await page.mouse.move(120, 120);
    await page.click(buttonSelector);

    const newTarget = await browser.waitForTarget(
      (target) => target.opener() === page.target(),
      { timeout: 10000 }
    );
    const newPage = await newTarget.page();

    let lastValidLink = null;

    newPage.on("response", async (response) => {
      const url = response.url();
      //   console.log("Response URL:", url);

      if (url.includes("fs4.oceanofpdf") && url.includes(".pdf")) {
        lastValidLink = url;
      }
    });

    const maxWaitTime = 30000;
    const interval = 1000;
    let elapsedTime = 0;

    while (!lastValidLink && elapsedTime < maxWaitTime) {
      await new Promise((resolve) => setTimeout(resolve, interval));
      elapsedTime += interval;
    }

    if (lastValidLink) return { ebook: lastValidLink };
    if (!lastValidLink) return { Error: "Sorry ebook not found!" };
  } catch (error) {
    // console.error("Error in downloadBookOne function:", error);
    return { Error: "Sorry ebook not found!" };
  } finally {
    await browser.close();
  }
};

const handleErrors = (error, functionName) => {
  console.error(`Error in ${functionName}:`, error.message);
};

app.listen(PORT, () => {
  console.log(`Server running on port ${PORT}`);
});

    here's package.json

    {
  "name": "backend",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "type": "commonjs",
  "scripts": {
    "start": "node index.js"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "cors": "^2.8.5",
    "dotenv": "^16.4.5",
    "express": "^4.21.1",
    "puppeteer": "^19.7.2",
    "puppeteer-real-browser": "^1.3.17"
  }
}

    The Base_URL will be https://oceanofpdf

    • javascript
    • linux
    • docker
    • puppeteer
    • render
    Share Improve this question edited Nov 16, 2024 at 17:25 ggorlen 58.1k8 gold badges115 silver badges157 bronze badges asked Nov 16, 2024 at 8:30 KuldeepKuldeep 12 bronze badges
    Add a comment  | 

    1 Answer 1

    Reset to default 0 
    #6 [2/6] RUN apt-get update && apt-get install -y xvfb
#6 0.050 Reading package lists...
#6 0.061 E: List directory /var/lib/apt/lists/partial is missing. - Acquire (13: Permission denied)
#6 0.061 W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/google-chrome.list:3 and /etc/apt/sources.list.d/google.list:1
#6 0.061 W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/google-chrome.list:3 and /etc/apt/sources.list.d/google.list:1
#6 ERROR: process "/bin/sh -c apt-get update && apt-get install -y xvfb" did not complete successfully: exit code: 100

    apt commands must be run as root, and your base image changes the user. You can change the user in the Dockerfile with:

    USER root

    Articles related to this article

    Post a comment

    comment list (0)

    1. No comments so far