Parallel Build for Multiple Regions, on a single device (not GitHub Matrix)

[anaclumos]

Can I parallelly build for Multiple Regions to be employed on a single device (not GitHub Matrix)?

[slorber]

You can build multiple regions in parallel locally but each build uses the temporary .docusaurus folder.

If you build in parallel on a single machine I'd recommend isolating the builds by duplicating the root folder so that there's no weird conflict here.

Maybe it's possible to build in parallel using distinct folders, that you can customize with this env variable. But I never tried that so far.

/**
 * Can be absolute or relative to site directory. Code should generally use
 * `context.generatedFilesDir` instead (which is always absolute).
 */
export const GENERATED_FILES_DIR_NAME =
  process.env.DOCUSAURUS_GENERATED_FILES_DIR_NAME ?? '.docusaurus';

[johnenderson]

Is it possible to consolidate sites that were created separately at a later time? Currently, Docusaurus always routes navigation to the main site.

[slorber]

Is it possible to consolidate sites that were created separately at a later time?

Yes. You can merge 2 static build folders into one as a build step, or configure a CDN to dispatch requests to 2 distinct subdomain deployments for example.

Currently, Docusaurus always routes navigation to the main site.

I have no idea what you mean here.

Docusaurus doesn't "route", it's not a web server. It only builds a folder of static files.

You are responsible for configuring your web server / CDN to route requests to appropriate files.

You can also merge site1/build and site2/build into one static folder, but keep in mind that you'd rather use a baseUrl otherwise static files from one deployment will override the static files of another.

[johnenderson]

I was able to achieve what I wanted: I compiled my Docusaurus site separately in pt-BR and En, then merged them, adjusting the routes within AWS Amplify.

This approach was necessary because the i18n build process was consuming more memory than expected. Although I tried using NODE_OPTIONS="--max-old-space-size=4096", it didn't solve the issue. Additionally, my site is not only multilingual but also versioned, currently containing approximately 8,000 .mdx files. We are already using docusaurus faster, but due to the 7 GB memory limitation in the CI/CD pipeline, I had to split the build process to optimize resource usage.

[slorber]

I see thanks.

It's possible that you are affected by this memory leak issue: #10944

Would be curious to know at which step it crashes with OOM error. You can probably figure out using DOCUSAURUS_PERF_LOGGER=true when building.

[johnenderson]

Got it. Below is the output with DOCUSAURUS_PERF_LOGGER=true enabled.

yarn run v1.22.22
$ NODE_OPTIONS=${NODE_OPTIONS:-"--max-old-space-size=4096"} node node_modules/@docusaurus/core/bin/docusaurus.mjs build
[PERF] Get locales to build - 176.17 ms - (53mb -> 60mb)
[INFO] Website will be built for all these locales:
- pt-BR
- en
[INFO] [pt-BR] Creating an optimized production build...
[PERF] Build > pt-BR > Load site > Load plugins > Init plugins - 664.92 ms - (60mb -> 89mb)
[PERF] Build > pt-BR > Load site > Load plugins > Load plugins content > Load sitemap@default - 6.09 ms - (89mb -> 90mb)
[PERF] Build > pt-BR > Load site > Load plugins > Load plugins content > Load svgr@default - 6.20 ms - (89mb -> 90mb)
[PERF] Build > pt-BR > Load site > Load plugins > Load plugins content > Load sass@default - 5.48 ms - (89mb -> 90mb)
[PERF] Build > pt-BR > Load site > Load plugins > Load plugins content > Load mermaid@default - 5.53 ms - (89mb -> 90mb)
[PERF] Build > pt-BR > Load site > Load plugins > Load plugins content > Load docusaurus-bootstrap@default - 5.45 ms - (89mb -> 90mb)
[PERF] Build > pt-BR > Load site > Load plugins > Load plugins content > Load docusaurus-mdx-fallback@default - 5.74 ms - (89mb -> 90mb)
[PERF] Build > pt-BR > Load site > Load plugins > Load plugins content > Load classic@default > translatePluginContent() - 6.34 ms - (89mb -> 90mb)
[PERF] Build > pt-BR > Load site > Load plugins > Load plugins content > Load redoc@theme-redoc - 26.42 ms - (89mb -> 93mb)
[PERF] Build > pt-BR > Load site > Load plugins > Load plugins content > Load redoc@web-api-softexpert-2-2-0 > loadContent() - 99.29 ms - (89mb -> 100mb)
[PERF] Build > pt-BR > Load site > Load plugins > Load plugins content > Load redoc@web-api-softexpert-2-2-1 > loadContent() - 161.92 ms - (89mb -> 90mb)
[PERF] Build > pt-BR > Load site > Load plugins > Load plugins content > Load redoc@web-api-softexpert-2-2-2 > loadContent() - 205.76 ms - (89mb -> 93mb)
[PERF] Build > pt-BR > Load site > Load plugins > Load plugins content > Load redoc@web-api-softexpert-actual > loadContent() - 244.18 ms - (89mb -> 96mb)
[PERF] Build > pt-BR > Load site > Load plugins > Load plugins content > Load redoc@web-api-softexpert-suite-next > loadContent() - 266.10 ms - (89mb -> 96mb)
[PERF] Build > pt-BR > Load site > Load plugins > Load plugins content > Load search-algolia@default > getDefaultCodeTranslationMessages() - 265.14 ms - (90mb -> 96mb)
[PERF] Build > pt-BR > Load site > Load plugins > Load plugins content > Load classic@default > getDefaultCodeTranslationMessages() - 267.27 ms - (90mb -> 97mb)
[PERF] Build > pt-BR > Load site > Load plugins > Load plugins content > Load classic@default - 276.07 ms - (89mb -> 97mb)
[PERF] Build > pt-BR > Load site > Load plugins > Load plugins content > Load search-algolia@default - 292.88 ms - (89mb -> 99mb)
[PERF] Build > pt-BR > Load site > Load plugins > Load plugins content > Load pages@default > loadContent() - 323.48 ms - (89mb -> 102mb)
[PERF] Build > pt-BR > Load site > Load plugins > Load plugins content > Load redoc@web-api-softexpert-2-2-1 > contentLoaded() - 76.42 ms - (98mb -> 93mb)
[PERF] Build > pt-BR > Load site > Load plugins > Load plugins content > Load redoc@web-api-softexpert-2-2-1 - 362.21 ms - (89mb -> 93mb)
[PERF] Build > pt-BR > Load site > Load plugins > Load plugins content > Load redoc@web-api-softexpert-2-2-0 > contentLoaded() - 79.04 ms - (98mb -> 93mb)
[PERF] Build > pt-BR > Load site > Load plugins > Load plugins content > Load redoc@web-api-softexpert-2-2-0 - 366.43 ms - (89mb -> 93mb)
[PERF] Build > pt-BR > Load site > Load plugins > Load plugins content > Load redoc@web-api-softexpert-2-2-2 > contentLoaded() - 78.27 ms - (98mb -> 93mb)
[PERF] Build > pt-BR > Load site > Load plugins > Load plugins content > Load redoc@web-api-softexpert-2-2-2 - 366.08 ms - (89mb -> 93mb)
[PERF] Build > pt-BR > Load site > Load plugins > Load plugins content > Load redoc@web-api-softexpert-actual > contentLoaded() - 79.37 ms - (99mb -> 93mb)
[PERF] Build > pt-BR > Load site > Load plugins > Load plugins content > Load redoc@web-api-softexpert-actual - 369.15 ms - (89mb -> 93mb)
[PERF] Build > pt-BR > Load site > Load plugins > Load plugins content > Load redoc@web-api-softexpert-suite-next > contentLoaded() - 82.30 ms - (99mb -> 93mb)
[PERF] Build > pt-BR > Load site > Load plugins > Load plugins content > Load redoc@web-api-softexpert-suite-next - 373.15 ms - (89mb -> 93mb)
[PERF] Build > pt-BR > Load site > Load plugins > Load plugins content > Load pages@default > contentLoaded() - 32.13 ms - (93mb -> 95mb)
[PERF] Build > pt-BR > Load site > Load plugins > Load plugins content > Load pages@default - 384.81 ms - (89mb -> 95mb)
[PERF] Build > pt-BR > Load site > Load plugins > Load plugins content > Load docs@wiki > loadContent() - 394.62 ms - (89mb -> 96mb)
[PERF] Build > pt-BR > Load site > Load plugins > Load plugins content > Load docs@wiki > translatePluginContent() - 29.20 ms - (96mb -> 106mb)
[PERF] Build > pt-BR > Load site > Load plugins > Load plugins content > Load docs@wiki > contentLoaded() - 21.03 seconds! - (128mb -> 173mb)
[PERF] Build > pt-BR > Load site > Load plugins > Load plugins content > Load docs@wiki - 21.65 seconds! - (89mb -> 173mb)
[WARNING] Cannot infer the update date for some files, as they are not tracked by git.
[PERF] Build > pt-BR > Load site > Load plugins > Load plugins content > Load docs@default > loadContent() - 22.71 seconds! - (89mb -> 188mb)
[PERF] Build > pt-BR > Load site > Load plugins > Load plugins content > Load docs@default > translatePluginContent() - 6.98 ms - (188mb -> 178mb)
[PERF] Build > pt-BR > Load site > Load plugins > Load plugins content > Load docs@default > contentLoaded() - 404.73 ms - (178mb -> 212mb)
[PERF] Build > pt-BR > Load site > Load plugins > Load plugins content > Load docs@default - 23.12 seconds! - (89mb -> 212mb)
[PERF] Build > pt-BR > Load site > Load plugins > Load plugins content - 23.12 seconds! - (89mb -> 212mb)
[PERF] Build > pt-BR > Load site > Load plugins - 23.81 seconds! - (60mb -> 212mb)
[PERF] Build > pt-BR > Load site > Create site files - 411.12 ms - (215mb -> 246mb)
[PERF] Build > pt-BR > Load site - 24.23 seconds! - (60mb -> 246mb)
[PERF] Build > pt-BR > Creating rspack bundler configs - 162.72 ms - (246mb -> 242mb)
[PERF] Build > pt-BR > Bundling with rspack - 198.07 seconds! - (242mb -> 204mb)
[PERF] Build > pt-BR > SSG > Read client manifest - 11.93 ms - (204mb -> 207mb)
[PERF] Build > pt-BR > SSG > Generate static files > Load App renderer > Load server bundle - 6.17 ms - (212mb -> 212mb)
[PERF] Build > pt-BR > SSG > Generate static files > Load App renderer > Server bundle size = 6.228 MB
[PERF] Build > pt-BR > SSG > Generate static files > Load App renderer > Evaluate server bundle - 280.97 ms - (212mb -> 243mb)
[PERF] Build > pt-BR > SSG > Generate static files > Load App renderer - 288.01 ms - (212mb -> 243mb)
[PERF] Build > pt-BR > Deleting server bundle - 304.58 ms - (933mb -> 939mb)
[PERF] Build > pt-BR > postBuild() - 551.86 ms - (939mb -> 944mb)
[PERF] Build > pt-BR > Broken links checker - 1.75 seconds! - (944mb -> 952mb)
[SUCCESS] Generated static files in "build".
[PERF] Build > pt-BR - 319.15 seconds! - (60mb -> 952mb)
[INFO] [en] Creating an optimized production build...
[PERF] Build > en > Load site > Load context - 8.67 ms - (952mb -> 953mb)
[PERF] Build > en > Load site > Load plugins > Init plugins - 19.01 ms - (953mb -> 957mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load sitemap@default - 7.72 ms - (957mb -> 958mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load svgr@default - 7.81 ms - (957mb -> 958mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load sass@default - 7.15 ms - (957mb -> 958mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load mermaid@default - 7.15 ms - (957mb -> 958mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load docusaurus-bootstrap@default - 7.04 ms - (957mb -> 958mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load docusaurus-mdx-fallback@default - 7.02 ms - (957mb -> 958mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load search-algolia@default > getDefaultCodeTranslationMessages() - 9.46 ms - (958mb -> 960mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load redoc@theme-redoc - 17.19 ms - (957mb -> 960mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load redoc@web-api-softexpert-2-2-0 > loadContent() - 75.59 ms - (957mb -> 965mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load redoc@web-api-softexpert-actual > loadContent() - 130.53 ms - (957mb -> 952mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load redoc@web-api-softexpert-2-2-2 > loadContent() - 176.08 ms - (957mb -> 954mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load redoc@web-api-softexpert-2-2-1 > loadContent() - 225.65 ms - (957mb -> 956mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load redoc@web-api-softexpert-suite-next > loadContent() - 248.63 ms - (957mb -> 956mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load classic@default > translatePluginContent() - 247.24 ms - (958mb -> 956mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load search-algolia@default - 262.69 ms - (957mb -> 958mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load classic@default > getDefaultCodeTranslationMessages() - 18.77 ms - (956mb -> 959mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load classic@default - 269.29 ms - (957mb -> 959mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load pages@default > loadContent() - 278.40 ms - (957mb -> 960mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load redoc@web-api-softexpert-2-2-0 > contentLoaded() - 59.91 ms - (958mb -> 964mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load redoc@web-api-softexpert-2-2-0 - 325.84 ms - (957mb -> 964mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load redoc@web-api-softexpert-actual > contentLoaded() - 59.86 ms - (958mb -> 964mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load redoc@web-api-softexpert-actual - 326.04 ms - (957mb -> 964mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load redoc@web-api-softexpert-2-2-1 > contentLoaded() - 58.92 ms - (959mb -> 964mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load redoc@web-api-softexpert-2-2-1 - 327.16 ms - (957mb -> 964mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load redoc@web-api-softexpert-suite-next > contentLoaded() - 58.50 ms - (959mb -> 964mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load redoc@web-api-softexpert-suite-next - 327.30 ms - (957mb -> 964mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load redoc@web-api-softexpert-2-2-2 > contentLoaded() - 59.95 ms - (958mb -> 964mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load redoc@web-api-softexpert-2-2-2 - 327.57 ms - (957mb -> 964mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load pages@default > contentLoaded() - 31.28 ms - (963mb -> 965mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load pages@default - 336.36 ms - (957mb -> 965mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load docs@wiki > loadContent() - 351.25 ms - (957mb -> 966mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load docs@wiki > translatePluginContent() - 231.47 ms - (966mb -> 983mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load docs@wiki > contentLoaded() - 330.10 seconds! - (593mb -> 604mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load docs@wiki - 409.88 seconds! - (957mb -> 604mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load docs@default > loadContent() - 410.69 seconds! - (957mb -> 593mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load docs@default > translatePluginContent() - 12.90 ms - (593mb -> 588mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load docs@default > contentLoaded() - 933.47 ms - (588mb -> 552mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load docs@default - 411.64 seconds! - (957mb -> 552mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content - 411.64 seconds! - (957mb -> 552mb)
[PERF] Build > en > Load site > Load plugins > allContentLoaded() - 11.25 ms - (552mb -> 552mb)
[PERF] Build > en > Load site > Load plugins - 411.68 seconds! - (953mb -> 552mb)
[PERF] Build > en > Load site > Create site files - 408.82 ms - (554mb -> 585mb)
[PERF] Build > en > Load site - 412.12 seconds! - (952mb -> 585mb)
[PERF] Build > en > Creating rspack bundler configs - 177.85 ms - (585mb -> 579mb)
error Command failed with signal "SIGKILL".
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

[slorber]

Hmmm, there's not enough logs, some I added recently

Could you try using a Docusaurus canary? It might improve things by running SSG in worker threads (less likely to leak memory)

error Command failed with signal "SIGKILL".

Weird, did you ctrl+c to stop the process?