perf(map): chunk album fetching to prevent OOM errors (#138)

Replaced the unbounded `Promise.all` in `getMapData` with a chunked
fetching approach. This prevents Out of Memory crashes by limiting
the concurrent fetching of large arrays of assets to chunks of 10.

Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
This commit is contained in:
Ralksta
2026-04-24 22:12:45 +02:00
committed by GitHub
parent f18d9ae3d9
commit 56f0305763
2 changed files with 12 additions and 1 deletions

View File

@@ -11,3 +11,6 @@
**Learning:** `Buffer` inherits from `Uint8Array` in Node.js. Passing a `Buffer` into `new Uint8Array(...)` creates a completely new `ArrayBuffer` and copies all elements, which is an O(N) memory allocation and copy.
**Action:** When decoding base64 strings or similar byte streams in Node.js where a `Uint8Array` is expected, return `Buffer.from(data, 'base64')` directly instead of wrapping it.
## 2026-04-24 - [OOM prevention through promise chunking]
**Learning:** When fetching unbounded large collections (like photo albums and their assets), using concurrent `Promise.all` over the entire array can cause Node.js Out of Memory (OOM) crashes.
**Action:** Instead of `Promise.all(array.map(...))`, use a chunked data fetching approach (e.g., `Promise.all` within a `for` loop processing chunks of e.g. 10 items) to balance memory constraints with concurrent network speed.

View File

@@ -34,7 +34,15 @@ export async function getMapData(): Promise<MapLocation[]> {
}
// Fetch full album data (with assets) for each
const fullAlbums = await Promise.all(albums.map((a) => immich.getAlbum(a.id)));
// Process in chunks of 10 to balance network speed and prevent Out of Memory (OOM)
// crashes from fetching thousands of assets simultaneously.
const fullAlbums: (Awaited<ReturnType<typeof immich.getAlbum>> | null)[] = [];
const chunkSize = 10;
for (let i = 0; i < albums.length; i += chunkSize) {
const chunk = albums.slice(i, i + chunkSize);
const chunkResults = await Promise.all(chunk.map((a) => immich.getAlbum(a.id)));
fullAlbums.push(...chunkResults);
}
// Bucket assets by city+country
const buckets = new Map<