perf(map): chunk album fetching to prevent OOM errors (#138)
Replaced the unbounded `Promise.all` in `getMapData` with a chunked fetching approach. This prevents Out of Memory crashes by limiting the concurrent fetching of large arrays of assets to chunks of 10. Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
This commit is contained in:
@@ -11,3 +11,6 @@
|
|||||||
|
|
||||||
**Learning:** `Buffer` inherits from `Uint8Array` in Node.js. Passing a `Buffer` into `new Uint8Array(...)` creates a completely new `ArrayBuffer` and copies all elements, which is an O(N) memory allocation and copy.
|
**Learning:** `Buffer` inherits from `Uint8Array` in Node.js. Passing a `Buffer` into `new Uint8Array(...)` creates a completely new `ArrayBuffer` and copies all elements, which is an O(N) memory allocation and copy.
|
||||||
**Action:** When decoding base64 strings or similar byte streams in Node.js where a `Uint8Array` is expected, return `Buffer.from(data, 'base64')` directly instead of wrapping it.
|
**Action:** When decoding base64 strings or similar byte streams in Node.js where a `Uint8Array` is expected, return `Buffer.from(data, 'base64')` directly instead of wrapping it.
|
||||||
|
## 2026-04-24 - [OOM prevention through promise chunking]
|
||||||
|
**Learning:** When fetching unbounded large collections (like photo albums and their assets), using concurrent `Promise.all` over the entire array can cause Node.js Out of Memory (OOM) crashes.
|
||||||
|
**Action:** Instead of `Promise.all(array.map(...))`, use a chunked data fetching approach (e.g., `Promise.all` within a `for` loop processing chunks of e.g. 10 items) to balance memory constraints with concurrent network speed.
|
||||||
|
|||||||
@@ -34,7 +34,15 @@ export async function getMapData(): Promise<MapLocation[]> {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Fetch full album data (with assets) for each
|
// Fetch full album data (with assets) for each
|
||||||
const fullAlbums = await Promise.all(albums.map((a) => immich.getAlbum(a.id)));
|
// Process in chunks of 10 to balance network speed and prevent Out of Memory (OOM)
|
||||||
|
// crashes from fetching thousands of assets simultaneously.
|
||||||
|
const fullAlbums: (Awaited<ReturnType<typeof immich.getAlbum>> | null)[] = [];
|
||||||
|
const chunkSize = 10;
|
||||||
|
for (let i = 0; i < albums.length; i += chunkSize) {
|
||||||
|
const chunk = albums.slice(i, i + chunkSize);
|
||||||
|
const chunkResults = await Promise.all(chunk.map((a) => immich.getAlbum(a.id)));
|
||||||
|
fullAlbums.push(...chunkResults);
|
||||||
|
}
|
||||||
|
|
||||||
// Bucket assets by city+country
|
// Bucket assets by city+country
|
||||||
const buckets = new Map<
|
const buckets = new Map<
|
||||||
|
|||||||
Reference in New Issue
Block a user