perf(map): chunk album fetching to prevent OOM errors (#138)
Replaced the unbounded `Promise.all` in `getMapData` with a chunked fetching approach. This prevents Out of Memory crashes by limiting the concurrent fetching of large arrays of assets to chunks of 10. Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
This commit is contained in:
@@ -11,3 +11,6 @@
|
||||
|
||||
**Learning:** `Buffer` inherits from `Uint8Array` in Node.js. Passing a `Buffer` into `new Uint8Array(...)` creates a completely new `ArrayBuffer` and copies all elements, which is an O(N) memory allocation and copy.
|
||||
**Action:** When decoding base64 strings or similar byte streams in Node.js where a `Uint8Array` is expected, return `Buffer.from(data, 'base64')` directly instead of wrapping it.
|
||||
## 2026-04-24 - [OOM prevention through promise chunking]
|
||||
**Learning:** When fetching unbounded large collections (like photo albums and their assets), using concurrent `Promise.all` over the entire array can cause Node.js Out of Memory (OOM) crashes.
|
||||
**Action:** Instead of `Promise.all(array.map(...))`, use a chunked data fetching approach (e.g., `Promise.all` within a `for` loop processing chunks of e.g. 10 items) to balance memory constraints with concurrent network speed.
|
||||
|
||||
@@ -34,7 +34,15 @@ export async function getMapData(): Promise<MapLocation[]> {
|
||||
}
|
||||
|
||||
// Fetch full album data (with assets) for each
|
||||
const fullAlbums = await Promise.all(albums.map((a) => immich.getAlbum(a.id)));
|
||||
// Process in chunks of 10 to balance network speed and prevent Out of Memory (OOM)
|
||||
// crashes from fetching thousands of assets simultaneously.
|
||||
const fullAlbums: (Awaited<ReturnType<typeof immich.getAlbum>> | null)[] = [];
|
||||
const chunkSize = 10;
|
||||
for (let i = 0; i < albums.length; i += chunkSize) {
|
||||
const chunk = albums.slice(i, i + chunkSize);
|
||||
const chunkResults = await Promise.all(chunk.map((a) => immich.getAlbum(a.id)));
|
||||
fullAlbums.push(...chunkResults);
|
||||
}
|
||||
|
||||
// Bucket assets by city+country
|
||||
const buckets = new Map<
|
||||
|
||||
Reference in New Issue
Block a user