homelab/stack
1
0
Fork 0
Code Issues 47 Pull Requests 1 Actions Packages Projects 1 Releases Wiki Activity

[nb] gpu_test.py: exception: Try lowering target_partition_size (current 322122547) and/or RAPIDSMPF_SPILL_DEVICE_LIMIT (default '80%') to reduce p #810

New Issue
Closed
opened 2026-09-24 03:37:06 +00:00 by kert · 1 comment
kert commented 2026-09-24 03:37:06 +00:00
Owner
Copy Link

Auto-filed by the notebook error filer (nb-sig:ef329a173e1d — do not edit this marker; dedup and auto-close key on it).

  • Notebook: gpu_test.py
  • Error: exception: Try lowering target_partition_size (current 322122547) and/or RAPIDSMPF_SPILL_DEVICE_LIMIT (default '80%') to reduce peak memory.
    See https://docs.rapids.ai/api/cudf/stable/cudf_polars/memory_errors/ for troubleshooting guidance.
    Original error:
    std::bad_alloc: out_of_memory: CUDA error (failed to allocate 16000000 bytes) at: /__w/rmm/rmm/cpp/src/mr/cuda_async_view_memory_resource.cpp:43: cudaErrorMemoryAllocation out of memory
  • Source: nb-watcher
+ Exception Group Traceback (most recent call last):
  |   File "/home/kert/workspace/.venv/lib/python3.13/site-packages/cudf_polars/engine/core.py", line 513, in execute_ir_on_rank
  |     run_actor_network(ctx, actors=nodes)
  |     ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  |   File "rapidsmpf/streaming/core/actor.pyx", line 361, in rapidsmpf.streaming.core.actor.run_actor_network
  |   File "rapidsmpf/streaming/core/actor.pyx", line 369, in rapidsmpf.streaming.core.actor.run_actor_network
  |   File "rapidsmpf/streaming/core/actor.pyx", line 366, in rapidsmpf.streaming.core.actor.run_actor_network
  |   File "/home/kert/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/concurrent/futures/_base.py", line 460, in result
  |     return self.__get_result()
  |            ~~~~~~~~~~~~~~~~~^^
  |   File "/home/kert/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/concurrent/futures/_base.py", line 402, in __get_result
  |     raise self._exception
  |   File "/home/kert/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/concurrent/futures/thread.py", line 59, in run
  |     result = self.fn(*self.args, **self.kwargs)
  |   File "rapidsmpf/streaming/core/actor.pyx", line 268, in rapidsmpf.streaming.core.actor.sync_wait
  |   File "rapidsmpf/streaming/core/actor.pyx", line 263, in rapidsmpf.streaming.core.actor.sync_wait
  |   File "rapidsmpf/streaming/core/actor.pyx", line 264, in rapidsmpf.streaming.core.actor.sync_wait
  |   File "/home/kert/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/asyncio/runners.py", line 119, in run
  |     return self._loop.run_until_complete(task)
  |            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  |   File "/home/kert/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/asyncio/base_events.py", line 726, in run_until_complete
  |     return future.result()
  |            ~~~~~~~~~~~~~^^
  |   File "rapidsmpf/streaming/core/actor.pyx", line 246, in run_and_publish_task
  |   File "rapidsmpf/streaming/core/actor.pyx", line 172, in run_py_actors
  |   File "/home/kert/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/asyncio/taskgroups.py", line 71, in __aexit__
  |     return await self._aexit(et, exc)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/kert/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/asyncio/taskgroups.py", line 173, in _aexit
  |     raise BaseExceptionGroup(
  |     ...<2 lines>...
  |     ) from None
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Exception Group Traceback (most recent call last):
    |   File "rapidsmpf/streaming/core/actor.pyx", line 158, in py_actor
    |   File "/home/kert/workspace/.venv/lib/python3.13/site-packages/cudf_polars/streaming/actor_graph/io.py", line 285, in dataframescan_node
    |     await gather_in_task_group(
    |     ...<5 lines>...
    |     )
    |   File "/home/kert/workspace/.venv/lib/python3.13/site-packages/cudf_polars/streaming/actor_graph/utils.py", line 220, in gather_in_task_group
    |     async with asyncio.TaskGroup() as tg:
    |                ~~~~~~~~~~~~~~~~~^^
    |   File "/home/kert/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/asyncio/taskgroups.py", line 71, in __aexit__
    |     return await self._aexit(et, exc)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/kert/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/asyncio/taskgroups.py", line 173, in _aexit
    |     raise BaseExceptionGroup(
    |     ...<2 lines>...
    |     ) from None
    | ExceptionGroup: unhandled errors in a TaskGroup (2 sub-exceptions)
    +-+---------------- 1 ----------------
      | Traceback (most recent call last):
      |   File "/home/kert/workspace/.venv/lib/python3.13/site-packages/cudf_polars/streaming/actor_graph/io.py", line 271, in _producer

This issue is closed automatically when the error stops occurring.

Auto-filed by the notebook error filer (`nb-sig:ef329a173e1d` — do not edit this marker; dedup and auto-close key on it). - **Notebook:** `gpu_test.py` - **Error:** `exception`: Try lowering `target_partition_size` (current 322122547) and/or RAPIDSMPF_SPILL_DEVICE_LIMIT (default '80%') to reduce peak memory. See https://docs.rapids.ai/api/cudf/stable/cudf_polars/memory_errors/ for troubleshooting guidance. Original error: std::bad_alloc: out_of_memory: CUDA error (failed to allocate 16000000 bytes) at: /__w/rmm/rmm/cpp/src/mr/cuda_async_view_memory_resource.cpp:43: cudaErrorMemoryAllocation out of memory - **Source:** nb-watcher ``` + Exception Group Traceback (most recent call last): | File "/home/kert/workspace/.venv/lib/python3.13/site-packages/cudf_polars/engine/core.py", line 513, in execute_ir_on_rank | run_actor_network(ctx, actors=nodes) | ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ | File "rapidsmpf/streaming/core/actor.pyx", line 361, in rapidsmpf.streaming.core.actor.run_actor_network | File "rapidsmpf/streaming/core/actor.pyx", line 369, in rapidsmpf.streaming.core.actor.run_actor_network | File "rapidsmpf/streaming/core/actor.pyx", line 366, in rapidsmpf.streaming.core.actor.run_actor_network | File "/home/kert/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/concurrent/futures/_base.py", line 460, in result | return self.__get_result() | ~~~~~~~~~~~~~~~~~^^ | File "/home/kert/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/concurrent/futures/_base.py", line 402, in __get_result | raise self._exception | File "/home/kert/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/concurrent/futures/thread.py", line 59, in run | result = self.fn(*self.args, **self.kwargs) | File "rapidsmpf/streaming/core/actor.pyx", line 268, in rapidsmpf.streaming.core.actor.sync_wait | File "rapidsmpf/streaming/core/actor.pyx", line 263, in rapidsmpf.streaming.core.actor.sync_wait | File "rapidsmpf/streaming/core/actor.pyx", line 264, in rapidsmpf.streaming.core.actor.sync_wait | File "/home/kert/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/asyncio/runners.py", line 119, in run | return self._loop.run_until_complete(task) | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ | File "/home/kert/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/asyncio/base_events.py", line 726, in run_until_complete | return future.result() | ~~~~~~~~~~~~~^^ | File "rapidsmpf/streaming/core/actor.pyx", line 246, in run_and_publish_task | File "rapidsmpf/streaming/core/actor.pyx", line 172, in run_py_actors | File "/home/kert/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/asyncio/taskgroups.py", line 71, in __aexit__ | return await self._aexit(et, exc) | ^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/kert/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/asyncio/taskgroups.py", line 173, in _aexit | raise BaseExceptionGroup( | ...<2 lines>... | ) from None | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception) +-+---------------- 1 ---------------- | Exception Group Traceback (most recent call last): | File "rapidsmpf/streaming/core/actor.pyx", line 158, in py_actor | File "/home/kert/workspace/.venv/lib/python3.13/site-packages/cudf_polars/streaming/actor_graph/io.py", line 285, in dataframescan_node | await gather_in_task_group( | ...<5 lines>... | ) | File "/home/kert/workspace/.venv/lib/python3.13/site-packages/cudf_polars/streaming/actor_graph/utils.py", line 220, in gather_in_task_group | async with asyncio.TaskGroup() as tg: | ~~~~~~~~~~~~~~~~~^^ | File "/home/kert/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/asyncio/taskgroups.py", line 71, in __aexit__ | return await self._aexit(et, exc) | ^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/kert/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/asyncio/taskgroups.py", line 173, in _aexit | raise BaseExceptionGroup( | ...<2 lines>... | ) from None | ExceptionGroup: unhandled errors in a TaskGroup (2 sub-exceptions) +-+---------------- 1 ---------------- | Traceback (most recent call last): | File "/home/kert/workspace/.venv/lib/python3.13/site-packages/cudf_polars/streaming/actor_graph/io.py", line 271, in _producer ``` This issue is closed automatically when the error stops occurring.
kert added the notebooks label 2026-09-24 03:37:06 +00:00
kert commented 2026-09-24 03:44:30 +00:00
Author
Owner
Copy Link

No longer occurring (nightly-llm-golden); auto-closing.

No longer occurring (nightly-llm-golden); auto-closing.
kert closed this issue 2026-09-24 03:44:30 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
api
REST API (FastAPI)
 auth
Authentication and authorization
 bib
Bibliography, Zotero, citations
 ci
CI/CD pipelines and Woodpecker plugins
 cli
CLI commands and UX
 cloud
Cloud provider abstraction (AWS, GCP, Azure, self-hosted)
 config
Configuration and wiring
 data
Data loading, CCLF, BCDA, seeds
 databricks
Databricks jobs and deployment
 docs
Documentation and generated references
 image:api
image:docs
image:mc
image:notebooks
image:zotero
infra
lakehouse
Nessie, Polaris, Trino, Iceberg
 llm
llm module — local RAG over comments + Zotero corpus
 notebooks
marimo notebook errors (auto-filed by nb-watcher/nb-integration)
 packaging
Python packaging, optional deps, build system
 pfs
pipeline
Pipeline execution and validation
 pipeline:ci
pipeline:deploy
pipeline:harden
pipeline:infra-ci
pipeline:rebuild-all
pipeline:release
qpp
quality
Testing, coverage, error handling
 step:build
step:deploy
step:hadolint
step:lint
step:publish
step:scan
step:test
step:validate
styles
Charts, figures, visual theming
 tree
Project directory structure and file organization
No Label notebooks
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
kert renovate-bot
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: homelab/stack#810
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?