Skip to content

Commit

Permalink
[cuda][hip] Fix launch host func and worker thread state update (#16568)
Browse files Browse the repository at this point in the history
This commits fixes a few issues in pending action queue to
resolve driver deadlock issues:

* In host launch func, which is called from a driver thread,
  we cannot invoke any GPU API. Otherwise we might see
  deadlock. This includes cleaning up the actions after
  execution--it may involve buffer releasing/unregistering
  which was the issue causing hip driver hang. Now move
  this cleanup into the worker thread. This is done by adding
  a state field to each action to indicate whether it's alive
  or zombie. We enqueue each action again after done
  execution by flipping its state to zombie to let the worker
  thread to cleanup.
* The worker thread can have five states--two normal states
  (idle waiting or workload pending), three exit states (requested,
  committed, error). They have increasing priorities w.r.t.
  overwriting. We cannot overwrite state later in the list
  without checking. This guarantees that exit requests are
  properly respected and not dropping to the floor so to have
  clean exit.
* When the worker thread is waken to process ready list, we
  need to immediately flip the worker state from workload
  pending to idle waiting, before any real processing. This
  makes sure we don't drop new workload enqueued while
  we are processing, and the worker thread can be waken
  up again properly later.

With the above fixes, we can pass all stablehlo/tosa e2e op
tests on hip driver without hang or crashes. The same
change is mirrored to the cuda pending action queue.

Fixes #15790
Progress towards #16504
  • Loading branch information
antiagainst authored Feb 27, 2024
1 parent baeffa7 commit 9dc8ae4
Show file tree
Hide file tree
Showing 2 changed files with 278 additions and 124 deletions.
Loading

0 comments on commit 9dc8ae4

Please sign in to comment.