Document heterogeneous node groups for tasks - #4154
Conversation
Co-authored-by: Cursor <cursoragent@cursor.com>
peterschmidt85
left a comment
peterschmidt85
left a comment
There was a problem hiding this comment.
- Do you think we could also add "???" (collapsible) section with P/D example?
- Do you think it would make sense to also re-write our
mpirunand ray examples - and link them from here?
| `${{ groups[i].nodes[j].IP_ADDRESS }}`, where `i` is the index of the group in `groups` and `j` is | ||
| the index of the node within that group. | ||
|
|
||
| Node `groups[0].nodes[0]` is the run's master node — it is what `DSTACK_MASTER_NODE_IP` resolves |
There was a problem hiding this comment.
IMO, this can use a blockquote to highlight this?
P/D is a service setup, and the next PR switches services from replicas: to groups: and adds interpolation. I think it would be better to add it once that PR lands, when the example can use groups: and ${{ groups[i].replicas[j].IP_ADDRESS }}.
|
I meant adding a task example |
| the index of the node within that group. | ||
|
|
||
| > Node `groups[0].nodes[0]` is the run's master node — it is what `DSTACK_MASTER_NODE_IP` resolves | ||
| > to, and `startup_order` and `stop_criteria` apply to it across all groups. |
There was a problem hiding this comment.
and
startup_orderandstop_criteriaapply to it across all groups.
is't it a bit vague? what exactly is the point we want to make?
There was a problem hiding this comment.
Agreed. Groups don't change startup_order / stop_criteria behavior. The master is still job_num 0 — so I've removed that line.
| cpu: 4 | ||
|
|
||
| - name: prefill | ||
| nodes: 1 |
There was a problem hiding this comment.
is 1 a representative example? just want to confirm
same question for prefill group
There was a problem hiding this comment.
I think yes for a Node groups example.
There was a problem hiding this comment.
But it may mislead, for master its most often 1 but for workers most often not
| fi | ||
|
|
||
| groups: | ||
| - name: master # node group name is optional |
There was a problem hiding this comment.
Maybe node group name is optional -> The name property is optional
?
| # Run RCCL tests via MPI | ||
| - | | ||
| mpirun --allow-run-as-root \ | ||
| --hostfile $DSTACK_MPI_HOSTFILE \ |
There was a problem hiding this comment.
Just a question, do we explain anywhere what DSTACK_MPI_HOSTFILE means in the context of node groups?
There was a problem hiding this comment.
I have added below line
The file lists GPU nodes as <ip> slots=<gpus> and CPU nodes as <ip>.
| disk: 1000GB.. | ||
|
|
||
| groups: | ||
| - name: head # node group name is optional |
There was a problem hiding this comment.
Maybe node group name is optional -> The name property is optional
?
peterschmidt85
left a comment
peterschmidt85
left a comment
There was a problem hiding this comment.
Left few minor comments
2 participants
Summary
nodes/resources/commands/portsand${{ groups[i].nodes[j].IP_ADDRESS }}references.