Handle message in argo submit --watch #4

New Issue

pb · 2025-04-09T18:37:06+02:00

pb commented

2025-04-09 18:37:06 +02:00

When executing argo submit with the --watch option we can retrieve the evolution of the workflow, but we don't retrieve the content of the message displayed for each steps, like in the following :

ServiceAccount:      sa-test-admiralty
Status:              Running
Conditions:          
 PodRunning          False
Created:             Wed Apr 09 18:05:29 +0200 (5 minutes ago)
Started:             Wed Apr 09 18:05:29 +0200 (5 minutes ago)
Duration:            5 minutes 29 seconds
Progress:            1/2
ResourcesDuration:   24s*(1 cpu),24s*(100Mi memory)

STEP                                              TEMPLATE                                     PODNAME                                                                           DURATION  MESSAGE
 ● oc-monitor-triling-graler                      dag                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
 ├─✔ alpine-fbe8cdaf-49a6-417c-bf2a-892b4e1ea4bf  alpine-fbe8cdaf-49a6-417c-bf2a-892b4e1ea4bf  oc-monitor-triling-graler-alpine-fbe8cdaf-49a6-417c-bf2a-892b4e1ea4bf-2933567912  15s                                                                                                                                                                                                                                                                                                                                            
 └─◷ alpine-f5eeb1f8-567b-4d96-bc2b-4d4af177a1d9  alpine-f5eeb1f8-567b-4d96-bc2b-4d4af177a1d9  oc-monitor-triling-graler-alpine-f5eeb1f8-567b-4d96-bc2b-4d4af177a1d9-1055991309  5m        Unschedulable: 0/4 nodes are available: 1 Too many pods, 1 Unauthorized, 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) had untolerated taint {node.kubernetes.io/unreachable: }. preemption: 0/4 nodes are available: 1 No preemption victims found for incoming pod, 3 Preemption is not helpful for scheduling..

In this case the workflow stays in the Running state although it is stuck.

We should monitor for the output of each step and check for indicator of failure, here Unschedulable. This can also create richer logs

When executing argo submit with the `--watch` option we can retrieve the evolution of the workflow, but we don't retrieve the content of the message displayed for each steps, like in the following : ``` ServiceAccount: sa-test-admiralty Status: Running Conditions: PodRunning False Created: Wed Apr 09 18:05:29 +0200 (5 minutes ago) Started: Wed Apr 09 18:05:29 +0200 (5 minutes ago) Duration: 5 minutes 29 seconds Progress: 1/2 ResourcesDuration: 24s*(1 cpu),24s*(100Mi memory) STEP TEMPLATE PODNAME DURATION MESSAGE ● oc-monitor-triling-graler dag ├─✔ alpine-fbe8cdaf-49a6-417c-bf2a-892b4e1ea4bf alpine-fbe8cdaf-49a6-417c-bf2a-892b4e1ea4bf oc-monitor-triling-graler-alpine-fbe8cdaf-49a6-417c-bf2a-892b4e1ea4bf-2933567912 15s └─◷ alpine-f5eeb1f8-567b-4d96-bc2b-4d4af177a1d9 alpine-f5eeb1f8-567b-4d96-bc2b-4d4af177a1d9 oc-monitor-triling-graler-alpine-f5eeb1f8-567b-4d96-bc2b-4d4af177a1d9-1055991309 5m Unschedulable: 0/4 nodes are available: 1 Too many pods, 1 Unauthorized, 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) had untolerated taint {node.kubernetes.io/unreachable: }. preemption: 0/4 nodes are available: 1 No preemption victims found for incoming pod, 3 Preemption is not helpful for scheduling.. ``` In this case the workflow stays in the `Running` state although it is stuck. We should monitor for the output of each step and check for indicator of failure, here `Unschedulable`. This can also create richer logs

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: core/oc-monitord#4