Knative Parallel and Sequence demo fails

Question

I am trying out knative sequence/parallel flows.

I started with sequence example mentioned in official website for release v1.1 I created steps, sequence and pingSource as mentioned in the document but pingSource failed because sequence was not up.

Sequence has below exception

Events:
  Type     Reason        Age              From                 Message
  ----     ------        ----             ----                 -------
  Warning  UpdateFailed  8s (x6 over 8s)  sequence-controller  Failed to update status for "sequence": Sequence.flows.knative.dev "sequence" is invalid: [status.channelStatuses: Invalid value: "null": status.channelStatuses in body must be of type array: "null", status.subscriptionStatuses: Invalid value: "null": status.subscriptionStatuses in body must be of type array: "null"]
  Warning  UpdateFailed  3s (x5 over 8s)  sequence-controller  Failed to update status for "sequence": Sequence.flows.knative.dev "sequence" is invalid: [status.subscriptionStatuses: Invalid value: "null": status.subscriptionStatuses in body must be of type array: "null", status.channelStatuses: Invalid value: "null": status.channelStatuses in body must be of type array: "null"]

but steps are running fine

NAME                                     TYPE           CLUSTER-IP      EXTERNAL-IP                            PORT(S)                                      AGE
service/first                            ExternalName   <none>          first.varadhi.example.com              80/TCP                                       9m46s
service/first-00001                      ClusterIP      10.96.116.201   <none>                                 80/TCP                                       9m51s
service/first-00001-private              ClusterIP      10.96.155.146   <none>                                 80/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   9m51s
service/second                           ExternalName   <none>          second.varadhi.example.com             80/TCP                                       9m45s
service/second-00001                     ClusterIP      10.96.208.230   <none>                                 80/TCP                                       9m51s
service/second-00001-private             ClusterIP      10.96.171.83    <none>                                 80/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   9m51s
service/third                            ExternalName   <none>          third.varadhi.example.com              80/TCP                                       9m45s
service/third-00001                      ClusterIP      10.96.131.110   <none>                                 80/TCP                                       9m51s
service/third-00001-private              ClusterIP      10.96.55.219    <none>                                 80/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   9m51s

NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/first-00001-deployment              0/0     0            0           9m52s
deployment.apps/second-00001-deployment             0/0     0            0           9m52s
deployment.apps/third-00001-deployment              0/0     0            0           9m52s

NAME                                                           DESIRED   CURRENT   READY   AGE
replicaset.apps/first-00001-deployment-594dc84cb8              0         0         0       9m52s
replicaset.apps/second-00001-deployment-79d9f8b7b8             0         0         0       9m52s
replicaset.apps/third-00001-deployment-7479456fdf              0         0         0       9m51s

NAME                                                  URL   AGE   READY     REASON
channel.messaging.knative.dev/varadhi-inmem-channel         26h   Unknown   NewObservedGenFailure

NAME                                           URL                                           LATESTCREATED            LATESTREADY              READY     REASON
service.serving.knative.dev/first              http://first.varadhi.example.com              first-00001              first-00001              Unknown   IngressNotConfigured
service.serving.knative.dev/second             http://second.varadhi.example.com             second-00001             second-00001             Unknown   IngressNotConfigured
service.serving.knative.dev/third              http://third.varadhi.example.com              third-00001              third-00001              Unknown   IngressNotConfigured

NAME                                                  CONFIG NAME        K8S SERVICE NAME   GENERATION   READY   REASON             ACTUAL REPLICAS   DESIRED REPLICAS
revision.serving.knative.dev/first-00001              first                                 1            True                       0                 0
revision.serving.knative.dev/second-00001             second                                1            True                       0                 0
revision.serving.knative.dev/third-00001              third                                 1            True                       0                 0

NAME                                                 LATESTCREATED            LATESTREADY              READY   REASON
configuration.serving.knative.dev/first              first-00001              first-00001              True
configuration.serving.knative.dev/second             second-00001             second-00001             True
configuration.serving.knative.dev/third              third-00001              third-00001              True

NAME                                         URL                                           READY     REASON
route.serving.knative.dev/first              http://first.varadhi.example.com              Unknown   IngressNotConfigured
route.serving.knative.dev/second             http://second.varadhi.example.com             Unknown   IngressNotConfigured
route.serving.knative.dev/third              http://third.varadhi.example.com              Unknown   IngressNotConfigured

After spending ample time on knative sequence, I decided to try out knative parallel

I referred official documentation of parallel for v1.1 and used multiple-branches examples I created filters, transformers, parallel and pingSource but here also parallel didn't come up with below exception

Events:
  Type     Reason        Age                   From                 Message
  ----     ------        ----                  ----                 -------
  Warning  UpdateFailed  6m37s (x18 over 17m)  parallel-controller  Failed to update status for "odd-even-parallel": Parallel.flows.knative.dev "odd-even-parallel" is invalid: status.branchStatuses: Invalid value: "null": status.branchStatuses in body must be of type array: "null"

Here also parallel and sequence are up and running fine.

Has anyone faced similar issues or am I missing anything from official documentation ?

Environment info:

Using Kind Cluster
Using local docker registry (have bypassed digest check on images)
namespace : varadhi
Serving and Eventing CRDS are of knative v1.1

Edit #1

I have not created any channels explicitly and I do not see any channels being created implicitly by controller either

kubectl get channel -n varadhi 
No resources found in varadhi namespace.

Also my default channel is InMemoryChannel

anil.gowda@faas-dev-kafka-8420816:~/knative$ kubectl get configmaps -n knative-eventing  default-ch-webhook -o yaml
apiVersion: v1
data:
  default-ch-config: |
    clusterDefault:
      apiVersion: messaging.knative.dev/v1
      kind: InMemoryChannel
    namespaceDefaults:
      varadhi:
        apiVersion: messaging.knative.dev/v1
        kind: InMemoryChannel
kind: ConfigMap

Parallel :

Example Used : https://github.com/knative/docs/tree/main/code-samples/eventing/parallel/multiple-branches

Status

NAME                                     TYPE           CLUSTER-IP      EXTERNAL-IP                            PORT(S)                                      AGE
service/even-filter                      ExternalName   <none>          even-filter.varadhi.example.com        80/TCP                                       4d19h
service/even-filter-00001                ClusterIP      10.96.85.252    <none>                                 80/TCP                                       4d19h
service/even-filter-00001-private        ClusterIP      10.96.98.109    <none>                                 80/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   4d19h
service/even-transformer                 ExternalName   <none>          even-transformer.varadhi.example.com   80/TCP                                       4d19h
service/even-transformer-00001           ClusterIP      10.96.152.53    <none>                                 80/TCP                                       4d19h
service/even-transformer-00001-private   ClusterIP      10.96.130.58    <none>                                 80/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   4d19h
service/event-display                    ExternalName   <none>          event-display.varadhi.example.com      80/TCP                                       4d19h
service/event-display-00001              ClusterIP      10.96.237.175   <none>                                 80/TCP                                       4d19h
service/event-display-00001-private      ClusterIP      10.96.81.3      <none>                                 80/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   4d19h
service/odd-filter                       ExternalName   <none>          odd-filter.varadhi.example.com         80/TCP                                       4d19h
service/odd-filter-00001                 ClusterIP      10.96.84.239    <none>                                 80/TCP                                       4d19h
service/odd-filter-00001-private         ClusterIP      10.96.16.17     <none>                                 80/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   4d19h
service/odd-transformer                  ExternalName   <none>          odd-transformer.varadhi.example.com    80/TCP                                       4d19h
service/odd-transformer-00001            ClusterIP      10.96.61.11     <none>                                 80/TCP                                       4d19h
service/odd-transformer-00001-private    ClusterIP      10.96.203.185   <none>                                 80/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   4d19h
service/odd-transformer-00002            ClusterIP      10.96.115.147   <none>                                 80/TCP                                       4d19h
service/odd-transformer-00002-private    ClusterIP      10.96.235.117   <none>                                 80/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   4d19h

NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/even-filter-00001-deployment        0/0     0            0           4d19h
deployment.apps/even-transformer-00001-deployment   0/0     0            0           4d19h
deployment.apps/event-display-00001-deployment      0/0     0            0           4d19h
deployment.apps/odd-filter-00001-deployment         0/0     0            0           4d19h
deployment.apps/odd-transformer-00001-deployment    0/0     0            0           4d19h
deployment.apps/odd-transformer-00002-deployment    0/0     0            0           4d19h

NAME                                                           DESIRED   CURRENT   READY   AGE
replicaset.apps/even-filter-00001-deployment-6b7bdd866f        0         0         0       4d19h
replicaset.apps/even-transformer-00001-deployment-666bf9d776   0         0         0       4d19h
replicaset.apps/event-display-00001-deployment-758c9f7595      0         0         0       4d19h
replicaset.apps/odd-filter-00001-deployment-c86bd4799          0         0         0       4d19h
replicaset.apps/odd-transformer-00001-deployment-6bf46bc88f    0         0         0       4d19h
replicaset.apps/odd-transformer-00002-deployment-5c5f7b8b75    0         0         0       4d19h

NAME                                           URL   AGE     READY   REASON
parallel.flows.knative.dev/odd-even-parallel         4d19h

NAME                                  URL   AGE   READY   REASON

NAME                                                 LATESTCREATED            LATESTREADY              READY   REASON
configuration.serving.knative.dev/even-filter        even-filter-00001        even-filter-00001        True
configuration.serving.knative.dev/even-transformer   even-transformer-00001   even-transformer-00001   True
configuration.serving.knative.dev/event-display      event-display-00001      event-display-00001      True
configuration.serving.knative.dev/odd-filter         odd-filter-00001         odd-filter-00001         True
configuration.serving.knative.dev/odd-transformer    odd-transformer-00002    odd-transformer-00002    True

NAME                                         URL                                           READY     REASON
route.serving.knative.dev/even-filter        http://even-filter.varadhi.example.com        Unknown   IngressNotConfigured
route.serving.knative.dev/even-transformer   http://even-transformer.varadhi.example.com   Unknown   IngressNotConfigured
route.serving.knative.dev/event-display      http://event-display.varadhi.example.com      Unknown   IngressNotConfigured
route.serving.knative.dev/odd-filter         http://odd-filter.varadhi.example.com         Unknown   IngressNotConfigured
route.serving.knative.dev/odd-transformer    http://odd-transformer.varadhi.example.com    Unknown   IngressNotConfigured

NAME                                           URL                                           LATESTCREATED            LATESTREADY              READY     REASON
service.serving.knative.dev/even-filter        http://even-filter.varadhi.example.com        even-filter-00001        even-filter-00001        Unknown   IngressNotConfigured
service.serving.knative.dev/even-transformer   http://even-transformer.varadhi.example.com   even-transformer-00001   even-transformer-00001   Unknown   IngressNotConfigured
service.serving.knative.dev/event-display      http://event-display.varadhi.example.com      event-display-00001      event-display-00001      Unknown   IngressNotConfigured
service.serving.knative.dev/odd-filter         http://odd-filter.varadhi.example.com         odd-filter-00001         odd-filter-00001         Unknown   IngressNotConfigured
service.serving.knative.dev/odd-transformer    http://odd-transformer.varadhi.example.com    odd-transformer-00002    odd-transformer-00002    Unknown   IngressNotConfigured

NAME                                                  CONFIG NAME        K8S SERVICE NAME   GENERATION   READY   REASON             ACTUAL REPLICAS   DESIRED REPLICAS
revision.serving.knative.dev/even-filter-00001        even-filter                           1            True                       0                 0
revision.serving.knative.dev/even-transformer-00001   even-transformer                      1            True                       0                 0
revision.serving.knative.dev/event-display-00001      event-display                         1            True                       0                 0
revision.serving.knative.dev/odd-filter-00001         odd-filter                            1            True                       0                 0
revision.serving.knative.dev/odd-transformer-00001    odd-transformer                       1            False   ImagePullBackOff   0
revision.serving.knative.dev/odd-transformer-00002    odd-transformer                       2            True                       0                 0

NAME                                         SINK   SCHEDULE      AGE     READY   REASON
pingsource.sources.knative.dev/ping-source          */1 * * * *   4d19h   False   NotFound

Edit #2 I installed few more eventing crds (eventing.yaml)

Few changes that I could see now.

Channels are getting created and their status is true

NAME                                                                    URL                                                                           AGE   READY   REASON
inmemorychannel.messaging.knative.dev/odd-even-parallel-kn-parallel     http://odd-even-parallel-kn-parallel-kn-channel.varadhi.svc.cluster.local     73m   True
inmemorychannel.messaging.knative.dev/odd-even-parallel-kn-parallel-0   http://odd-even-parallel-kn-parallel-0-kn-channel.varadhi.svc.cluster.local   73m   True
inmemorychannel.messaging.knative.dev/odd-even-parallel-kn-parallel-1   http://odd-even-parallel-kn-parallel-1-kn-channel.varadhi.svc.cluster.local   73m   True
inmemorychannel.messaging.knative.dev/sequence-kn-sequence-0            http://sequence-kn-sequence-0-kn-channel.varadhi.svc.cluster.local            70m   True
inmemorychannel.messaging.knative.dev/sequence-kn-sequence-1            http://sequence-kn-sequence-1-kn-channel.varadhi.svc.cluster.local            50m   True
inmemorychannel.messaging.knative.dev/sequence-kn-sequence-2            http://sequence-kn-sequence-2-kn-channel.varadhi.svc.cluster.local            50m   True

but sequence and parallel are not in true state yet, they say subscriptionsNotReady

$ kubectl -n varadhi get sequence
NAME       URL                                                                  AGE   READY     REASON
sequence   http://sequence-kn-sequence-0-kn-channel.varadhi.svc.cluster.local   71m   Unknown   SubscriptionsNotReady
$ kubectl -n varadhi get parallel
NAME                URL                                                                         AGE     READY   REASON
odd-even-parallel   http://odd-even-parallel-kn-parallel-kn-channel.varadhi.svc.cluster.local   5d21h   False   SubscriptionsNotReady

On debugging further I could see below message in sequence

Ready:
      Last Transition Time:  2022-01-12T08:19:17Z
      Message:               Failed to get subscription status: subscription "sequence-kn-sequence-0" not present in channel "sequence-kn-sequence-0" subscriber's list
      Reason:                SubscriptionNotMarkedReadyByChannel
      Status:                Unknown
      Type:                  Ready
    Subscription:
      Name:       sequence-kn-sequence-0
      Namespace:  varadhi

and on looking into channel, it gives below message

   Last Transition Time:  2022-01-12T08:19:17Z
    Message:               The status of Dispatcher Deployment is False: MinimumReplicasUnavailable : Deployment does not have minimum availability.
    Reason:                DispatcherDeploymentFalse
    Severity:              Info
    Status:                False
    Type:                  DispatcherReady

Looks like channel is not able to fetch service deployment status

I also installed Kourier but I cannot see external ip. Note that I am running my own kind cluster for this.

$ kubectl --namespace kourier-system get service kourier
NAME      TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
kourier   LoadBalancer   10.96.112.238   <pending>     80:32002/TCP,443:32733/TCP   73m
$
$ kubectl get pods -n knative-serving
NAME                                      READY   STATUS    RESTARTS   AGE
activator-d4cd7dfd5-mcxsl                 1/1     Running   0          7d18h
autoscaler-69689d8b7-rx75h                1/1     Running   0          7d18h
controller-766f74d9f8-fwdk9               1/1     Running   0          7d18h
domain-mapping-7dbbb5c7d-xk5m5            1/1     Running   0          7d18h
domainmapping-webhook-747f79dbdc-qm5nn    1/1     Running   0          7d18h
net-kourier-controller-5657664b99-zr9cj   1/1     Running   0          73m
webhook-8f6866966-8z8tt                   1/1     Running   0          7d18h

Few changes in services post installing kourier

$ kubectl get service.serving.knative.dev -n varadhi
NAME               URL                                           LATESTCREATED            LATESTREADY              READY     REASON
even-filter        http://even-filter.varadhi.example.com        even-filter-00001        even-filter-00001        Unknown
even-transformer   http://even-transformer.varadhi.example.com   even-transformer-00001   even-transformer-00001   Unknown
event-display      http://event-display.varadhi.example.com      event-display-00001      event-display-00001      Unknown
first              http://first.varadhi.example.com              first-00001              first-00001              Unknown
odd-filter         http://odd-filter.varadhi.example.com         odd-filter-00001         odd-filter-00001         Unknown
odd-transformer    http://odd-transformer.varadhi.example.com    odd-transformer-00002    odd-transformer-00002    Unknown
second             http://second.varadhi.example.com             second-00001             second-00001             Unknown
third              http://third.varadhi.example.com              third-00001              third-00001              Unknown

IngressNotConfigured status goes away.

score 0 · Answer 1 · answered Jan 06 '22 at 13:16

for the Sequence example I think there might be something wrong with your default channel configuration.

Sequences create channels to communicate between every step, it is expected that three channels are created for that example:

$ kubectl get channel

inmemorychannel.messaging.knative.dev/sequence-kn-sequence-0           http://sequence-kn-sequence-0-kn-channel.default.svc.cluster.local           56s   True    
inmemorychannel.messaging.knative.dev/sequence-kn-sequence-1           http://sequence-kn-sequence-1-kn-channel.default.svc.cluster.local           56s   True    
inmemorychannel.messaging.knative.dev/sequence-kn-sequence-2           http://sequence-kn-sequence-2-kn-channel.default.svc.cluster.local           56s   True

Can you check what channels are created for you and their status?

Can you also make sure your default channel is properly setup? See: https://knative.dev/docs/eventing/channels/channel-types-defaults/

Can you also post here the Parallel you are using? Status is up to the controller to fill, that one sounds like an issue.

Thanks for your response, I have updated question with channel info and parallel example that I am trying out. — Anil Gowda, Jan 11 '22 at 07:29

score 0 · Answer 2 · answered Jan 11 '22 at 16:31

At both of your outputs for sequence and parallel an issue with services can be found:

service.serving.knative.dev/even-filter        http://even-filter.varadhi.example.com        even-filter-00001        even-filter-00001        Unknown   IngressNotConfigured

Although Knative Eventing does not depend on Serving, the examples for sequence/parallel requires it to be properly installed because they use serverless services:

sequence example: event-display, first, second and third
parallel example: event-display, even-filter, even-transformer, odd-filter even-transformer.

Eventing can use regular kubernetes services instead of knative services, but I think the best way to make the examples work for you is making sure Knative Serving works as expected.

Did you configure a network provider for Knative Serving? If you did not, can you go through this step:

https://knative.dev/docs/install/serving/install-serving-with-yaml/#install-a-networking-layer

If you are in doubt about which one to choose I would go for Kourier, which is maintained by the Knative project.

Thanks for your response, I have updated question with new status after installing few more crds and kourier. — Anil Gowda, Jan 12 '22 at 09:32

Knative Parallel and Sequence demo fails

2 Answers2