Describe the bug
argocd-image-updater is running, sees valid candidate images from our docker.io repo, and updates them in the k8s Application, but the k8s deployment does not update it's image. Deployment image does have a kustomization.yaml. This is probably just a bone headed configuration issue in argocd-image-updater or in argocd, but I can't find any examples of a full argocd-image-update kustomization implementation. (I've posted this in their issues and did some digging, but the project has been pretty quiet). Even just some examples of a working kustomize example may do the trick here; I'm mostly unsure how the Application image should update the kustomize image and thus the kubernetes deployment image.
Updater logs:
time="2023-03-31T17:52:45Z" level=info msg="Starting image update cycle, considering 1 annotated application(s) for update"
time="2023-03-31T17:52:45Z" level=debug msg="Processing application doc-storage-ui"
time="2023-03-31T17:52:45Z" level=debug msg="Considering this image for update" alias=doc-storage-ui application=doc-storage-ui image_name=cblairupliv/doc-storage-ui image_tag= registry=
time="2023-03-31T17:52:45Z" level=debug msg="Using no version constraint when looking for a new tag" alias=doc-storage-ui application=doc-storage-ui image_name=cblairupliv/doc-storage-ui image_tag= registry=
time="2023-03-31T17:52:45Z" level=debug msg="could not parse input tag latest as semver: Invalid Semantic Version"
time="2023-03-31T17:52:45Z" level=debug msg="could not parse input tag pr-latest as semver: Invalid Semantic Version"
time="2023-03-31T17:52:45Z" level=debug msg="could not parse input tag staging as semver: Invalid Semantic Version"
time="2023-03-31T17:52:45Z" level=debug msg="found 5 from 5 tags eligible for consideration" image=cblairupliv/doc-storage-ui
time="2023-03-31T17:52:45Z" level=info msg="Setting new image to cblairupliv/doc-storage-ui:0.1.45" alias=doc-storage-ui application=doc-storage-ui image_name=cblairupliv/doc-storage-ui image_tag= registry=
time="2023-03-31T17:52:45Z" level=info msg="Successfully updated image 'cblairupliv/doc-storage-ui' to 'cblairupliv/doc-storage-ui:0.1.45', but pending spec update (dry run=false)" alias=doc-storage-ui application=doc-storage-ui image_name=cblairupliv/doc-storage-ui image_tag= registry=
time="2023-03-31T17:52:45Z" level=debug msg="Using commit message: "
time="2023-03-31T17:52:45Z" level=info msg="Committing 1 parameter update(s) for application doc-storage-ui" application=doc-storage-ui
time="2023-03-31T17:52:45Z" level=info msg="Successfully updated the live application spec" application=doc-storage-ui
time="2023-03-31T17:52:45Z" level=info msg="Processing results: applications=1 images_considered=1 images_skipped=0 images_updated=1 errors=0"
Application updated (notice that Images is the correct latest semver of cblairupliv/doc-storage-ui:0.1.45; sorry so verbose but may shed some light):
kc describe app -nargocd doc-storage-ui
Name: doc-storage-ui
Namespace: argocd
Labels: <none>
Annotations: argocd-image-updater.argoproj.io/argocd.kustomize.image-name: index.docker.io/cblairupliv/doc-storage-ui
argocd-image-updater.argoproj.io/doc-storage-ui.update-strategy: semver
argocd-image-updater.argoproj.io/image-list: doc-storage-ui=index.docker.io/cblairupliv/doc-storage-ui
argocd-image-updater.argoproj.io/write-back-target: kustomization
API Version: argoproj.io/v1alpha1
Kind: Application
Metadata:
Creation Timestamp: 2023-03-31T17:34:54Z
Generation: 1422
Managed Fields:
API Version: argoproj.io/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:argocd-image-updater.argoproj.io/argocd.kustomize.image-name:
f:argocd-image-updater.argoproj.io/doc-storage-ui.update-strategy:
f:argocd-image-updater.argoproj.io/image-list:
f:argocd-image-updater.argoproj.io/write-back-target:
f:spec:
.:
f:destination:
.:
f:namespace:
f:server:
f:project:
f:source:
.:
f:path:
f:repoURL:
f:targetRevision:
f:syncPolicy:
.:
f:automated:
.:
f:prune:
f:selfHeal:
f:syncOptions:
f:status:
.:
f:health:
f:summary:
f:sync:
.:
f:comparedTo:
.:
f:destination:
f:source:
Manager: argocd-server
Operation: Update
Time: 2023-03-31T17:34:54Z
API Version: argoproj.io/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:spec:
f:source:
f:kustomize:
.:
f:images:
Manager: argocd-image-updater
Operation: Update
Time: 2023-03-31T17:36:42Z
API Version: argoproj.io/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:status:
f:health:
f:status:
f:history:
f:operationState:
.:
f:finishedAt:
f:message:
f:operation:
.:
f:initiatedBy:
.:
f:automated:
f:retry:
.:
f:limit:
f:sync:
.:
f:prune:
f:revision:
f:syncOptions:
f:phase:
f:startedAt:
f:syncResult:
.:
f:resources:
f:revision:
f:source:
.:
f:path:
f:repoURL:
f:targetRevision:
f:reconciledAt:
f:resources:
f:sourceType:
f:summary:
f:images:
f:sync:
f:comparedTo:
f:destination:
f:namespace:
f:server:
f:source:
f:kustomize:
.:
f:images:
f:path:
f:repoURL:
f:targetRevision:
f:revision:
f:status:
Manager: argocd-application-controller
Operation: Update
Time: 2023-03-31T17:36:43Z
Resource Version: 6907890
UID: 830639e6-0df5-4135-9713-5ba57453a892
Spec:
Destination:
Namespace: default
Server: https://kubernetes.default.svc
Project: default
Source:
Kustomize:
Images:
cblairupliv/doc-storage-ui:0.1.45
Path: argocd/prod/doc-storage-ui/k8s
Repo URL: https://github.com/upliv-health/iac
Target Revision: main
Sync Policy:
Automated:
Prune: true
Self Heal: true
Sync Options:
CreateNamespace=true
Status:
Health:
Status: Healthy
History:
Deploy Started At: 2023-03-31T17:34:54Z
Deployed At: 2023-03-31T17:34:54Z
Id: 0
Revision: e580c3e3b944f17c590a498bd27b7e35be088f86
Source:
Path: argocd/prod/doc-storage-ui/k8s
Repo URL: https://github.com/upliv-health/iac
Target Revision: main
Operation State:
Finished At: 2023-03-31T17:34:54Z
Message: successfully synced (all tasks run)
Operation:
Initiated By:
Automated: true
Retry:
Limit: 5
Sync:
Prune: true
Revision: e580c3e3b944f17c590a498bd27b7e35be088f86
Sync Options:
CreateNamespace=true
Phase: Succeeded
Started At: 2023-03-31T17:34:54Z
Sync Result:
Resources:
Group:
Hook Phase: Running
Kind: Service
Message: service/doc-storage-ui created
Name: doc-storage-ui
Namespace: default
Status: Synced
Sync Phase: Sync
Version: v1
Group:
Hook Phase: Running
Kind: Service
Message: service/doc-storage-ui-service-cluster-ip created
Name: doc-storage-ui-service-cluster-ip
Namespace: default
Status: Synced
Sync Phase: Sync
Version: v1
Group:
Hook Phase: Running
Kind: Service
Message: service/doc-storage-ui-service-loadbalancer created
Name: doc-storage-ui-service-loadbalancer
Namespace: default
Status: Synced
Sync Phase: Sync
Version: v1
Group: apps
Hook Phase: Running
Kind: Deployment
Message: deployment.apps/doc-storage-ui created
Name: doc-storage-ui
Namespace: default
Status: Synced
Sync Phase: Sync
Version: v1
Revision: e580c3e3b944f17c590a498bd27b7e35be088f86
Source:
Path: argocd/prod/doc-storage-ui/k8s
Repo URL: https://github.com/upliv-health/iac
Target Revision: main
Reconciled At: 2023-04-03T16:02:40Z
Resources:
Health:
Status: Healthy
Kind: Service
Name: doc-storage-ui
Namespace: default
Status: Synced
Version: v1
Health:
Status: Healthy
Kind: Service
Name: doc-storage-ui-service-cluster-ip
Namespace: default
Status: Synced
Version: v1
Health:
Status: Healthy
Kind: Service
Name: doc-storage-ui-service-loadbalancer
Namespace: default
Status: Synced
Version: v1
Group: apps
Health:
Status: Healthy
Kind: Deployment
Name: doc-storage-ui
Namespace: default
Status: Synced
Version: v1
Source Type: Kustomize
Summary:
Images:
index.docker.io/cblairupliv/doc-storage-ui
Sync:
Compared To:
Destination:
Namespace: default
Server: https://kubernetes.default.svc
Source:
Kustomize:
Images:
cblairupliv/doc-storage-ui:0.1.45
Path: argocd/prod/doc-storage-ui/k8s
Repo URL: https://github.com/upliv-health/iac
Target Revision: main
Revision: e580c3e3b944f17c590a498bd27b7e35be088f86
Status: Synced
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ImagesUpdated 59m ArgocdImageUpdater Successfully updated application 'doc-storage-ui'
Normal ImagesUpdated 57m ArgocdImageUpdater Successfully updated application 'doc-storage-ui'
Normal ImagesUpdated 55m ArgocdImageUpdater Successfully updated application 'doc-storage-ui'
Normal ImagesUpdated 53m ArgocdImageUpdater Successfully updated application 'doc-storage-ui'
Normal ImagesUpdated 51m ArgocdImageUpdater Successfully updated application 'doc-storage-ui'
Normal ImagesUpdated 48m ArgocdImageUpdater Successfully updated application 'doc-storage-ui'
Normal ImagesUpdated 46m ArgocdImageUpdater Successfully updated application 'doc-storage-ui'
Normal ImagesUpdated 44m ArgocdImageUpdater Successfully updated application 'doc-storage-ui'
Normal ImagesUpdated 42m ArgocdImageUpdater Successfully updated application 'doc-storage-ui'
Normal ImagesUpdated 40m ArgocdImageUpdater Successfully updated application 'doc-storage-ui'
Normal ImagesUpdated 38m ArgocdImageUpdater Successfully updated application 'doc-storage-ui'
Normal ImagesUpdated 36m ArgocdImageUpdater Successfully updated application 'doc-storage-ui'
Normal ImagesUpdated 34m ArgocdImageUpdater Successfully updated application 'doc-storage-ui'
Normal ImagesUpdated 32m ArgocdImageUpdater Successfully updated application 'doc-storage-ui'
Normal ImagesUpdated 30m ArgocdImageUpdater Successfully updated application 'doc-storage-ui'
Normal ImagesUpdated 28m ArgocdImageUpdater Successfully updated application 'doc-storage-ui'
Normal ImagesUpdated 26m ArgocdImageUpdater Successfully updated application 'doc-storage-ui'
Normal ImagesUpdated 24m ArgocdImageUpdater Successfully updated application 'doc-storage-ui'
Normal ImagesUpdated 22m ArgocdImageUpdater Successfully updated application 'doc-storage-ui'
Normal ImagesUpdated 20m ArgocdImageUpdater Successfully updated application 'doc-storage-ui'
Normal ImagesUpdated 18m ArgocdImageUpdater Successfully updated application 'doc-storage-ui'
Normal ImagesUpdated 16m ArgocdImageUpdater Successfully updated application 'doc-storage-ui'
Normal ImagesUpdated 14m ArgocdImageUpdater Successfully updated application 'doc-storage-ui'
Normal ImagesUpdated 12m ArgocdImageUpdater Successfully updated application 'doc-storage-ui'
Normal ImagesUpdated 10m ArgocdImageUpdater Successfully updated application 'doc-storage-ui'
Normal ImagesUpdated 8m53s ArgocdImageUpdater Successfully updated application 'doc-storage-ui'
Normal ImagesUpdated 6m52s ArgocdImageUpdater Successfully updated application 'doc-storage-ui'
Normal ImagesUpdated 4m52s ArgocdImageUpdater Successfully updated application 'doc-storage-ui'
Normal ImagesUpdated 2m52s ArgocdImageUpdater Successfully updated application 'doc-storage-ui'
Normal ImagesUpdated 51s ArgocdImageUpdater Successfully updated application 'doc-storage-ui'
Deployment not updated in k8s:
kc describe deployment doc-storage-ui
Name: doc-storage-ui
Namespace: default
CreationTimestamp: Fri, 31 Mar 2023 10:34:54 -0700
Labels: app=doc-storage-ui
app.kubernetes.io/instance=doc-storage-ui
tier=web
Annotations: deployment.kubernetes.io/revision: 1
Selector: app=doc-storage-ui,tier=web
Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app=doc-storage-ui
tier=web
Containers:
doc-storage-ui:
Image: index.docker.io/cblairupliv/doc-storage-ui
Port: 3000/TCP
Host Port: 0/TCP
Command:
node
server.js
Environment: <none>
Mounts: <none>
Volumes: <none>
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True NewReplicaSetAvailable
OldReplicaSets: <none>
NewReplicaSet: doc-storage-ui-68c575f855 (1/1 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 35m deployment-controller Scaled up replica set doc-storage-ui-68c575f855 to 1
Deploymement yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: doc-storage-ui
labels:
app: doc-storage-ui
tier: web
spec:
replicas: 1
selector:
matchLabels:
app: doc-storage-ui
tier: web
template:
metadata:
labels:
app: doc-storage-ui
tier: web
spec:
# Credential for private docker.com repository.
imagePullSecrets:
- name: regcred
containers:
- name: doc-storage-ui
image: index.docker.io/cblairupliv/doc-storage-ui
imagePullPolicy: Always
ports:
- containerPort: 3000
command: ["node", "server.js"]
And the kustomize.yaml in the same directory as the deployment.yaml:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- cluster-ip.yaml
- deployment.yaml
- loadbalancer.yaml
- service.yaml
images:
- name: doc-storage-ui
newName: index.docker.io/cblairupliv/doc-storage-ui
newTag: latest
To Reproduce Steps to reproduce the behavior:
- configure argocd similar to the above in yaml ^
- watch the argocd-image-update pod k8s logs and verify image found and updated
- Verify the Application image is updated as described above
- Verify the deployment image is not updated
Expected behavior The image version to update in the deployment
Additional context Newest installs for argocd and argocd-image-updater as of today.
Version See above
Logs Lots of logs above.
