revision: events: service: fails to reconcile predictor: fails to update knative service: Operation cannot be fulfilled on services.serving.knative.dev "paddle-predictor-default": the object has been modified; please apply your changes to the latest version and try again replicaset: ['Created pod: paddle-predictor-default-00001-deployment-5dcbffdf99-jmt6c'] paddle-predictor-default-00001-deployment-5dcbffdf99-jmt6c: status: {'running': None, 'terminated': None, 'waiting': None} {'running': None, 'terminated': None, 'waiting': {'message': None, 'reason': 'PodInitializing'}} ['Successfully assigned paddle/paddle-predictor-default-00001-deployment-5dcbffdf99-jmt6c to node16'] logs: ['failed to update deployment "paddle-predictor-default-00001-deployment": Operation cannot be fulfilled on deployments.apps "paddle-predictor-default-00001-deployment": the object has been modified; please apply your changes to the latest version and try again'] {'running': {'started_at': datetime.datetime(2025, 9, 25, 6, 36, 28, tzinfo=tzutc())}, 'terminated': None, 'waiting': None} {'running': None, 'terminated': {'container_id': 'docker://601234ba689e3d280395059710dd6144f6199adbea51e9df85933d2a37dc3f5f', 'exit_code': 0, 'finished_at': datetime.datetime(2025, 9, 25, 6, 36, 29, tzinfo=tzutc()), 'message': None, 'reason': 'Completed', 'signal': None, 'started_at': datetime.datetime(2025, 9, 25, 6, 36, 29, tzinfo=tzutc())}, 'waiting': None} {'running': None, 'terminated': {'container_id': 'docker://6bf50395bc699e9a74e77a482ab56c31d391bd74af23a23efdab29ee1296b47e', 'exit_code': 0, 'finished_at': datetime.datetime(2025, 9, 25, 6, 36, 45, tzinfo=tzutc()), 'message': None, 'reason': 'Completed', 'signal': None, 'started_at': datetime.datetime(2025, 9, 25, 6, 36, 45, tzinfo=tzutc())}, 'waiting': None} {'running': None, 'terminated': {'container_id': 'docker://7908cdc32efc1a2f1b6a07b85ac94d38ed50a7953e5be4ebe3af426257a447be', 'exit_code': 0, 'finished_at': datetime.datetime(2025, 9, 25, 6, 36, 28, tzinfo=tzutc()), 'message': None, 'reason': 'Completed', 'signal': None, 'started_at': datetime.datetime(2025, 9, 25, 6, 36, 27, tzinfo=tzutc())}, 'waiting': None} ['Successfully assigned paddle/paddle-predictor-default-00001-deployment-5dcbffdf99-jmt6c to node16', 'Add eth0 [172.20.35.84/32] from cni0', 'Container image "10.224.105.5:5000/inais/proxyv2:1.12.0" already present on machine', 'Created container istio-init', 'Started container istio-init', 'Container image "10.224.105.5:5000/mine/paddle/vscode-paddle:V1-19" already present on machine', 'Created container kserve-container', 'Started container kserve-container', 'Container image "10.224.105.5:5000/inais/knative-queue:v1.1" already present on machine', 'Created container queue-proxy', 'Started container queue-proxy', 'Container image "10.224.105.5:5000/inais/kserve-agent:graph" already present on machine', 'Created container agent', 'Started container agent', 'Container image "10.224.105.5:5000/inais/proxyv2:1.12.0" already present on machine', 'Created container istio-proxy', 'Started container istio-proxy', 'Back-off restarting failed container', 'Readiness probe failed: Get "http://172.20.35.84:15020/app-health/queue-proxy/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)', 'Readiness probe failed: HTTP probe failed with statuscode: 500'] {"severity":"INFO","timestamp":"2025-09-25T14:36:29.433151719+08:00","caller":"logging/config.go:80","message":"Unable to read vcs.revision from binary"} {"severity":"INFO","timestamp":"2025-09-25T14:36:29.631058896+08:00","caller":"agent/main.go:122","message":"Starting logger"} {"severity":"INFO","timestamp":"2025-09-25T14:36:29.631086366+08:00","caller":"agent/main.go:247","message":"Starting the log dispatcher"} {"severity":"INFO","timestamp":"2025-09-25T14:36:29.631097026+08:00","caller":"logger/dispatcher.go:31","message":"Starting worker 1"} {"severity":"INFO","timestamp":"2025-09-25T14:36:29.631124346+08:00","caller":"logger/dispatcher.go:31","message":"Starting worker 2"} {"severity":"INFO","timestamp":"2025-09-25T14:36:29.631137025+08:00","caller":"logger/dispatcher.go:31","message":"Starting worker 3"} {"severity":"INFO","timestamp":"2025-09-25T14:36:29.631145595+08:00","caller":"logger/dispatcher.go:31","message":"Starting worker 4"} {"severity":"INFO","timestamp":"2025-09-25T14:36:29.631154625+08:00","caller":"logger/dispatcher.go:31","message":"Starting worker 5"} {"severity":"INFO","timestamp":"2025-09-25T14:36:29.631161755+08:00","caller":"agent/main.go:131","message":"Starting agent http server..."} {"severity":"INFO","timestamp":"2025-09-25T14:36:29.730330293+08:00","caller":"agent/main.go:288","message":"Building server user port 8080 port 9081"} aggressive probe error (failed 202 times): dial tcp 127.0.0.1:8080: connect: connection refused timed out waiting for the condition

2025-09-25T06:36:28.916768Z info FLAG: --concurrency="2" 2025-09-25T06:36:28.916839Z info FLAG: --domain="paddle.svc.cluster.local" 2025-09-25T06:36:28.916845Z info FLAG: --help="false" 2025-09-25T06:36:28.916847Z info FLAG: --log_as_json="false" 2025-09-25T06:36:28.916851Z info FLAG: --log_caller="" 2025-09-25T06:36:28.916853Z info FLAG: --log_output_level="default:info" 2025-09-25T06:36:28.916855Z info FLAG: --log_rotate="" 2025-09-25T06:36:28.916858Z info FLAG: --log_rotate_max_age="30" 2025-09-25T06:36:28.916861Z info FLAG: --log_rotate_max_backups="1000" 2025-09-25T06:36:28.916865Z info FLAG: --log_rotate_max_size="104857600" 2025-09-25T06:36:28.916869Z info FLAG: --log_stacktrace_level="default:none" 2025-09-25T06:36:28.916877Z info FLAG: --log_target="[stdout]" 2025-09-25T06:36:28.916882Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2025-09-25T06:36:28.916885Z info FLAG: --outlierLogPath="" 2025-09-25T06:36:28.916890Z info FLAG: --proxyComponentLogLevel="misc:error" 2025-09-25T06:36:28.916893Z info FLAG: --proxyLogLevel="warning" 2025-09-25T06:36:28.916896Z info FLAG: --serviceCluster="istio-proxy" 2025-09-25T06:36:28.916900Z info FLAG: --stsPort="0" 2025-09-25T06:36:28.916903Z info FLAG: --templateFile="" 2025-09-25T06:36:28.916906Z info FLAG: --tokenManagerPlugin="GoogleTokenExchange" 2025-09-25T06:36:28.916910Z info FLAG: --vklog="0" 2025-09-25T06:36:28.916916Z info Version 1.12.0-016bc46f4a5e0ef3fa135b3c5380ab7765467c1a-Clean 2025-09-25T06:36:28.917386Z info Proxy role ips=[172.20.35.84] type=sidecar id=paddle-predictor-default-00001-deployment-5dcbffdf99-jmt6c.paddle domain=paddle.svc.cluster.local 2025-09-25T06:36:28.917479Z info Apply proxy config from env {"terminationDrainDuration":"20s"} 2025-09-25T06:36:28.918560Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod.istio-system.svc:15012 drainDuration: 45s parentShutdownDuration: 60s proxyAdminPort: 15000 serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 20s tracing: zipkin: address: zipkin.istio-system:9411 2025-09-25T06:36:28.918572Z info JWT policy is third-party-jwt 2025-09-25T06:36:28.970789Z info CA Endpoint istiod.istio-system.svc:15012, provider Citadel 2025-09-25T06:36:28.970817Z info Opening status port 15020 2025-09-25T06:36:28.970899Z info Using CA istiod.istio-system.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2025-09-25T06:36:28.970968Z info citadelclient Citadel client using custom root cert: istiod.istio-system.svc:15012 2025-09-25T06:36:28.987637Z info ads All caches have been synced up in 74.685848ms, marking server ready 2025-09-25T06:36:28.988074Z info sds SDS server for workload certificates started, listening on "etc/istio/proxy/SDS" 2025-09-25T06:36:28.988102Z info xdsproxy Initializing with upstream address "istiod.istio-system.svc:15012" and cluster "Kubernetes" 2025-09-25T06:36:28.988136Z info sds Starting SDS grpc server 2025-09-25T06:36:28.988487Z info Pilot SAN: [istiod.istio-system.svc] 2025-09-25T06:36:28.988561Z info starting Http service at 127.0.0.1:15004 2025-09-25T06:36:28.990212Z info Pilot SAN: [istiod.istio-system.svc] 2025-09-25T06:36:28.992298Z info Starting proxy agent 2025-09-25T06:36:28.992327Z info Epoch 0 starting 2025-09-25T06:36:28.992354Z info Envoy command: [-c etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --drain-strategy immediate --parent-shutdown-time-s 60 --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --log-format %Y-%m-%dT%T.%fZ %l envoy %n %v -l warning --component-log-level misc:error --concurrency 2] 2025-09-25T06:36:29.061031Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012 2025-09-25T06:36:29.191402Z info ads ADS: new connection for node:paddle-predictor-default-00001-deployment-5dcbffdf99-jmt6c.paddle-1 2025-09-25T06:36:29.201179Z info ads ADS: new connection for node:paddle-predictor-default-00001-deployment-5dcbffdf99-jmt6c.paddle-2 2025-09-25T06:36:29.285650Z info cache generated new workload certificate latency=297.481189ms ttl=23h59m59.714396654s 2025-09-25T06:36:29.285748Z info cache Root cert has changed, start rotating root cert 2025-09-25T06:36:29.285773Z info ads XDS: Incremental Pushing:0 ConnectedEndpoints:2 Version: 2025-09-25T06:36:29.285816Z info cache returned workload trust anchor from cache ttl=23h59m59.714186756s 2025-09-25T06:36:29.285829Z info cache returned workload certificate from cache ttl=23h59m59.714173336s 2025-09-25T06:36:29.285970Z info cache returned workload trust anchor from cache ttl=23h59m59.714059197s

2025-09-25T06:36:29.286228Z info ads SDS: PUSH request for node:paddle-predictor-default-00001-deployment-5dcbffdf99-jmt6c.paddle resources:1 size:4.0kB resource:default 2025-09-25T06:36:29.286424Z info ads SDS: PUSH request for node:paddle-predictor-default-00001-deployment-5dcbffdf99-jmt6c.paddle resources:1 size:1.1kB resource:ROOTCA 2025-09-25T06:36:29.286503Z info cache returned workload trust anchor from cache ttl=23h59m59.713500643s 2025-09-25T06:36:29.286539Z info ads SDS: PUSH for node:paddle-predictor-default-00001-deployment-5dcbffdf99-jmt6c.paddle resources:1 size:1.1kB resource:ROOTCA 2025-09-25T06:36:30.172553Z info Readiness succeeded in 1.258243292s 2025-09-25T06:36:30.173201Z info Envoy proxy is ready 2025-09-25T06:36:31.431028Z error Request to probe app failed: Get "http://172.20.35.84:8012/": context deadline exceeded (Client.Timeout exceeded while awaiting headers), original URL path = /app-health/queue-proxy/readyz app URL path = / 2025-09-25T06:36:32.432019Z error Request to probe app failed: Get "http://172.20.35.84:8012/": context deadline exceeded (Client.Timeout exceeded while awaiting headers), original URL path = /app-health/queue-proxy/readyz 2025-09-25T06:36:33.432376Z error Request to probe app failed: Get "http://172.20.35.84:8012/": context deadline exceeded (Client.Timeout exceeded while awaiting headers), original URL path = /app-health/queue-proxy/readyz 2025-09-25T06:36:34.432799Z error Request to probe app failed: Get "http://172.20.35.84:8012/": context deadline exceeded (Client.Timeout exceeded while awaiting headers), original URL path = /app-health/queue-proxy/readyz