Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.44.0-wmf.2 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 0 4108 2242420 2242347 2024-11-07T12:55:10Z ScheduleDeploymentBot 37566 Add [[gerrit:1087927]] to Thursday, November 07 UTC afternoon backport window 2242420 wikitext text/x-wiki 6vplrmjb8ehrh92n2ouwkg8vc0wp5si 498 6726 2242433 2242120 2024-11-07T13:43:29Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api 2242433 wikitext text/x-wiki qnljzxkh6w99ylhu20tskzq06prc848 2242435 2242433 2024-11-07T13:50:16Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api 2242435 wikitext text/x-wiki jbgsea1gwh9wj97ub5isjncqqlurhpz 2242452 2242435 2024-11-07T14:26:18Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api 2242452 wikitext text/x-wiki itgfstfqli332s5asglr44a4dm6wue3 2242454 2242452 2024-11-07T14:30:35Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api 2242454 wikitext text/x-wiki kajtz07bogi5r4i39kk3poi24cvnw4i 2242455 2242454 2024-11-07T14:30:43Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway 2242455 wikitext text/x-wiki hr5cvc79oo7oluwx2unxpoic7zusyyy 2242456 2242455 2024-11-07T14:36:04Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway 2242456 wikitext text/x-wiki 9b1ccz3gmp249uqnihj7m0yuc5hcquj 2242477 2242456 2024-11-07T15:51:14Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api 2242477 wikitext text/x-wiki sgt3gx8xtl58f35u5rmy9oy83wda3hb 2242482 2242477 2024-11-07T15:56:39Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api 2242482 wikitext text/x-wiki cbgulnivg7htg5xbb3ys89l9vr08cp8 0 7919 2242413 2242411 2024-11-07T12:16:00Z Stashbot 7414 vgutierrez: repool liberica on lvs1013 2242413 wikitext text/x-wiki aee5wndd5a33nhkb0xemdlnmhe51vm6 2242414 2242413 2024-11-07T12:36:13Z Stashbot 7414 joal@deploy2002: Started deploy [analytics/refinery@4bec064]: Regular analytics weekly train [analytics/refinery@4bec0640] 2242414 wikitext text/x-wiki ilmw8xv9zmf7tbo1mozzki54rsi7nvd 2242415 2242414 2024-11-07T12:37:40Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1047 2242415 wikitext text/x-wiki odvf9kvhh13ylz72p98jsnrf253k4ke 2242416 2242415 2024-11-07T12:39:19Z Stashbot 7414 jmm@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host ganeti1047 2242416 wikitext text/x-wiki rmj8bg2zm6lj8y0938xhrcv5ff0ndln 2242417 2242416 2024-11-07T12:40:19Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1047 2242417 wikitext text/x-wiki t856etveefwaba99h4z8frr29vshf2w 2242418 2242417 2024-11-07T12:40:36Z Stashbot 7414 jmm@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host ganeti1047 2242418 wikitext text/x-wiki 7qvu2ro4235uddbri76e7zn0zhxz4bk 2242419 2242418 2024-11-07T12:53:00Z Stashbot 7414 joal@deploy2002: Finished deploy [analytics/refinery@4bec064]: Regular analytics weekly train [analytics/refinery@4bec0640] (duration: 16m 47s) 2242419 wikitext text/x-wiki 9ocs2myn9214618n23yafkma4kkzmtb 2242421 2242419 2024-11-07T13:08:37Z Stashbot 7414 joal@deploy2002: Started deploy [analytics/refinery@4bec064] (thin): Regular analytics weekly train THIN [analytics/refinery@4bec0640] 2242421 wikitext text/x-wiki i28wnf0xhjuau048brrxtyoewackqv1 2242422 2242421 2024-11-07T13:13:40Z Stashbot 7414 joal@deploy2002: Finished deploy [analytics/refinery@4bec064] (thin): Regular analytics weekly train THIN [analytics/refinery@4bec0640] (duration: 05m 03s) 2242422 wikitext text/x-wiki ats002vz6cb9uhchcqwe9zp6jx5eeom 2242424 2242422 2024-11-07T13:20:03Z Stashbot 7414 joal@deploy2002: Started deploy [analytics/refinery@4bec064] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@4bec0640] 2242424 wikitext text/x-wiki 9uu5q4vm6d5017if4znbw63ibths5ja 2242427 2242424 2024-11-07T13:23:48Z Stashbot 7414 joal@deploy2002: Finished deploy [analytics/refinery@4bec064] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@4bec0640] (duration: 03m 44s) 2242427 wikitext text/x-wiki j9yxq96k9hhibpye3geoqqvsm9lnkvt 2242429 2242427 2024-11-07T13:34:50Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1047 2242429 wikitext text/x-wiki gks9lown43k6kxb8wrvpvuphz7k4ltg 2242430 2242429 2024-11-07T13:36:00Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1047 2242430 wikitext text/x-wiki qmgw0iwj5exf1ijopnc5b5ntcqj4god 2242431 2242430 2024-11-07T13:36:51Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1048 2242431 wikitext text/x-wiki bh6s68wqk7g7wvyyysi7wv6e9koa0w5 2242432 2242431 2024-11-07T13:37:55Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1048 2242432 wikitext text/x-wiki ezz7873m4ualrfpgqquzl40qtcp90et 2242436 2242432 2024-11-07T13:52:21Z Stashbot 7414 cwhite: running thanos bucket cleanup on titan1001 - T351927 2242436 wikitext text/x-wiki dvv13cykgd22x5ozyp4obui1rjvgiig 2242440 2242436 2024-11-07T14:03:09Z Stashbot 7414 joal@deploy2002: Started deploy [airflow-dags/analytics@23bc4ad]: Regular analytics weekly train [airflow-dags/analytics@23bc4ad3] 2242440 wikitext text/x-wiki ng82fqk0ypxh0mb4r3li41kaupf8wrg 2242441 2242440 2024-11-07T14:03:49Z Stashbot 7414 kartik@deploy2002: Started scap sync-world: Backport for [[gerrit:1088215|Enable Section Translation in ann, iba, nr and, tdd Wikipedias (T371420)]] 2242441 wikitext text/x-wiki 54j6q06gc80ef0sutzz76tc6x2pjgwt 2242442 2242441 2024-11-07T14:04:53Z Stashbot 7414 joal@deploy2002: Finished deploy [airflow-dags/analytics@23bc4ad]: Regular analytics weekly train [airflow-dags/analytics@23bc4ad3] (duration: 01m 44s) 2242442 wikitext text/x-wiki inmr9z5v61h41ygolf97qhyuq6ihure 2242443 2242442 2024-11-07T14:06:27Z Stashbot 7414 kartik@deploy2002: kartik: Backport for [[gerrit:1088215|Enable Section Translation in ann, iba, nr and, tdd Wikipedias (T371420)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) 2242443 wikitext text/x-wiki ln2qxa6w3tgks6orjet5fui3ncl20zm 2242444 2242443 2024-11-07T14:09:21Z Stashbot 7414 kartik@deploy2002: kartik: Continuing with sync 2242444 wikitext text/x-wiki ib8hnd3vhy8b4rq2aiaumgfp6d9mx4o 2242445 2242444 2024-11-07T14:13:58Z Stashbot 7414 kartik@deploy2002: Finished scap sync-world: Backport for [[gerrit:1088215|Enable Section Translation in ann, iba, nr and, tdd Wikipedias (T371420)]] (duration: 10m 08s) 2242445 wikitext text/x-wiki 3hsgzxey7ydcbgnbd3i5mclaot7oit3 2242446 2242445 2024-11-07T14:15:44Z Stashbot 7414 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for [[gerrit:1087927|Deploy EditCheck (references) to hiwiki, bnwiki, idwiki (T366381)]] 2242446 wikitext text/x-wiki h56l485fcuwt14y4j5g5v5g5x6fxyyh 2242447 2242446 2024-11-07T14:15:55Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.dns.netbox 2242447 wikitext text/x-wiki j1zm12tsi57fsvuha0cd2emaer7hxvi 2242448 2242447 2024-11-07T14:18:13Z Stashbot 7414 lucaswerkmeister-wmde@deploy2002: esanders, lucaswerkmeister-wmde: Backport for [[gerrit:1087927|Deploy EditCheck (references) to hiwiki, bnwiki, idwiki (T366381)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) 2242448 wikitext text/x-wiki 1c7k0xfjqbcbnsnzngv26n114cq95o6 2242450 2242448 2024-11-07T14:20:43Z Stashbot 7414 lucaswerkmeister-wmde@deploy2002: esanders, lucaswerkmeister-wmde: Continuing with sync 2242450 wikitext text/x-wiki j7pqvscl4mgq7bo5w516js3sft8qqmm 2242451 2242450 2024-11-07T14:25:22Z Stashbot 7414 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for [[gerrit:1087927|Deploy EditCheck (references) to hiwiki, bnwiki, idwiki (T366381)]] (duration: 09m 37s) 2242451 wikitext text/x-wiki gtyuwwww36dc7md5vmo2qw45oim03te 2242453 2242451 2024-11-07T14:29:17Z Stashbot 7414 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be2082.codfw.wmnet with OS bullseye 2242453 wikitext text/x-wiki adho0kiv2fwfyy3sanl5l1o6suz1fsa 2242457 2242453 2024-11-07T14:41:52Z Stashbot 7414 moritzm: installing python-git security updates 2242457 wikitext text/x-wiki i76jpfb0z6ooxdyml6ot4p65w2rca4y 2242459 2242457 2024-11-07T14:55:57Z Stashbot 7414 hashar: Restarted CI Jenkins for plugins update 2242459 wikitext text/x-wiki 1ngrl0umuwlvd3z2gqqnyfykaahiy4l 2242460 2242459 2024-11-07T15:07:14Z Stashbot 7414 herron@cumin1002: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-logging-codfw 2242460 wikitext text/x-wiki 49ugcqkf514khjub5gbuljehg4d4v3i 2242461 2242460 2024-11-07T15:10:38Z Stashbot 7414 jnuche@deploy2002: Started deploy [releng/jenkins-deploy@abc27c0] (releasing): (no justification provided) 2242461 wikitext text/x-wiki j0rwnbxux4244u96trggqdzq7szeahm 2242462 2242461 2024-11-07T15:11:27Z Stashbot 7414 jnuche@deploy2002: Finished deploy [releng/jenkins-deploy@abc27c0] (releasing): (no justification provided) (duration: 00m 52s) 2242462 wikitext text/x-wiki 48co6thx6c7qdhgkhy8l8pe25cdo9c2 2242463 2242462 2024-11-07T15:14:28Z Stashbot 7414 jnuche@deploy2002: Started deploy [releng/jenkins-deploy@abc27c0] (releasing): (no justification provided) 2242463 wikitext text/x-wiki bap02npop6xszfhpf9whf2qi21r0cdx 2242464 2242463 2024-11-07T15:15:38Z Stashbot 7414 jnuche@deploy2002: Finished deploy [releng/jenkins-deploy@abc27c0] (releasing): (no justification provided) (duration: 01m 13s) 2242464 wikitext text/x-wiki lei99wr2tgobvr30ochiw7oo25bqo2r 2242465 2242464 2024-11-07T15:16:52Z Stashbot 7414 elukey@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be2082.codfw.wmnet with OS bullseye 2242465 wikitext text/x-wiki mbesk7pdn96gekt26cdwmfnry9o0q7j 2242466 2242465 2024-11-07T15:18:25Z Stashbot 7414 jgiannelos@deploy2002: Started deploy [restbase/deploy@6d0b97e]: Add new wikis to RESTBase 2242466 wikitext text/x-wiki 5k7rgp7jwycm2rf4rtudjlot5dqvkyz 2242467 2242466 2024-11-07T15:29:24Z Stashbot 7414 herron@cumin1002: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-logging-codfw 2242467 wikitext text/x-wiki jvmoiso0tntdluk74qdzu4dmufok7a2 2242468 2242467 2024-11-07T15:31:16Z Stashbot 7414 joelyrookewmde: joelyrookewmde@mwmaint2002:~$ foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https 2242468 wikitext text/x-wiki 1jf6a57d8gsi9s4zxle5hqpkuwqlevb 2242470 2242468 2024-11-07T15:31:57Z Stashbot 7414 taavi: taavi@deploy2002 ~ $ mwscript-k8s migrateUserGroup.php -- --wiki=labswiki contentadmin sysop # T375950 2242470 wikitext text/x-wiki cbuy0aynrdur6thoqih7zl3oidhc9fv 2242471 2242470 2024-11-07T15:33:12Z Stashbot 7414 herron@cumin1002: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-logging-eqiad 2242471 wikitext text/x-wiki pdyetcm319nty3uz3v2yvbhji2704xl 2242475 2242471 2024-11-07T15:39:59Z Stashbot 7414 jgiannelos@deploy2002: Finished deploy [restbase/deploy@6d0b97e]: Add new wikis to RESTBase (duration: 21m 33s) 2242475 wikitext text/x-wiki 5dnwnbizjx3u28fggc8lgayz65azy48 2242478 2242475 2024-11-07T15:52:02Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1010.eqiad.wmnet 2242478 wikitext text/x-wiki 3cz0g9a175llozz6038bfr4zl9wkhng 2242479 2242478 2024-11-07T15:53:22Z Stashbot 7414 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be2082.codfw.wmnet with OS bullseye 2242479 wikitext text/x-wiki pdcxcfa8rqx20rpfo4a2yvlr0opdzpw 2242480 2242479 2024-11-07T15:53:36Z Stashbot 7414 joelyrookewmde: Finished populateSitesTable for tcywiktionary (T378466) and tcywikisource (T378474) 2242480 wikitext text/x-wiki g0rkpt6neohrzld0di3xdifdxr9chhc 2242481 2242480 2024-11-07T15:54:54Z Stashbot 7414 moritzm: remove ganeti1010 from active ganeti nodes T378921 2242481 wikitext text/x-wiki 9ajexll466jvrt4vaprzcg7yaxz9dqm 2242483 2242481 2024-11-07T15:57:48Z Stashbot 7414 herron@cumin1002: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-logging-eqiad 2242483 wikitext text/x-wiki ata63wnucnyhh56dg82woigc3m2n35j 2242484 2242483 2024-11-07T16:04:19Z Stashbot 7414 elukey@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2082.codfw.wmnet with reason: host reimage 2242484 wikitext text/x-wiki tu9fu95wk7hs7s2noklkl5s17atg0u6 2242485 2242484 2024-11-07T16:07:48Z Stashbot 7414 elukey@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2082.codfw.wmnet with reason: host reimage 2242485 wikitext text/x-wiki pgvhkm6jo4kmq01o2z3rjkunlclh9qh 2242486 2242485 2024-11-07T16:15:33Z Stashbot 7414 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be2083.codfw.wmnet with OS bullseye 2242486 wikitext text/x-wiki qyhfn7zw9yufyif4u2ffubsd50fii4x 2242488 2242486 2024-11-07T16:23:37Z Stashbot 7414 arlolra@deploy2002: helmfile [staging] START helmfile.d/services/mobileapps: apply 2242488 wikitext text/x-wiki 4m02yp02q0g45rwqd8sxbh9oi37x2bc 2242489 2242488 2024-11-07T16:24:09Z Stashbot 7414 arlolra@deploy2002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply 2242489 wikitext text/x-wiki dz26d71dfyxuuzggle9q4vvaczugw20 2242492 2242489 2024-11-07T16:28:18Z Stashbot 7414 elukey@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2083.codfw.wmnet with reason: host reimage 2242492 wikitext text/x-wiki d44g0sbxrl7wesw5cu2j6tg6ee21pgi 2242494 2242492 2024-11-07T16:28:34Z Stashbot 7414 elukey@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin2002" 2242494 wikitext text/x-wiki kpgwi0hch8073woeg3mmskqc45e7z8j 2242497 2242494 2024-11-07T16:32:04Z Stashbot 7414 elukey@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2083.codfw.wmnet with reason: host reimage 2242497 wikitext text/x-wiki s4xiayusp012q1yv8l4p9155e5jdzsx 2242500 2242497 2024-11-07T16:35:00Z Stashbot 7414 elukey@cumin1002: START - Cookbook sre.hosts.provision for host ms-be2084.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2242500 wikitext text/x-wiki d835wglltblemoydv66b3yh3645gcqp 2242501 2242500 2024-11-07T16:41:35Z Stashbot 7414 fnegri@cumin1002: START - Cookbook sre.hosts.reimage for host cloudvirt1063.eqiad.wmnet with OS bookworm 2242501 wikitext text/x-wiki ngr6pl6fpyac0nm6vvllgi1q7wdjgah 2242502 2242501 2024-11-07T16:45:24Z Stashbot 7414 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be2084.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2242502 wikitext text/x-wiki e80176b1cozrl8q6wmz76eu1q80szmq 2242503 2242502 2024-11-07T16:46:53Z Stashbot 7414 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be2084.codfw.wmnet with OS bullseye 2242503 wikitext text/x-wiki 3owmmgr0hrm9ihbrgjkdt4zg66a044v 2242504 2242503 2024-11-07T16:48:54Z Stashbot 7414 elukey@cumin1002: START - Cookbook sre.hosts.provision for host fransc1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242504 wikitext text/x-wiki 3s66900bbvk6u0ond8nfj0bb8wa8z63 2242505 2242504 2024-11-07T16:48:56Z Stashbot 7414 elukey@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host fransc1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242505 wikitext text/x-wiki jq6fe0ww7j3jm8b8si30plz3fjlxsvr 2242506 2242505 2024-11-07T16:54:22Z Stashbot 7414 elukey@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2083.codfw.wmnet with OS bullseye 2242506 wikitext text/x-wiki nxpm4s5gmvy03dtnvbn0ju58n9w89hl 2242507 2242506 2024-11-07T16:54:32Z Stashbot 7414 arlolra@deploy2002: helmfile [eqiad] START helmfile.d/services/mobileapps: apply 2242507 wikitext text/x-wiki 93cd4izlhjus783ckhb0wcin6bh94ge 2242508 2242507 2024-11-07T16:56:01Z Stashbot 7414 fnegri@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1063.eqiad.wmnet with reason: host reimage 2242508 wikitext text/x-wiki 1hihj2xplq85p1g8vq2wusq4f7pykbd 2242509 2242508 2024-11-07T16:56:43Z Stashbot 7414 arlolra@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply 2242509 wikitext text/x-wiki d1aypneuxlobogeauorfcnnbqwrrirl 2242510 2242509 2024-11-07T16:56:51Z Stashbot 7414 arlolra@deploy2002: helmfile [codfw] START helmfile.d/services/mobileapps: apply 2242510 wikitext text/x-wiki b3owfzd7qf7e0sghw6ztlpn1jyv5alu 2242511 2242510 2024-11-07T16:57:10Z Stashbot 7414 arlolra@deploy2002: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply 2242511 wikitext text/x-wiki r70j79k9mcmx3ir9cnhhydm3d6tm28e 2242512 2242511 2024-11-07T16:57:14Z Stashbot 7414 elukey@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be2084.codfw.wmnet with OS bullseye 2242512 wikitext text/x-wiki j7skizz7wrq8f6f05nl0un6xgcfcbp0 2242513 2242512 2024-11-07T16:57:37Z Stashbot 7414 elukey@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin2002" 2242513 wikitext text/x-wiki irfdfglw1wrjeouuqgol9xgtcdw2l8k 2242514 2242513 2024-11-07T16:57:38Z Stashbot 7414 elukey@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2082.codfw.wmnet with OS bullseye 2242514 wikitext text/x-wiki iffnj9wy9z5j7m0l27wkc49qi9p1twm 2242515 2242514 2024-11-07T17:01:01Z Stashbot 7414 fnegri@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1063.eqiad.wmnet with reason: host reimage 2242515 wikitext text/x-wiki caurdxob0u254aqox9wlqovw2b6pfnu 2242516 2242515 2024-11-07T17:02:32Z Stashbot 7414 arlolra@deploy2002: helmfile [eqiad] START helmfile.d/services/mobileapps: apply 2242516 wikitext text/x-wiki rd0e5sx82n5ynow6v2lv8tm8og703r8 2242517 2242516 2024-11-07T17:03:35Z Stashbot 7414 arlolra@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply 2242517 wikitext text/x-wiki ipijy3i344es46nihozt5istf4qkh4q 2242518 2242517 2024-11-07T17:06:21Z Stashbot 7414 rzl: manually run mediawiki_job_wikimediaevents-UpdatePeriodicMetrics-per-wiki # T375508 2242518 wikitext text/x-wiki 0tm2fluhts352cir12vc6ntouksfing 2242519 2242518 2024-11-07T17:08:26Z Stashbot 7414 arlolra@deploy2002: helmfile [codfw] START helmfile.d/services/mobileapps: apply 2242519 wikitext text/x-wiki lyupzsm0valdahzb6nntohlswkjdu8u 2242520 2242519 2024-11-07T17:09:11Z Stashbot 7414 arlolra@deploy2002: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply 2242520 wikitext text/x-wiki fpdoqtzzsp31jqvc35ea2tvzjw9djg8 2242521 2242520 2024-11-07T17:12:11Z Stashbot 7414 rzl: manually run mediawiki_job_wikimediaevents-UpdatePeriodicMetrics-global # T375508 2242521 wikitext text/x-wiki k6kocuu0luajea9ly0zygbes58ujlgz 2242522 2242521 2024-11-07T17:16:03Z Stashbot 7414 cmooney@cumin1002: START - Cookbook sre.network.tls for network device fasw2-c1a-eqiad 2242522 wikitext text/x-wiki lpsmvtx1535533dvnsnie2lc53iz3jt 2242523 2242522 2024-11-07T17:18:18Z Stashbot 7414 cmooney@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device fasw2-c1a-eqiad 2242523 wikitext text/x-wiki lbn4damw4yyktc043c2vs6iot2guo4r 2242524 2242523 2024-11-07T17:27:30Z Stashbot 7414 fnegri@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - fnegri@cumin1002" 2242524 wikitext text/x-wiki 1z6upsrxk4h78croqncsqhmywt0nct1 2242525 2242524 2024-11-07T17:29:24Z Stashbot 7414 fnegri@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - fnegri@cumin1002" 2242525 wikitext text/x-wiki 4yszmaola3g2hz00kjusgpma6cbrfnb 2242526 2242525 2024-11-07T17:29:26Z Stashbot 7414 fnegri@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1063.eqiad.wmnet with OS bookworm 2242526 wikitext text/x-wiki lrn8tpt7vemt712ykqxn7mn87s0vx7l 2242527 2242526 2024-11-07T17:41:46Z Stashbot 7414 swfrench@deploy2002: helmfile [staging] START helmfile.d/services/changeprop: apply 2242527 wikitext text/x-wiki qaev8oms38w48tlyf8hiokza3k5iinw 2242529 2242527 2024-11-07T17:42:29Z Stashbot 7414 swfrench@deploy2002: helmfile [staging] DONE helmfile.d/services/changeprop: apply 2242529 wikitext text/x-wiki 683ayz07bv48on9k8a9h0nlj02r147j 2242531 2242529 2024-11-07T17:43:53Z Stashbot 7414 swfrench@deploy2002: helmfile [eqiad] START helmfile.d/services/changeprop: apply 2242531 wikitext text/x-wiki mnqbnbwuuq8l6p1eopro5n6d8plal5e 2242532 2242531 2024-11-07T17:44:30Z Stashbot 7414 swfrench@deploy2002: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply 2242532 wikitext text/x-wiki dww8sapskhaouiit45jot9zyqog3ikd 2242533 2242532 2024-11-07T17:48:06Z Stashbot 7414 swfrench@deploy2002: helmfile [codfw] START helmfile.d/services/changeprop: apply 2242533 wikitext text/x-wiki diblymgnfzb8nc1hr4j3u1azta3l7v3 2242534 2242533 2024-11-07T17:48:35Z Stashbot 7414 swfrench@deploy2002: helmfile [codfw] DONE helmfile.d/services/changeprop: apply 2242534 wikitext text/x-wiki 7vmmnwgxuzz8ko6r5qcltv0wxn69ajz 2242535 2242534 2024-11-07T17:55:21Z Stashbot 7414 fnegri@cumin1002: START - Cookbook sre.hosts.remove-downtime for cloudvirt1063.eqiad.wmnet 2242535 wikitext text/x-wiki 2ra6db0rnwdcis9dvfujwd5j5sab8uw 2242536 2242535 2024-11-07T17:55:23Z Stashbot 7414 fnegri@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cloudvirt1063.eqiad.wmnet 2242536 wikitext text/x-wiki 427bp7gzajwycev3lnirg11lyp7v36a 2242537 2242536 2024-11-07T17:57:31Z Stashbot 7414 swfrench@deploy2002: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply 2242537 wikitext text/x-wiki dj8enc1kidleugvd4p59y0tmyv4emw8 2242538 2242537 2024-11-07T17:58:02Z Stashbot 7414 swfrench@deploy2002: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply 2242538 wikitext text/x-wiki 5cozrp4keuea9vnugj9rnqzel9ny5ea 2242539 2242538 2024-11-07T17:59:52Z Stashbot 7414 swfrench@deploy2002: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply 2242539 wikitext text/x-wiki fb9q6y78bmosgg5dmnbvid1rlflpmbr 2242542 2242539 2024-11-07T18:01:32Z Stashbot 7414 swfrench@deploy2002: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply 2242542 wikitext text/x-wiki 6jgkmbjm7gz28s3bw3sqqc89hsvnbpd 2242543 2242542 2024-11-07T18:11:46Z Stashbot 7414 swfrench@deploy2002: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply 2242543 wikitext text/x-wiki l8nkmdxhobdfzdvfdio4uihrm6g2v4y 2242544 2242543 2024-11-07T18:13:25Z Stashbot 7414 swfrench@deploy2002: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply 2242544 wikitext text/x-wiki 3m72kwkj9exz90fbchhjh0fpi17zu88 2242545 2242544 2024-11-07T18:14:33Z Stashbot 7414 swfrench-wmf: updated changeprop-jobqueue to 2024-11-05-170900-production - T356241 2242545 wikitext text/x-wiki pabtlhy9eulv2i1p3tlasfw53m37l4s 2242546 2242545 2024-11-07T18:43:41Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wikikube-worker2138 to codfw - jhancock@cumin2002" 2242546 wikitext text/x-wiki 3urbj6144ib0nvb8bk04v9m4cewfzyn 2242547 2242546 2024-11-07T18:43:47Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wikikube-worker2138 to codfw - jhancock@cumin2002" 2242547 wikitext text/x-wiki m1wdm0zlijbovjnorkq74h9gfjm7qvs 2242548 2242547 2024-11-07T18:43:48Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2242548 wikitext text/x-wiki mvtzpejzjoe9ne9lab0vydsvzz4xe6g 2242549 2242548 2024-11-07T18:50:53Z Stashbot 7414 herron@cumin1002: START - Cookbook sre.ganeti.makevm for new host aux-k8s-worker1004.eqiad.wmnet 2242549 wikitext text/x-wiki aiydxxidx3v97oa8786aqb5imknz6wy 2242550 2242549 2024-11-07T18:50:55Z Stashbot 7414 herron@cumin1002: START - Cookbook sre.dns.netbox 2242550 wikitext text/x-wiki bdo0r1nbmkuodbvvrx3z9ouknx67qxe 2242551 2242550 2024-11-07T18:58:37Z Stashbot 7414 herron@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM aux-k8s-worker1004.eqiad.wmnet - herron@cumin1002" 2242551 wikitext text/x-wiki 14gk0htsirgv2dz35sbk3ulrxtcds2j 2242552 2242551 2024-11-07T18:58:52Z Stashbot 7414 herron@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM aux-k8s-worker1004.eqiad.wmnet - herron@cumin1002" 2242552 wikitext text/x-wiki c70duhjcxu5u66x39ymj9zi14b73v63 2242553 2242552 2024-11-07T18:59:04Z Stashbot 7414 herron@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2242553 wikitext text/x-wiki izsh97ntmdjdocffvxdukxoduaxuqib 2242554 2242553 2024-11-07T18:59:12Z Stashbot 7414 herron@cumin1002: START - Cookbook sre.dns.wipe-cache aux-k8s-worker1004.eqiad.wmnet on all recursors 2242554 wikitext text/x-wiki sx9b1eqz6rz3a3v9h5498hioho6wyon 2242555 2242554 2024-11-07T18:59:18Z Stashbot 7414 herron@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) aux-k8s-worker1004.eqiad.wmnet on all recursors 2242555 wikitext text/x-wiki ndyf054ndsy8fx43li7bb3qrm76wb5t 2242556 2242555 2024-11-07T18:59:24Z Stashbot 7414 herron@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM aux-k8s-worker1004.eqiad.wmnet - herron@cumin1002" 2242556 wikitext text/x-wiki 3l2qt1pt09dgaewmod4cll4k4g24p0x 2242557 2242556 2024-11-07T18:59:28Z Stashbot 7414 herron@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM aux-k8s-worker1004.eqiad.wmnet - herron@cumin1002" 2242557 wikitext text/x-wiki tadqmypcp7o73z5gln4jm10i4tjvpmo 2242558 2242557 2024-11-07T19:00:58Z Stashbot 7414 herron@cumin1002: START - Cookbook sre.hosts.reimage for host aux-k8s-worker1004.eqiad.wmnet with OS bookworm 2242558 wikitext text/x-wiki ow7jpxe6eqhb5u20tqbjnxhmcqw6ra4 2242559 2242558 2024-11-07T19:03:10Z Stashbot 7414 herron@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host aux-k8s-worker1004.eqiad.wmnet with OS bookworm 2242559 wikitext text/x-wiki 3gs1g7wy8pr2hl56dpounett2cg39zm 2242560 2242559 2024-11-07T19:03:11Z Stashbot 7414 herron@cumin1002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host aux-k8s-worker1004.eqiad.wmnet 2242560 wikitext text/x-wiki 5mfya7tjw419qir0i9a40igirnbb5s6 2242561 2242560 2024-11-07T19:06:24Z Stashbot 7414 aokoth@cumin1002: START - Cookbook sre.hosts.reboot-single for host vrts1003.eqiad.wmnet 2242561 wikitext text/x-wiki rdn8dftlu6j31456o6fclejcinexsw1 2242562 2242561 2024-11-07T19:08:36Z Stashbot 7414 mutante: VRTS - switching firewall provider from iptables to nftables 2242562 wikitext text/x-wiki kdyiqfzem43g9qz7zu7mb5ysl14pyoi 2242563 2242562 2024-11-07T19:10:36Z Stashbot 7414 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 0:10:00 on vrts2002.codfw.wmnet with reason: nftables 2242563 wikitext text/x-wiki qco4t2uooe76w9551pjicl16moby27j 2242564 2242563 2024-11-07T19:10:51Z Stashbot 7414 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on vrts2002.codfw.wmnet with reason: nftables 2242564 wikitext text/x-wiki g0f53iwo1ka4k6y6ruyxd4mr4xzl5da 2242565 2242564 2024-11-07T19:11:37Z Stashbot 7414 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 0:10:00 on vrts1003.eqiad.wmnet with reason: nftables 2242565 wikitext text/x-wiki mf8jot6b1et31m2hpap1s2240y7zclg 2242566 2242565 2024-11-07T19:11:41Z Stashbot 7414 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on vrts1003.eqiad.wmnet with reason: nftables 2242566 wikitext text/x-wiki jiuytc54blds2ub7sp4zifnclp08hae 2242568 2242566 2024-11-07T19:18:18Z Stashbot 7414 aokoth@cumin1002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host vrts1003.eqiad.wmnet 2242568 wikitext text/x-wiki 2pdqzk98gkutpj7u9xj498y9hf6igyz 2242569 2242568 2024-11-07T19:19:43Z Stashbot 7414 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 0:10:00 on vrts1003.eqiad.wmnet with reason: nftables 2242569 wikitext text/x-wiki gpxisl7cg7vry2sdprd3ss9gctskndm 2242570 2242569 2024-11-07T19:19:58Z Stashbot 7414 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on vrts1003.eqiad.wmnet with reason: nftables 2242570 wikitext text/x-wiki j4ez9o133tsijlki9hr6k8vk48cfl9z 2242571 2242570 2024-11-07T19:23:07Z Stashbot 7414 cdanis: T379199 💙cdanis@mwmaint2002.codfw.wmnet ~ 🕝☕ mwscript sql.php --wiki=testcommonswiki /srv/mediawiki/php-1.44.0-wmf.2/extensions/JsonConfig/sql/mysql/tables-generated.sql 2242571 wikitext text/x-wiki so4u48ivn2mihs22ehay145hojc1lk3 2242572 2242571 2024-11-07T19:33:22Z Stashbot 7414 cmooney@cumin1002: START - Cookbook sre.dns.netbox 2242572 wikitext text/x-wiki 51sm5mkrpttk2h7z9q1f6jlpf3yzvgu 2242573 2242572 2024-11-07T19:33:52Z Stashbot 7414 cmooney@cumin1002: END (ERROR) - Cookbook sre.dns.netbox (exit_code=97) 2242573 wikitext text/x-wiki o2mytpe3442pblloyu14kf3tfbwfuh1 2242574 2242573 2024-11-07T19:37:09Z Stashbot 7414 cmooney@cumin1002: START - Cookbook sre.dns.netbox 2242574 wikitext text/x-wiki tnz7qzzfgsralxqngm1fnnq90n6nvgz 2242575 2242574 2024-11-07T19:42:24Z Stashbot 7414 cmooney@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add dummy record for pfw1-eqiad.wikimedia.org - cmooney@cumin1002" 2242575 wikitext text/x-wiki 2qusygmmohn39qq39nfpm0zf458kb4y 2242576 2242575 2024-11-07T19:42:51Z Stashbot 7414 cmooney@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add dummy record for pfw1-eqiad.wikimedia.org - cmooney@cumin1002" 2242576 wikitext text/x-wiki k92682f7q95wg3o6zvxsyfgejtb7y2m 2242577 2242576 2024-11-07T19:42:53Z Stashbot 7414 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2242577 wikitext text/x-wiki 63natmw9ea1pexzx1sh41ydfl1qv2b6 2242578 2242577 2024-11-07T20:02:40Z Stashbot 7414 dduvall@deploy2002: Installing scap version "4.122.0" for 209 hosts 2242578 wikitext text/x-wiki ppjrq0potm3pdyfpwy6vs67j2d2cjcr 2242579 2242578 2024-11-07T20:10:17Z Stashbot 7414 cdanis@deploy2002: Started scap sync-world: Backport for [[gerrit:1087975|DB config for testcommonswiki deployment for Charts (T379199)]] 2242579 wikitext text/x-wiki 9u3vj4c06qfj7pxfpp2tpsmuqaldrmy 2242580 2242579 2024-11-07T20:13:07Z Stashbot 7414 cdanis@deploy2002: cdanis, bvibber: Backport for [[gerrit:1087975|DB config for testcommonswiki deployment for Charts (T379199)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) 2242580 wikitext text/x-wiki hdd85y2tsdrx3tbwr0xallnyn7gnnbn 2242581 2242580 2024-11-07T20:15:58Z Stashbot 7414 cdanis@deploy2002: cdanis, bvibber: Continuing with sync 2242581 wikitext text/x-wiki 7xx6t5mo8ruwbizeq1qaklrle8p2q17 2242582 2242581 2024-11-07T20:21:06Z Stashbot 7414 cdanis@deploy2002: Finished scap sync-world: Backport for [[gerrit:1087975|DB config for testcommonswiki deployment for Charts (T379199)]] (duration: 10m 45s) 2242582 wikitext text/x-wiki 4qfl1nn1j4zyblzyo4r04lygdm939u7 2242583 2242582 2024-11-07T20:22:19Z Stashbot 7414 cdanis@deploy2002: Started scap sync-world: Backport for [[gerrit:1087987|Enable Chart extension on testwiki and testcommonswiki (T378127)]] 2242583 wikitext text/x-wiki q7pyc83boqg5uz1ilgl10yko0kqledv 2242584 2242583 2024-11-07T20:25:02Z Stashbot 7414 cdanis@deploy2002: cdanis, aude: Backport for [[gerrit:1087987|Enable Chart extension on testwiki and testcommonswiki (T378127)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) 2242584 wikitext text/x-wiki fymaoss4xgkrh6ogl4p6ryv343yvye9 2242585 2242584 2024-11-07T20:30:20Z Stashbot 7414 cdanis@deploy2002: cdanis, aude: Continuing with sync 2242585 wikitext text/x-wiki 2wckj122kcfj78ea5b79sr1ax7a8dr1 2242586 2242585 2024-11-07T20:35:21Z Stashbot 7414 cdanis@deploy2002: Finished scap sync-world: Backport for [[gerrit:1087987|Enable Chart extension on testwiki and testcommonswiki (T378127)]] (duration: 13m 02s) 2242586 wikitext text/x-wiki k32q4rdp5etqbsgefpu50o4rbl21ipn 2242587 2242586 2024-11-07T20:43:22Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.dns.netbox 2242587 wikitext text/x-wiki o4xnscgqyyygw6ln4rl2jgtaklj6dux 2242588 2242587 2024-11-07T20:46:12Z Stashbot 7414 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be2082.codfw.wmnet with OS bookworm 2242588 wikitext text/x-wiki tdopiujpocsbbp1kknkvvu6hvenfdt5 2242589 2242588 2024-11-07T20:49:29Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wdqs2026 to codfw - jhancock@cumin2002" 2242589 wikitext text/x-wiki 10ipa7t5ngluo1vp9r0cfbik0cym1g0 2242590 2242589 2024-11-07T20:49:35Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wdqs2026 to codfw - jhancock@cumin2002" 2242590 wikitext text/x-wiki he7931s9l1hkp1bz73ddhcdghwcmagh 2242591 2242590 2024-11-07T20:49:36Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2242591 wikitext text/x-wiki 84dofyyqeubkri6l9vtusbnbup042kl 2242592 2242591 2024-11-07T20:50:13Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2026.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242592 wikitext text/x-wiki 5c8eeg6jyfxcviv3wji4bn0wwge09bs 2242593 2242592 2024-11-07T20:50:15Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2027.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242593 wikitext text/x-wiki noxz8xm3k11ibykvll70zmd3tqrl7up 2242594 2242593 2024-11-07T20:59:36Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.dns.netbox 2242594 wikitext text/x-wiki 6oojc45eaw9m2ch62hdvs5rzikqlt2q 2242595 2242594 2024-11-07T20:59:47Z Stashbot 7414 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2082.codfw.wmnet with reason: host reimage 2242595 wikitext text/x-wiki r14j13aiju22vbfvwxxwjsglepn4gn1 2242596 2242595 2024-11-07T21:01:06Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs2026.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242596 wikitext text/x-wiki izyb46eoddio0fhd89bbuuyf4ivxi3z 2242597 2242596 2024-11-07T21:01:32Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs2027.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242597 wikitext text/x-wiki bwhikkk598dcbgejna9q68lktx7xo9u 2242598 2242597 2024-11-07T21:02:43Z Stashbot 7414 jsn@deploy2002: Started scap sync-world: Backport for [[gerrit:1084883|Enable AutoModerator on viwiki (T378343)]] 2242598 wikitext text/x-wiki npn5qe8munv2mk3b0344xxivs4wehst 2242599 2242598 2024-11-07T21:03:06Z Stashbot 7414 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2082.codfw.wmnet with reason: host reimage 2242599 wikitext text/x-wiki 4g1b6c5i9a0zn636sjh037bku0kxx20 2242600 2242599 2024-11-07T21:03:07Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wikikube-worker2128 to codfw - jhancock@cumin2002" 2242600 wikitext text/x-wiki dnrsxoyaw4zkuqg5fl006azmuu95u28 2242601 2242600 2024-11-07T21:03:11Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wikikube-worker2128 to codfw - jhancock@cumin2002" 2242601 wikitext text/x-wiki ljowy6ffufwimufdjyljrt618hgrfjm 2242602 2242601 2024-11-07T21:03:16Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2242602 wikitext text/x-wiki fimf4rbcsmtltqm0m53xom6gcok25up 2242603 2242602 2024-11-07T21:06:24Z Stashbot 7414 jsn@deploy2002: suecarmol, jsn: Backport for [[gerrit:1084883|Enable AutoModerator on viwiki (T378343)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) 2242603 wikitext text/x-wiki a5rrnn5yf9cm1wjwdwhnk5qhg4pb92k 2242604 2242603 2024-11-07T21:06:33Z Stashbot 7414 jsn@deploy2002: suecarmol, jsn: Continuing with sync 2242604 wikitext text/x-wiki rztza1yfcyz2wihe6g4q6lhg1e820gb 2242605 2242604 2024-11-07T21:09:18Z Stashbot 7414 herron@cumin1002: START - Cookbook sre.hosts.reimage for host aux-k8s-worker1004.eqiad.wmnet with OS bookworm 2242605 wikitext text/x-wiki 15pa95lnyjhwkxe7dwg9ynscvqxjose 2242606 2242605 2024-11-07T21:11:11Z Stashbot 7414 jsn@deploy2002: Finished scap sync-world: Backport for [[gerrit:1084883|Enable AutoModerator on viwiki (T378343)]] (duration: 08m 28s) 2242606 wikitext text/x-wiki 7bt1wp7t1v6bh17pjj37nuhzblpl03k 2242607 2242606 2024-11-07T21:11:35Z Stashbot 7414 herron@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host aux-k8s-worker1004.eqiad.wmnet with OS bookworm 2242607 wikitext text/x-wiki q6tysuc4icw5tn4k2z6oet310cqiztz 2242608 2242607 2024-11-07T21:17:38Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wdqs2026'] 2242608 wikitext text/x-wiki 7f0cwtvnfdgy5a8fi27rcfv5eu7cafj 2242609 2242608 2024-11-07T21:17:43Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wdqs2027'] 2242609 wikitext text/x-wiki 1qh68eyilxxszz69z8bb1blrm4ivlov 2242610 2242609 2024-11-07T21:17:57Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['wdqs2026'] 2242610 wikitext text/x-wiki m3y8njlvyj1f12l438hc98fiqrtmpz0 2242611 2242610 2024-11-07T21:18:10Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['wdqs2027'] 2242611 wikitext text/x-wiki mayich38703jly1h0q154spb0vs96on 2242612 2242611 2024-11-07T21:18:36Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wdqs2026.codfw.wmnet with OS bullseye 2242612 wikitext text/x-wiki d4pce6461anomi08q9depl3omcywrju 2242613 2242612 2024-11-07T21:18:37Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wdqs2027.codfw.wmnet with OS bullseye 2242613 wikitext text/x-wiki b4qnyoe0pv73f1r6bc2vy4tg38o3v8b 2242614 2242613 2024-11-07T21:21:56Z Stashbot 7414 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ms-be2082.codfw.wmnet with OS bookworm 2242614 wikitext text/x-wiki jr3750v7e92y7u286msvqlsunhj7npg 2242615 2242614 2024-11-07T21:22:27Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.dns.netbox 2242615 wikitext text/x-wiki reuxbjars4lu944h2g8q0xdyzmhv73d 2242616 2242615 2024-11-07T21:26:26Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wikikube-worker2143 to codfw - jhancock@cumin2002" 2242616 wikitext text/x-wiki 7mlwaw2kuhnved5e2iurw66wywqwsrg 2242617 2242616 2024-11-07T21:26:31Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wikikube-worker2143 to codfw - jhancock@cumin2002" 2242617 wikitext text/x-wiki kghxky8kz627ur83dc0d56wvvt08gyy 2242618 2242617 2024-11-07T21:26:33Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2242618 wikitext text/x-wiki o28ylyg2d08ca54bne2es113ijs4upo 2242619 2242618 2024-11-07T21:27:01Z Stashbot 7414 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host ms-be2082.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART 2242619 wikitext text/x-wiki h24zevn41v2l008h1u6h111alqc765v 2242621 2242619 2024-11-07T21:30:36Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.dns.netbox 2242621 wikitext text/x-wiki ak0epci1ujotzqok27qzts3plo4zuif 2242622 2242621 2024-11-07T21:33:58Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wikikube-worker2158 to codfw - jhancock@cumin2002" 2242622 wikitext text/x-wiki l6t38tmiln6x8bv26gim6xqbzusq3ek 2242623 2242622 2024-11-07T21:34:04Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wikikube-worker2158 to codfw - jhancock@cumin2002" 2242623 wikitext text/x-wiki br0zkccee6ubzwizwggoiw86heu3a15 2242624 2242623 2024-11-07T21:34:06Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2242624 wikitext text/x-wiki 2s2l01z6c4npj9wfxgo7ob2gl8iy11i 2242625 2242624 2024-11-07T21:41:37Z Stashbot 7414 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be2082.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART 2242625 wikitext text/x-wiki lhpwm4ll86yxa7a6dw19gitln8ohx4e 2242626 2242625 2024-11-07T21:46:39Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2027.codfw.wmnet with reason: host reimage 2242626 wikitext text/x-wiki lf3beexlxdfeyq56bbyk29xweu97h3e 2242627 2242626 2024-11-07T21:46:50Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2026.codfw.wmnet with reason: host reimage 2242627 wikitext text/x-wiki gh5a7uh4osritmhl24xet1gt9n8sj8b 2242628 2242627 2024-11-07T21:47:32Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.dns.netbox 2242628 wikitext text/x-wiki qjrqsn5m346pzuf9g7sj3m7ipkiz1ic 2242629 2242628 2024-11-07T21:50:01Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2027.codfw.wmnet with reason: host reimage 2242629 wikitext text/x-wiki k1fw5psp8o6qqjisnr3p78rcplbbq72 2242630 2242629 2024-11-07T21:50:58Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wikikube-worker2166 to codfw - jhancock@cumin2002" 2242630 wikitext text/x-wiki n4e1c4m5q50q4fbzdf8zruzsg8817h2 2242631 2242630 2024-11-07T21:52:00Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wikikube-worker2166 to codfw - jhancock@cumin2002" 2242631 wikitext text/x-wiki 8irkvnzm9ljpwwi83a76hzrizwics7m 2242632 2242631 2024-11-07T21:52:01Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2242632 wikitext text/x-wiki b9mvfxoo8rfml81fohnsnzzrj95ogbw 2242633 2242632 2024-11-07T21:53:16Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2026.codfw.wmnet with reason: host reimage 2242633 wikitext text/x-wiki 3j4ao2ejdimhktjd275bboozu04pm5j 2242634 2242633 2024-11-07T21:53:34Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.dns.netbox 2242634 wikitext text/x-wiki lc8c1pxarwt4a8j5v0d82xiecd9hwfr 2242635 2242634 2024-11-07T21:58:09Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wikikube-worker2170 to codfw - jhancock@cumin2002" 2242635 wikitext text/x-wiki oir8af6e74z3se3hrg7e3upf8fpvppc 2242636 2242635 2024-11-07T21:58:15Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wikikube-worker2170 to codfw - jhancock@cumin2002" 2242636 wikitext text/x-wiki tqbry8q505vt5prgh779le1ipwggc0l 2242637 2242636 2024-11-07T21:58:16Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2242637 wikitext text/x-wiki 9e04w0p30sje2sc5qexjg72zhso39s8 2242638 2242637 2024-11-07T22:06:47Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" 2242638 wikitext text/x-wiki fjna0drk1g3hoqk4nrg0ti3pv4isu2d 2242639 2242638 2024-11-07T22:07:25Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" 2242639 wikitext text/x-wiki 7x3dz2971o9x0q1y0purot5lvuppaz5 2242640 2242639 2024-11-07T22:07:27Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2027.codfw.wmnet with OS bullseye 2242640 wikitext text/x-wiki b468pu6sk1txaoia70i98f6ttp7tg3d 2242641 2242640 2024-11-07T22:08:49Z Stashbot 7414 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host ms-be2082.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART 2242641 wikitext text/x-wiki fd0ioh70jz2j1h9w9kig5qjttalb3jc 2242642 2242641 2024-11-07T22:10:47Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" 2242642 wikitext text/x-wiki 43bp05ork3pbtx9lnfn465oa6z93dzh 2242643 2242642 2024-11-07T22:12:29Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" 2242643 wikitext text/x-wiki b8b5cgl0wuynexxixjmy25ku1mleq5l 2242644 2242643 2024-11-07T22:12:30Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2026.codfw.wmnet with OS bullseye 2242644 wikitext text/x-wiki psifoxe2sp8nwy7i4zdtjwnhjpprtje 2242645 2242644 2024-11-07T22:14:36Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2128.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242645 wikitext text/x-wiki q8oe5e6ktjj2u5s4qie481ua04gpank 2242646 2242645 2024-11-07T22:15:17Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2129.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242646 wikitext text/x-wiki ro04j6k0inwkv1nvm7rpjkan4iizmyf 2242647 2242646 2024-11-07T22:16:30Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2136.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242647 wikitext text/x-wiki on6q6c6ufruong8vrsfem645wt7stt5 2242648 2242647 2024-11-07T22:17:09Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2137.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242648 wikitext text/x-wiki 9x1g7z8bpm7qxj0ivscuqz6jzfskwj5 2242649 2242648 2024-11-07T22:17:41Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2138.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242649 wikitext text/x-wiki j603yh7vf4szc5xox5laxmoe499jpk2 2242650 2242649 2024-11-07T22:19:23Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2139.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242650 wikitext text/x-wiki 1b5zdnkwq0mrpx26k6vkdttf3c1cnbe 2242651 2242650 2024-11-07T22:19:58Z Stashbot 7414 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be2082.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART 2242651 wikitext text/x-wiki azvnhcv3cv8mv9ybtxz2hugcu3zokcc 2242652 2242651 2024-11-07T22:20:08Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2140.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242652 wikitext text/x-wiki qkxi7qev3wsrf9ikdlxssz711hr2gpc 2242653 2242652 2024-11-07T22:21:05Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2141.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242653 wikitext text/x-wiki nfb90xkob936em6jv2w0inlltmki55j 2242654 2242653 2024-11-07T22:22:14Z Stashbot 7414 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be2082.codfw.wmnet with OS bookworm 2242654 wikitext text/x-wiki 5hv6pxkz9wtyg3nrwl9qi0jbu6ik2pa 2242655 2242654 2024-11-07T22:22:45Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2142.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242655 wikitext text/x-wiki 7kyjqm2lyi5il0j3yk626g00sxa94hh 2242656 2242655 2024-11-07T22:23:26Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2143.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242656 wikitext text/x-wiki tjruxk25spn63fuh9v0xa520p8mwbhn 2242657 2242656 2024-11-07T22:24:05Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2144.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242657 wikitext text/x-wiki qcrbo2o4t44qbxkyo5nxzyjoqpmz04k 2242658 2242657 2024-11-07T22:24:59Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2128.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242658 wikitext text/x-wiki d8k9rs1vwkobkjruacdh0on6nv6s869 2242659 2242658 2024-11-07T22:25:08Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2145.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242659 wikitext text/x-wiki m0lxxml8u5438ynz62p50zarsfrzr8i 2242660 2242659 2024-11-07T22:25:52Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2156.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242660 wikitext text/x-wiki r09v4fgluwkn5lyxs7rp74djsn9boay 2242661 2242660 2024-11-07T22:26:07Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2129.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242661 wikitext text/x-wiki f8hkky3oi6y7f5i26kveqlem4tv4a7d 2242662 2242661 2024-11-07T22:27:04Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2157.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242662 wikitext text/x-wiki r4fny126txaabtn80ex3l483v7cby4z 2242663 2242662 2024-11-07T22:27:10Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2136.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242663 wikitext text/x-wiki r9mxkvz80bd1uo4do418u8ieweh5go3 2242664 2242663 2024-11-07T22:27:46Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2158.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242664 wikitext text/x-wiki du76mfxs0gjyni4fell0lauqjwgiz8z 2242665 2242664 2024-11-07T22:27:56Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2137.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242665 wikitext text/x-wiki dxudh8vjd19ras7aql24x3okpx4gao4 2242666 2242665 2024-11-07T22:28:20Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2138.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242666 wikitext text/x-wiki mja8joj3bzprgm3juxliffw1zziekjq 2242667 2242666 2024-11-07T22:28:37Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2159.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242667 wikitext text/x-wiki 8obvldgfvy4bkpsamfmgumvj2lbap2g 2242668 2242667 2024-11-07T22:29:26Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2160.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242668 wikitext text/x-wiki t3wk8fzr8tbiueob8jtv956j3tafmgr 2242669 2242668 2024-11-07T22:30:08Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2161.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242669 wikitext text/x-wiki l1t2gc8l56d3ojl9tpt9cnoh2ds8vqg 2242670 2242669 2024-11-07T22:30:25Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2139.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242670 wikitext text/x-wiki pm6v8nh2g5z7dac5wn75kalu87eg90c 2242671 2242670 2024-11-07T22:30:38Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2140.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242671 wikitext text/x-wiki 4izcdsvqin7ym6suirjos3a8dwg81o1 2242672 2242671 2024-11-07T22:30:46Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2162.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242672 wikitext text/x-wiki 9i2xmzpiagy6hxya9rqr4ar8cc54srq 2242673 2242672 2024-11-07T22:31:29Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2163.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242673 wikitext text/x-wiki k2pct05zd22qtrabm1qej8eyjl2hlfl 2242674 2242673 2024-11-07T22:32:55Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2164.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242674 wikitext text/x-wiki c3ynh3jhpn39cw96a3cc7gxs8rhbv4b 2242675 2242674 2024-11-07T22:33:41Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2165.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242675 wikitext text/x-wiki kfw2emsv6pe3huscn1m0nj4u08y3u29 2242676 2242675 2024-11-07T22:33:44Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2142.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242676 wikitext text/x-wiki ok7jpmm7ab07dd3d51i25365xuibb74 2242677 2242676 2024-11-07T22:34:13Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2143.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242677 wikitext text/x-wiki qzm0qptdm2kt5q26kjc8wdkuui9ed8h 2242678 2242677 2024-11-07T22:34:28Z Stashbot 7414 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2082.codfw.wmnet with reason: host reimage 2242678 wikitext text/x-wiki 8sm8at14szoykm4jb6vjh7ymcmu9g4g 2242679 2242678 2024-11-07T22:34:39Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2166.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242679 wikitext text/x-wiki 1a722gyeovdtfrwrh1dd31500gema5v 2242680 2242679 2024-11-07T22:34:53Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2144.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242680 wikitext text/x-wiki 1z7m87t9cd18hty5wpz78d83w56ghwu 2242681 2242680 2024-11-07T22:35:40Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2167.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242681 wikitext text/x-wiki ry012s6kk2o5nbqqmijr8tloxerl1vf 2242682 2242681 2024-11-07T22:35:51Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2145.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242682 wikitext text/x-wiki iju6wsnjdf5xach04uskskpfehzah41 2242683 2242682 2024-11-07T22:36:19Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2168.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242683 wikitext text/x-wiki 73tqk1nb7dyo45dscxx23azxe082a88 2242684 2242683 2024-11-07T22:37:02Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2169.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242684 wikitext text/x-wiki 5p1dbmpgbf3kofxc3925u7n4qy6uh65 2242685 2242684 2024-11-07T22:37:04Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2156.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242685 wikitext text/x-wiki 57h4mv76i1cnvp30nzj6v149zdqmree 2242686 2242685 2024-11-07T22:37:30Z Stashbot 7414 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2082.codfw.wmnet with reason: host reimage 2242686 wikitext text/x-wiki nrcsq3kbrek40lk3jeyns9x9be65d9s 2242687 2242686 2024-11-07T22:37:39Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2170.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242687 wikitext text/x-wiki o55d09hl8o6qcwb74onx1wo2i44hagl 2242688 2242687 2024-11-07T22:37:46Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2157.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242688 wikitext text/x-wiki c34fqlqigg27tccwsru7ayt6i6ni6ki 2242689 2242688 2024-11-07T22:38:30Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2158.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242689 wikitext text/x-wiki hzmflvvpbwq72wupf5po21fcp716raq 2242690 2242689 2024-11-07T22:39:18Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2159.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242690 wikitext text/x-wiki i8fga05ogwi95s0c3m7m5sj88e422hy 2242691 2242690 2024-11-07T22:39:56Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2141.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242691 wikitext text/x-wiki 5fhuv52j1vruiazp2p32ou0zgqffa4y 2242692 2242691 2024-11-07T22:40:36Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2160.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242692 wikitext text/x-wiki 3xb2t4fcyu0ovh4sdz7gd6hcuzsw13t 2242693 2242692 2024-11-07T22:41:24Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2161.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242693 wikitext text/x-wiki 3rfn27f6ouzmff1zea7dpk9s9gekw9j 2242694 2242693 2024-11-07T22:41:41Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2162.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242694 wikitext text/x-wiki pt59edetfomepqxssa0kr0baff3suqg 2242695 2242694 2024-11-07T22:42:48Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2163.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242695 wikitext text/x-wiki g1y6fdyh7nj67jewrw590apjo4rdvhx 2242696 2242695 2024-11-07T22:43:45Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2164.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242696 wikitext text/x-wiki m21731dmg21g6trhtu6l69553483gpp 2242697 2242696 2024-11-07T22:44:33Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2165.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242697 wikitext text/x-wiki 7ihza3s9n7qvicvxf162itkyv7s88n7 2242698 2242697 2024-11-07T22:45:51Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2166.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242698 wikitext text/x-wiki 18231kdhyvf2dnal1zl00hrf4md9wak 2242699 2242698 2024-11-07T22:46:59Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2167.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242699 wikitext text/x-wiki hn8pwb17gu9kfm8q2wx39szz21zvxc9 2242700 2242699 2024-11-07T22:47:13Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2168.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242700 wikitext text/x-wiki b4nu18jhil359oe4bkqwxj8on93ko5e 2242701 2242700 2024-11-07T22:47:50Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2169.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242701 wikitext text/x-wiki 0rogp06ziped2vakwtfoo2zwpvdl9j1 2242702 2242701 2024-11-07T22:48:26Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2170.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2242702 wikitext text/x-wiki 4643tyr61hd8rr75srry9ik232b1pqg 2242703 2242702 2024-11-07T23:00:41Z Stashbot 7414 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2082.codfw.wmnet with OS bookworm 2242703 wikitext text/x-wiki fs94a0m7ng91vi3boiar9642f3q4z2z 2242707 2242703 2024-11-08T07:33:15Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1047.eqiad.wmnet to cluster eqiad and group C 2242707 wikitext text/x-wiki beepe063kajv9j7p9q5wkmtlwdx93ja 2242708 2242707 2024-11-08T07:33:23Z Stashbot 7414 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1047.eqiad.wmnet to cluster eqiad and group C 2242708 wikitext text/x-wiki s21s984rtnmg8t5n9l8rv8kr19fa0uw 2242709 2242708 2024-11-08T07:33:53Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1047.eqiad.wmnet 2242709 wikitext text/x-wiki pgsepshve9is1ujlvmrvhbbhtw4vvmu 2242710 2242709 2024-11-08T07:39:14Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1047.eqiad.wmnet 2242710 wikitext text/x-wiki lc3fzjap00d7u1qwboe1vslmbzbiqgz 2242711 2242710 2024-11-08T07:39:43Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet 2242711 wikitext text/x-wiki pa99m61g20662jio9ffe2js6zra9xxe 2242712 2242711 2024-11-08T07:45:04Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet 2242712 wikitext text/x-wiki ti7q7vvdzxwhs3udi4wvr0fnxjey0st 2242713 2242712 2024-11-08T07:46:40Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1047.eqiad.wmnet to cluster eqiad and group C 2242713 wikitext text/x-wiki qy2ihmbwuasblb9lqw1r4sge56z5wz1 2242714 2242713 2024-11-08T07:48:05Z Stashbot 7414 XioNoX: manually install/test gnmic 0.39 on netflow6001 2242714 wikitext text/x-wiki 5luo7271gwmii56qcpg0qpldkbhbik0 2242715 2242714 2024-11-08T07:48:14Z Stashbot 7414 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1047.eqiad.wmnet to cluster eqiad and group C 2242715 wikitext text/x-wiki 87l4coh38dbwvlbia1k6ovgwcm6zqqr 2242716 2242715 2024-11-08T08:05:55Z Stashbot 7414 XioNoX: add gnmic 0.39 from official git repo to bookworm reprepro - T347461 2242716 wikitext text/x-wiki h4d7jbbkt8zgu6vrhtxrudv3pug4atx 2242717 2242716 2024-11-08T08:09:00Z Stashbot 7414 XioNoX: update gnmic to 0.39 on all netflow hosts 2242717 wikitext text/x-wiki n67b14b8nqzkattbc76jkglqkv5djrc 2242718 2242717 2024-11-08T08:26:41Z Stashbot 7414 moritzm: upgraded ircstream on irc.wikimedia.org to 1.0.1 2242718 wikitext text/x-wiki qs711ckp7sxaqcvao8puyy4dxsaqnha 2242719 2242718 2024-11-08T08:27:24Z Stashbot 7414 elukey@cumin2002: START - Cookbook sre.hosts.provision for host ms-be2082.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART 2242719 wikitext text/x-wiki puk9uqgby4e5wipznbke6rddlqt0skz 2242720 2242719 2024-11-08T08:27:44Z Stashbot 7414 elukey@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be2082.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART 2242720 wikitext text/x-wiki 7pywykvw44625sd4yjprye1b8hwnwsy 2242721 2242720 2024-11-08T08:30:17Z Stashbot 7414 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device cr2-eqsin 2242721 wikitext text/x-wiki 0o8o0esg8o2axcha3waeapysiop5pqg 2242722 2242721 2024-11-08T08:30:58Z Stashbot 7414 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cr2-eqsin 2242722 wikitext text/x-wiki msmuf2t61emwpgv5tw29j7werq38nu0 2242723 2242722 2024-11-08T08:31:08Z Stashbot 7414 elukey@cumin2002: START - Cookbook sre.hosts.provision for host ms-be2085.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2242723 wikitext text/x-wiki ixy0fvzrnzecm05txbcdr6f1kskolyc 2242724 2242723 2024-11-08T08:33:52Z Stashbot 7414 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2242724 wikitext text/x-wiki gxlzpzfkicxd4awmj4p0rlzxm0h7zwh 2242725 2242724 2024-11-08T08:34:40Z Stashbot 7414 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device cloudsw2-d5-eqiad 2242725 wikitext text/x-wiki q98tb88ah570sj14daq6k256cyvdifl 2242726 2242725 2024-11-08T08:34:54Z Stashbot 7414 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2242726 wikitext text/x-wiki 25eq93h8i67gln4n341ry681vg8eozg 2242727 2242726 2024-11-08T08:34:58Z Stashbot 7414 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cloudsw2-d5-eqiad 2242727 wikitext text/x-wiki 6g5hbb1cx1hyr7sea6n7t3h234chwal 2242728 2242727 2024-11-08T08:35:38Z Stashbot 7414 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device ssw1-e1-eqiad 2242728 wikitext text/x-wiki dpajukb3nffkpzplwyh4txooet9rar9 2242729 2242728 2024-11-08T08:35:47Z Stashbot 7414 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-e1-eqiad 2242729 wikitext text/x-wiki mroqmr7v38rqx0xxh5jr2c9ug8zxzzh 2242730 2242729 2024-11-08T08:39:40Z Stashbot 7414 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device ssw1-f1-eqiad 2242730 wikitext text/x-wiki 11qatpn21mugsa6jwwa0lay3suzqg7q 2242731 2242730 2024-11-08T08:39:48Z Stashbot 7414 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-f1-eqiad 2242731 wikitext text/x-wiki 5cmyk8m98fg1h5noho9pk17acxilvvw 2242732 2242731 2024-11-08T08:40:53Z Stashbot 7414 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device lsw1-a2-codfw 2242732 wikitext text/x-wiki 4uoj0j2wv7e7roy4d58s1u24mdfdx4m 2242733 2242732 2024-11-08T08:41:05Z Stashbot 7414 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a2-codfw 2242733 wikitext text/x-wiki 9j2em8y52bihbe8457ync5c73fdi6e9 2242734 2242733 2024-11-08T08:41:30Z Stashbot 7414 elukey@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be2085.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2242734 wikitext text/x-wiki 4fkxeptlmo9of1qix2tdbuwefak2s5n 2242735 2242734 2024-11-08T08:41:34Z Stashbot 7414 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device lsw1-a3-codfw 2242735 wikitext text/x-wiki lrtksqhicu7syxw0gmvh4333ebg374f 2242736 2242735 2024-11-08T08:41:47Z Stashbot 7414 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a3-codfw 2242736 wikitext text/x-wiki ne0hqy0am7a6jf83nbaj2wlomur22tj 2242737 2242736 2024-11-08T08:41:56Z Stashbot 7414 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device lsw1-a4-codfw 2242737 wikitext text/x-wiki ioqtzkrmocfmqirfv01juz8bb11x5rq 2242738 2242737 2024-11-08T08:42:08Z Stashbot 7414 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a4-codfw 2242738 wikitext text/x-wiki pmlbd8ma2f46569le1x65f1sbogyi9f 2242739 2242738 2024-11-08T08:42:14Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1048.eqiad.wmnet to cluster eqiad and group C 2242739 wikitext text/x-wiki b0llcqox71gep7pytcpf30u3q8jl4ws 2242740 2242739 2024-11-08T08:42:42Z Stashbot 7414 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device lsw1-a5-codfw 2242740 wikitext text/x-wiki 8jar5fh2ljinlgmc2211bxrn61b5c2j 2242741 2242740 2024-11-08T08:42:55Z Stashbot 7414 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a5-codfw 2242741 wikitext text/x-wiki mbn4zdqph2idhmq3r77xmpvranz0zwx 2242742 2242741 2024-11-08T08:43:11Z Stashbot 7414 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device lsw1-a6-codfw 2242742 wikitext text/x-wiki 0vhqvp77wkz5h23pghz1nc1kagpcbxv 2242743 2242742 2024-11-08T08:43:24Z Stashbot 7414 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a6-codfw 2242743 wikitext text/x-wiki prjo5t1mfml7cdyrsrjgc6sx2iv7o8z 2242744 2242743 2024-11-08T08:43:25Z Stashbot 7414 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1048.eqiad.wmnet to cluster eqiad and group C 2242744 wikitext text/x-wiki jub1putlp46f8e2kgh5wjm547nc2ben 2242745 2242744 2024-11-08T08:43:35Z Stashbot 7414 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device lsw1-a7-codfw 2242745 wikitext text/x-wiki jjh5rup25gm1mq6mljztbxt1h65l0m9 2242746 2242745 2024-11-08T08:43:47Z Stashbot 7414 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a7-codfw 2242746 wikitext text/x-wiki bqbkbm559d83xzihzydibb008028w41 2242747 2242746 2024-11-08T08:43:58Z Stashbot 7414 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device lsw1-a8-codfw 2242747 wikitext text/x-wiki npr65z4ri9sphberh9patz3vx99wb3t 2242748 2242747 2024-11-08T08:44:10Z Stashbot 7414 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a8-codfw 2242748 wikitext text/x-wiki n47y1rwm477rbq8j6uuae7y3booowui 2242749 2242748 2024-11-08T08:52:18Z Stashbot 7414 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device lsw1-b2-codfw 2242749 wikitext text/x-wiki t7jcokxp7y3s2cormg63xij70ffzsbr 2242750 2242749 2024-11-08T08:52:29Z Stashbot 7414 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b2-codfw 2242750 wikitext text/x-wiki jcz76hdmwvzgjpi8ikck8f3cs3sj9l2 2242751 2242750 2024-11-08T08:52:41Z Stashbot 7414 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device lsw1-b3-codfw 2242751 wikitext text/x-wiki lcputs32zmiowdq1ntlxpbz9b2zcuha 2242752 2242751 2024-11-08T08:52:50Z Stashbot 7414 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b3-codfw 2242752 wikitext text/x-wiki qa3ikm5d8tfx54u16jsm9cuwo4y97cg 2242753 2242752 2024-11-08T08:52:56Z Stashbot 7414 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device lsw1-b4-codfw 2242753 wikitext text/x-wiki daecu273sny7orqofteape7nerud8uy 2242754 2242753 2024-11-08T08:53:04Z Stashbot 7414 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b4-codfw 2242754 wikitext text/x-wiki e0fibzy666bmi57gewt2v8ka4vo1he5 2242755 2242754 2024-11-08T08:53:26Z Stashbot 7414 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device lsw1-b5-codfw 2242755 wikitext text/x-wiki 054la2o24sehjbzqt3k9h7cnumwoefs 2242756 2242755 2024-11-08T08:53:35Z Stashbot 7414 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b5-codfw 2242756 wikitext text/x-wiki 7r4sdcsqzmmxnl47jutuephfdw5m0sz 2242758 2242756 2024-11-08T08:54:08Z Stashbot 7414 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device lsw1-b6-codfw 2242758 wikitext text/x-wiki dpsyo9mruw1oygdcv6wxzzr7oyvxy5n 2242759 2242758 2024-11-08T08:54:17Z Stashbot 7414 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b6-codfw 2242759 wikitext text/x-wiki 2j5acrvmwhdyapshjymqwq10a7ty6sv 2242760 2242759 2024-11-08T08:56:50Z Stashbot 7414 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be2085.codfw.wmnet with OS bullseye 2242760 wikitext text/x-wiki 7pb2dbjd0slxyb87oe74p9xb6k7g72o 2242761 2242760 2024-11-08T09:01:02Z Stashbot 7414 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device lsw1-b7-codfw 2242761 wikitext text/x-wiki 349ffyuwtvg5bchmo1ldbis4cpebhl3 2242762 2242761 2024-11-08T09:01:10Z Stashbot 7414 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b7-codfw 2242762 wikitext text/x-wiki 3y4kp5kzfx80rf8x8itdfnyd6cpfmlu 2242763 2242762 2024-11-08T09:01:16Z Stashbot 7414 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device lsw1-b8-codfw 2242763 wikitext text/x-wiki gowkas3kyouorvzsbyw581bbd7hvcoi 2242764 2242763 2024-11-08T09:01:25Z Stashbot 7414 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b8-codfw 2242764 wikitext text/x-wiki 8i0w2ljrcby08d7my4joi4xwgzfny3y 2242765 2242764 2024-11-08T09:03:10Z Stashbot 7414 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device ssw1-a1-codfw 2242765 wikitext text/x-wiki l5c4ccp9ra6bhn6r4zkjvakc5a821cp 2242766 2242765 2024-11-08T09:03:19Z Stashbot 7414 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-a1-codfw 2242766 wikitext text/x-wiki j3sghdcon2sl56v7hiwhktpajq4n7xk 2242767 2242766 2024-11-08T09:03:41Z Stashbot 7414 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device ssw1-a8-codfw 2242767 wikitext text/x-wiki l3aq9nxoz5dpuceb0y69aylvhpvjqc7 2242768 2242767 2024-11-08T09:03:50Z Stashbot 7414 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-a8-codfw 2242768 wikitext text/x-wiki jbgnyavnrcoubzihw6i6nxwab86g5pe 2242769 2242768 2024-11-08T09:09:20Z Stashbot 7414 elukey@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be2085.codfw.wmnet with OS bullseye 2242769 wikitext text/x-wiki home32i30hvk0miiqwj29cv06mmoaek 2242770 2242769 2024-11-08T09:09:38Z Stashbot 7414 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be2085.codfw.wmnet with OS bullseye 2242770 wikitext text/x-wiki ndtdfx1qr3q5tk81sryvs8g3dmqfuvz 2242771 2242770 2024-11-08T09:20:35Z Stashbot 7414 elukey@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2085.codfw.wmnet with reason: host reimage 2242771 wikitext text/x-wiki govgpk2ufdkv0e1eb6mc4l4egtilkd2 2242773 2242771 2024-11-08T09:24:20Z Stashbot 7414 elukey@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2085.codfw.wmnet with reason: host reimage 2242773 wikitext text/x-wiki l1a61ufj7ek271s91p5kdy08xl6pviz 2242775 2242773 2024-11-08T09:29:13Z Stashbot 7414 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 10 days, 0:00:00 on an-presto1018.eqiad.wmnet with reason: Downtimed for further troubleshooting possible Hardware failure 2242775 wikitext text/x-wiki 1enk4l99hqqxj8tjpjiczpbl93vn3fx 2242777 2242775 2024-11-08T09:29:39Z Stashbot 7414 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on an-presto1018.eqiad.wmnet with reason: Downtimed for further troubleshooting possible Hardware failure 2242777 wikitext text/x-wiki mbi09m42bdffm7ng6314008qw6xw8qh 2242778 2242777 2024-11-08T09:38:55Z Stashbot 7414 elukey@cumin1002: START - Cookbook sre.hosts.provision for host ms-be2086.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2242778 wikitext text/x-wiki 3glnd15ht89qpzdz4mtscw0t7pdfzfz 2242779 2242778 2024-11-08T09:39:50Z Stashbot 7414 elukey@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin2002" 2242779 wikitext text/x-wiki j6nw5uioycc15potm3e523oemg63arn 2242781 2242779 2024-11-08T09:41:55Z Stashbot 7414 elukey@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin2002" 2242781 wikitext text/x-wiki fvc48h4pcf3ntrclg3rk6qu88sm092n 2242782 2242781 2024-11-08T09:41:57Z Stashbot 7414 elukey@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2085.codfw.wmnet with OS bullseye 2242782 wikitext text/x-wiki 1cq46oct0h5nrhdeqtdxmr1uyoy06g2 2242783 2242782 2024-11-08T09:49:20Z Stashbot 7414 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be2086.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2242783 wikitext text/x-wiki mdxq5asivjctwp4y06hxglxcpj0w0uf 2242784 2242783 2024-11-08T09:57:53Z Stashbot 7414 apergos: testing account creation backfill script on mwmaint2001 in screen session as ariel 2242784 wikitext text/x-wiki 1hut7zgwzewy3rkexpofzt5s6d7nuyx 2242785 2242784 2024-11-08T10:01:58Z Stashbot 7414 gmodena@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-dump-rev-content-reconcile-enrich: apply 2242785 wikitext text/x-wiki 9zqn8dxlgpkzi6b834cs2gge0yvkd0g 2242786 2242785 2024-11-08T10:02:04Z Stashbot 7414 gmodena@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-dump-rev-content-reconcile-enrich: apply 2242786 wikitext text/x-wiki 9uytp5lixm0ilf5ba0s1631e4c9g5xa 2242787 2242786 2024-11-08T10:16:31Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1011.eqiad.wmnet 2242787 wikitext text/x-wiki p3fnczykraog5cm9lti8xzdwvngs8m4 2242788 2242787 2024-11-08T10:16:58Z Stashbot 7414 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be2086.codfw.wmnet with OS bullseye 2242788 wikitext text/x-wiki hi6ft7mjuz7yrg6ysb1gwtcm5k44nzw 2242789 2242788 2024-11-08T10:18:57Z Stashbot 7414 elukey@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ms-be2086.codfw.wmnet with OS bullseye 2242789 wikitext text/x-wiki iricykkwtij3spq1dzsebhpv5ffqz8d 2242790 2242789 2024-11-08T10:19:32Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1011.eqiad.wmnet 2242790 wikitext text/x-wiki rahcqh98s6k6a8mfasksx5ponxli3c7 2242791 2242790 2024-11-08T10:29:41Z Stashbot 7414 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be2086.codfw.wmnet with OS bullseye 2242791 wikitext text/x-wiki rdaj4isys6cpt04acpw482440nt3glr 2242792 2242791 2024-11-08T10:34:38Z Stashbot 7414 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ms-be2086.codfw.wmnet with OS bullseye 2242792 wikitext text/x-wiki hmpot1y3auyhank68igbpifz8844ijr 2242793 2242792 2024-11-08T10:39:05Z Stashbot 7414 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2242793 wikitext text/x-wiki mvftkjp26y3c9npp5vacsqxm38eaqq6 2242794 2242793 2024-11-08T10:45:29Z Stashbot 7414 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be2086.codfw.wmnet with OS bullseye 2242794 wikitext text/x-wiki knaghh7q6jz7jt9t3gvshnklhu9jlu3 2242795 2242794 2024-11-08T10:45:54Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti2015.codfw.wmnet 2242795 wikitext text/x-wiki ow1cposq345sc96e34lvdrmhrplb5ep 2242796 2242795 2024-11-08T10:51:09Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.dns.netbox 2242796 wikitext text/x-wiki 4o6010b8i625ojd2jsy6q7t9j8kyykb 2242797 2242796 2024-11-08T10:55:26Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti2015.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" 2242797 wikitext text/x-wiki k3hwz9o9bskfotax33y1ljxhpf2b4df 2242798 2242797 2024-11-08T10:56:00Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti2015.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" 2242798 wikitext text/x-wiki pgv5i5bd3is4x8co3s382cqqd2ihnbl 2242799 2242798 2024-11-08T10:56:02Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2242799 wikitext text/x-wiki d6x953h61oc77qmnb94qpsq0io69rrh 2242800 2242799 2024-11-08T10:56:04Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2015.codfw.wmnet 2242800 wikitext text/x-wiki dligsq5aiiq3lfi892eo8gqs8hqli3n 2242801 2242800 2024-11-08T10:56:42Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti2016.codfw.wmnet 2242801 wikitext text/x-wiki bszortr2i6f7mg9et43rtsp6kc0lkc5 2242802 2242801 2024-11-08T10:58:21Z Stashbot 7414 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ms-be2086.codfw.wmnet with OS bullseye 2242802 wikitext text/x-wiki enfzkstxox9to4r0ix66yh4d1oikj8p 2242803 2242802 2024-11-08T11:00:24Z Stashbot 7414 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be2086.codfw.wmnet with OS bullseye 2242803 wikitext text/x-wiki gi59iggybp2fzq7ymbhyurjt3vgiqiy 2242804 2242803 2024-11-08T11:04:24Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.dns.netbox 2242804 wikitext text/x-wiki 02m5j2uv3zciam0qfa2s82v8oyta7i4 2242805 2242804 2024-11-08T11:05:58Z Stashbot 7414 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2242805 wikitext text/x-wiki t1dtni589f1it37qk7ohobimxs7oxh4 2242806 2242805 2024-11-08T11:07:11Z Stashbot 7414 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2242806 wikitext text/x-wiki gumlbmucu0gmmzzctw7t79pqm9hbcr9 2242807 2242806 2024-11-08T11:13:02Z Stashbot 7414 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ms-be2086.codfw.wmnet with OS bullseye 2242807 wikitext text/x-wiki fznxg7ft0c2zgni7ko3ewlndxlade3a 2242808 2242807 2024-11-08T11:13:26Z Stashbot 7414 elukey@cumin2002: START - Cookbook sre.hosts.provision for host ms-be2086.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2242808 wikitext text/x-wiki sozp0alfgejj5ejx10srz2cqgm06ssa 2242809 2242808 2024-11-08T11:13:53Z Stashbot 7414 elukey@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be2086.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2242809 wikitext text/x-wiki q3ygpssueerxpdomq4pwvuxpo75133s 2242810 2242809 2024-11-08T11:16:56Z Stashbot 7414 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2242810 wikitext text/x-wiki 0d3s103jjgbc7rfiah3lqv0rijdmrpn 2242811 2242810 2024-11-08T11:18:00Z Stashbot 7414 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2242811 wikitext text/x-wiki if7sf3u2zvckudqbhwpozc6xzw3hri0 2242812 2242811 2024-11-08T11:24:55Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti2016.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" 2242812 wikitext text/x-wiki 26osp3ky3muuev04jrt1p8q1vltu0ca 2242813 2242812 2024-11-08T11:25:35Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti2016.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" 2242813 wikitext text/x-wiki brerow7ffsreg555544goegqox3tlfn 2242814 2242813 2024-11-08T11:25:36Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2242814 wikitext text/x-wiki sge8623pj4l2o9in88gqge3gr99rpn0 2242815 2242814 2024-11-08T11:25:38Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2016.codfw.wmnet 2242815 wikitext text/x-wiki fzq72n30fovx3nbzhafwwkub3pkyoj1 2242816 2242815 2024-11-08T11:27:12Z Stashbot 7414 elukey@cumin1002: START - Cookbook sre.hosts.provision for host ms-be2087.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2242816 wikitext text/x-wiki ejy37fkc2z7hf9moqsxq3lx0k2gg7kb 2242817 2242816 2024-11-08T11:37:29Z Stashbot 7414 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be2086.codfw.wmnet with OS bullseye 2242817 wikitext text/x-wiki i62e17awku3uma0ovdrs1xwcdn60urh 2242818 2242817 2024-11-08T11:37:36Z Stashbot 7414 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be2087.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2242818 wikitext text/x-wiki 3ohreh5iovps5xrct1venpppxp6hs3f 2242819 2242818 2024-11-08T11:48:08Z Stashbot 7414 elukey@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2086.codfw.wmnet with reason: host reimage 2242819 wikitext text/x-wiki 0kwc663xglg7nclw50ykwnilxre95ue 2242820 2242819 2024-11-08T11:51:47Z Stashbot 7414 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2086.codfw.wmnet with reason: host reimage 2242820 wikitext text/x-wiki adnhw4raj60xxf00j4zq4qj1yx4esk7 2242821 2242820 2024-11-08T11:53:22Z Stashbot 7414 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be2087.codfw.wmnet with OS bullseye 2242821 wikitext text/x-wiki 7owgk3brvc6poteft0xclaaf5dy49ig 498 30942 2242425 2242032 2024-11-07T13:23:22Z Stashbot 7414 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch 2242425 wikitext text/x-wiki 99bgd9nf4e2umbmj609jelzciycyaen 2242428 2242425 2024-11-07T13:24:27Z Stashbot 7414 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch 2242428 wikitext text/x-wiki hjrqrgwktxhn7t74af6smk052ihhrl3 2242528 2242428 2024-11-07T17:42:25Z Stashbot 7414 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance 2242528 wikitext text/x-wiki g7nafcsxb3kl7m658pf5i3ywyu85nmw 2242530 2242528 2024-11-07T17:42:31Z Stashbot 7414 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) 2242530 wikitext text/x-wiki n4h2hj0l39x2ji3p31g3bnscvy42twx 0 56257 2242487 2233628 2024-11-07T16:20:46Z RCoccioli (WMF) 7416 /* Preparation - a few days before */ Add prepare step for DBs 2242487 wikitext text/x-wiki ngbvoz5n2ze7ttg5l4bmjsrj99cilnz 2242496 2242487 2024-11-07T16:31:22Z RCoccioli (WMF) 7416 /* Databases */ Add finalize step for databases. 2242496 wikitext text/x-wiki 9fp4vf2ub4b1tx5drn6ls8yncfydltv 4 400496 2242476 2052310 2024-11-07T15:40:32Z Taavi 13997 archive 2242476 wikitext text/x-wiki 485felly6twfvcrs6nodu6rznweljul 4 441136 2242491 2234743 2024-11-07T16:26:33Z Taavi 13997 remove contentadmin mentions 2242491 wikitext text/x-wiki irny9u6wet1opmvaiw8in0dclkttnq1 0 441827 2242704 2242009 2024-11-08T06:45:05Z Phedenskog 5986 Added missing descriptions 2242704 wikitext text/x-wiki s1bcaz6c3lysz91ccdz05yo0skaad1p 2242705 2242704 2024-11-08T06:47:13Z Phedenskog 5986 Fix link 2242705 wikitext text/x-wiki fy8nm5lqdiyaswj2y0mqybkvbomhccd 2242706 2242705 2024-11-08T07:09:00Z Phedenskog 5986 Add CSS/JS info 2242706 wikitext text/x-wiki ile6jzllza4cdvbbkl5g3fkd82eb213 0 446140 2242434 2202082 2024-11-07T13:47:40Z Cwhite 12916 add prompt indicating commands are to be run on titan hosts 2242434 wikitext text/x-wiki 09jdw6351ultktputbigbnn5s9eu9vg 2242449 2242434 2024-11-07T14:19:16Z Cwhite 12916 /* Delete blocks / free up space */ 2242449 wikitext text/x-wiki 6twditksm6lmtk462atp9ie098n9c03 498 446692 2242540 2208125 2024-11-07T18:00:11Z Stashbot 7414 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary on eqiad1, with recreate False, for hosts list: ['cloudvirt1063'] 2242540 wikitext text/x-wiki mvtmk1kbwbqbxocxjj79v522qxcwnan 2242541 2242540 2024-11-07T18:00:31Z Stashbot 7414 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary (exit_code=0) on eqiad1, with recreate False, for hosts list: ['cloudvirt1063'] 2242541 wikitext text/x-wiki myvzkyy8fpoy3epjc0oq38ye1q0zcht 0 447732 2242437 2174850 2024-11-07T13:55:37Z FNegri-WMF 32595 /* Reserved quota does not go down */ replace "tf-infra-test" with "tofuinfratest" 2242437 wikitext text/x-wiki s4haz4lfx4gxexw5swm7pkebdipjx66 2242458 2242437 2024-11-07T14:49:24Z FNegri-WMF 32595 /* Reserved quota does not go down */ more accurate description of the issue 2242458 wikitext text/x-wiki cswzus6tazt5933q65p3ymnlgo7ecws 0 448713 2242498 1932299 2024-11-07T16:34:12Z RCoccioli (WMF) 7416 /* The week before the switchover */ Add the prepare cookbook 2242498 wikitext text/x-wiki nhu4enol8vov3eahrhqwioyw26gv32u 2242499 2242498 2024-11-07T16:34:48Z RCoccioli (WMF) 7416 /* After the switchover */ Add the finalize cookbook 2242499 wikitext text/x-wiki hcqw00orjnefkt5uc5l4c9aerlqvtbq 0 451161 2242620 2162199 2024-11-07T21:30:02Z Cathal Mooney 24288 /* Verification */ 2242620 wikitext text/x-wiki 55eqt1l55hgnilhten2dal522wiggch 0 452701 2242473 2113043 2024-11-07T15:36:33Z Lucas Werkmeister (WMDE) 10037 /* Adding a new Wikidata Client */ use alias hostname for right mwmaint host in current datacenter 2242473 wikitext text/x-wiki d5rje3vv74ipl4pq22wqq356k9ckiuv 0 453382 2242757 2230043 2024-11-08T08:53:59Z Ayounsi 10040 /* Future improvements */ 2242757 wikitext text/x-wiki 94k81v74efxanfgzv8l885ojhhhhq5s 2242780 2242757 2024-11-08T09:40:14Z Ayounsi 10040 2242780 wikitext text/x-wiki q34aiezl0yvxsoco3p7ge6jx5qgi5or 116 453906 2242567 2242313 2024-11-07T19:15:13Z Gitlabaccountapprovalbot 37332 @zulf was approved. 2242567 wikitext text/x-wiki qope1yqnj4v76e6w8m5w1gm3rm4z09p 0 454593 2242469 2241972 2024-11-07T15:31:37Z ABorrero (WMF) 11379 /* Known problems */ T360626 is now resolved 2242469 wikitext text/x-wiki joviydkioinxi03pme7nkcncv6qsav0 2242472 2242469 2024-11-07T15:35:01Z ABorrero (WMF) 11379 /* Automated workflow via cookbook */ refresh cookbook commands 2242472 wikitext text/x-wiki 0e9n4u4s0lpkxq4z9b7nqd3w07i51mu 2242474 2242472 2024-11-07T15:37:50Z ABorrero (WMF) 11379 /* Manual workflow */ mention how to manually debug a MR 2242474 wikitext text/x-wiki sexuapn5unxu2lk607z3s3s3znal84z 498 455249 2242438 2199198 2024-11-07T13:58:00Z Stashbot 7414 dhinus: reset trove quota_usages: update quota_usages set reserved=0,in_use=0 where tenant_id='tofuinfratest'; 2242438 wikitext text/x-wiki li5n35lc5z33kanbrah9kl106ur507t 0 455675 2242490 2241140 2024-11-07T16:24:29Z Brouberol 37217 /* UNIX user impersonation */ 2242490 wikitext text/x-wiki 4jb0mn5vsyi3lyfhjnbdevc9re557q3 2242493 2242490 2024-11-07T16:28:18Z Brouberol 37217 /* Creating a new instance */ 2242493 wikitext text/x-wiki 0merq5p2e4e7i17eyts7f1zl9azumo4 2242495 2242493 2024-11-07T16:30:04Z Brouberol 37217 /* Migrate the scheduler and kerberos components to Kubernetes */ 2242495 wikitext text/x-wiki tf4vl4y7zo8bg5t5k2xhlx6ixl1ur2u 0 455855 2242423 2241959 2024-11-07T13:19:43Z Ayounsi 10040 /* Resources */ 2242423 wikitext text/x-wiki qtxppslza0hvg5mmx2wdaexrgf4rmi4 498 456098 2242412 2024-11-07T12:00:16Z Labslogbot 55 Auto update of instance info. 2242412 wikitext text/x-wiki ckfygu7jzc309udrwmc73i67k3p47b7 498 456099 2242426 2024-11-07T13:23:35Z Labslogbot 55 Auto update of instance info. 2242426 wikitext text/x-wiki jljizt37oxo9l2cnthc53c6xeu5kj8u 498 456100 2242439 2024-11-07T13:59:30Z Labslogbot 55 Auto update of instance info. 2242439 wikitext text/x-wiki p1s88fdb1s2mriiiiyrzvpbvm69gmd2 0 456101 2242772 2024-11-08T09:22:47Z Gehel 6860 Created page with "= Ongoing work = == Search Update Pipeline / Weighted tags == * Finalized schema-validating wrapper for spark kafka writer (released as part of eventutilities 1.4.0). However, first runs with real data revealed limitations in handling null values: Spark data frames may hold null that have to be excluded from the JSON validated against an event-stream-schema, since they do not support excplicit nulls (via union types). - https://phabricator.wikimedia.org/T374341 == Lang..." 2242772 wikitext text/x-wiki k1vlcvaaizmgyvmlvb42fcx1135mu01 2242774 2242772 2024-11-08T09:28:02Z Gehel 6860 /* Search Update Pipeline / Weighted tags */ 2242774 wikitext text/x-wiki c2ypj51xogp4kicuq2kzbq4gaxv1p60 2242776 2242774 2024-11-08T09:29:31Z Gehel 6860 /* Search Update Pipeline / Weighted tags */ 2242776 wikitext text/x-wiki f6avlcx07q0qu2wjqut09ezwudae78i