v4: per-app fan-out, correct MTA usage, Jinja manifests, gitea PR bundles #2
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "final-pipeline-v3"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Rollup of all v4 work — see CHANGELOG.md for the per-PR breakdown.
* upsert_application: drop host-filesystem fallback for Application.binary. Previously `binary` was set to entry.deployment_path (e.g. /opt/tomcat/webapps/ROOT) when there were no maven coords. The MTA analyzer pod inside the cluster cannot read source-host paths and exited 1 every time. Now apps without mvn:// stay binary-less and the analyzer-queue logic in main.yml correctly skips them with mode=none. * pair-source-repo: PUT /hub/applications/{id} with repository set before queueing source-mode analyzer. Without it Konveyor fails the task immediately with "Application repository not defined." * pair-source-repo: drop the password-precondition `when:` — the token call already fails fast if password is empty. The previous condition silently skipped the POST in EEs where MTA_PASSWORD env var isn't injected, but the Summary still claimed success. * pair-source-repo: Summary message now reflects whether the analyzer was actually queued, skipped due to missing mta-app-id, or disabled via pair_reanalyze_on_mta=false. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>Replaces the hybrid Jinja+kai_refine generation with LLM-as-primary author (OpenRouter Claude Sonnet 4.6), Satellite as the system-package source, pair_apps as the MTA repo-attach + source-mode-analyzer step, and validate_artifacts (hadolint + kubeconform with LLM-retry on failure) as the safety net. End-to-end run against legacy-tomcat (live): - satellite_lookup: 692 packages parsed - generate_artifacts: bundle authored via Claude Sonnet 4.6 - validate_artifacts: hadolint + kubeconform pass - publish_to_mta: 6 Applications + 74 facts/app + host.satellite_packages - emit_survey_spec: 13-field survey for JT-B - pair_apps: PUT repository + queue source-mode taskgroup (live-verified app 71 + TG 41) Removed: generate_manifests, kai_refine, RHDH catalog YAML emission, pair-source-repo.yml, step2-containerize.yml, three KAI/Lightspeed docs. Kept dormant: rhdh/ directory. AAP wiring: meta/aap/{workflow-templates,job-templates,credential-types}. JTs 97/98/107 to be decommissioned; see docs/runbooks/06. 7 bugs caught and fixed during live testing: - satellite_lookup parser (nvra string format) - LLM markdown-fence wrapping - pair_apps tag-attach endpoint shape - validate_artifacts end_host gating - discover.yml localhost play + --limit interaction - Jinja `break` unsupported in Ansible - kubeconform schema placeholders + missing -ignore-missing-schemasThe /hub/applications/{id}/facts endpoint returns: {"<key>": {"source": "...", "value": <any>}, ...} Earlier code expected a list of {key, value, source} dicts (the shape used by the per-key PUT endpoint), which crashed with: 'str object' has no attribute 'key' Iterate .items() and pull v.value.ansible_date_time is undefined when the play uses gather_facts:false (containerize.yml does this to avoid SSH attempts during delegated runs). Switch the VALIDATED.txt/VALIDATION_FAILED.txt timestamp to lookup('pipe', 'date -u +%Y-%m-%dT%H:%M:%SZ') which doesn't depend on fact gathering.MTA /hub/applications/{id}/tags POST expects a single TagRef, not a list, so the array we sent returned 400 'cannot unmarshal array into Go value of type api.TagRef'. Switch to GET-merge-PUT pattern matching pair_apps: - Read current Application body - Drop the opposite verdict tag (bundle:ready vs bundle:invalid) so the verdict reflects the latest run - Add the current verdict - PUT the merged body Surfaced live in AAP JT-C run 21571 where validation passed but the verdict tag never landed on the MTA Application.ARCHITECTURE_V3.md: - §10 URLs: replace broken /ui/execution/... patterns with confirmed-working /execution/jobs/playbook/<id>/details links (user confirmed these load). For the workflow template detail page, document the navigation steps instead of an unstable deep link. - §11 new: honest answer to 'is MTA doing anything useful?' — covers what the Facts/Tags actually buy you (always useful) vs. the binary-mode analyzer (often fails for Tomcat sample webapps with no real artifact). - §12 new: known issues / gotchas — failed binary-mode taskgroups for sample webapps (e.g. legacy-jboss-docs Taskgroup 3 the user surfaced), AAP SPA deep-link flakiness, OpenRouter credit drain, one-bundle-per-app-server. - §13 new: status snapshot dated 2026-05-21 so the next session can pick up without re-deriving where we are. runbooks/04-workflow-template.md: - Topology diagram: collapse JT-D into JT-C since they were merged (one runner, shared filesystem for generate→validate). - Satellite pre-flight: correct username (anaeem, not admin) since Foreman PATs are bound to a specific user. - Troubleshooting: rename JT-D entries to JT-C; add OpenRouter 402 case + approval-node guidance. - Apply-AAP-manifests: replace the apocryphal 'ansible-playbook meta/aap/...' invocation with the actual REST POST pattern + note that the manifests are already applied in this AAP. runbooks/{04,06}: bulk fix /api/v2/job_templates → /api/controller/v2/job_templates (AAP 2.5 routes the controller API under /api/controller/, not /api/).Deleted 23 files that referenced deleted roles/playbooks/RHDH/legacy JT IDs as current state: docs/ - RUNBOOK.md, RHDH_INTEGRATION.md, EDA_INTEGRATION.md, AUDIT_OLD_VS_NEW.md - DESIGN_THIN_STEP2.md, RH_PRODUCT_SWAP_INVENTORY.md, MTA_TO_CONTAINERFILE.md - OPENROUTER_MIGRATION.md (history lives in CHANGELOG) - AAP_E2E_VALIDATION.md, INTERNAL_VERIFICATION.md, FLEET_VALIDATION.md - E2E_PROOF.md, BLIND_RUN_AUDIT.md, ANSIBLE_REVIEW.md - MTA_ANALYZER_VALIDATION.md, presentation.md, presentation.html - runbooks/02-end-to-end-migration.md - architecture/01-system-architecture.md, 03-step2-manifest-generation.md, 04-discovery-role-internals.md - fixtures/{fleet-report-final.json,petclinic-mta-partial.json} Updated to current state: - README.md: Documentation Map rebuilt around ARCHITECTURE_V3 + runbooks. - AGENTS.md: example playbook role list updated for v3. - docs/architecture/README.md: notes which diagrams were retired and points at ARCHITECTURE_V3 as the canonical reference. - docs/SBOM.md: SBOM consumer is now generate_artifacts, not kai_refine. - docs/SECURITY.md: LLM data transmission section rewritten for the direct OpenRouter call (no kai_refine redaction layer). - playbooks/README.md: rewritten around JT-A/B/C + migration-discovery-e2e. - roles/publish_to_mta/README.md: removes refs to publish_catalog, pair-source-repo.yml; documents the emit_survey_spec / fetch_facts / noop tasks that are part of v3. - roles/publish_to_mta/defaults/main.yml: comment on publish_to_mta_reanalyze_on_pair updated for the pair_apps path. - roles/publish_to_mta/templates/comments.md.j2: replaces the RHDH pair- source-repo link with the AAP workflow pairing instructions. - roles/fleet_report/README.md: rewritten — catalog data is built in memory only; no more YAML emission or migration-catalog push. - roles/fleet_report/templates/fleet-report.html.j2: Wave 1 callout references generate_artifacts (LLM-driven), not generate_manifests. - roles/{detect_apps,extract_config,deep_inspect}/README.md: substitute generate_manifests → generate_artifacts in role consumer references. - roles/sbom_scan/README.md: substitute kai_refine → generate_artifacts. All remaining 'kai_refine' / 'generate_manifests' / 'pair-source-repo' / 'emit_catalog' mentions are intentional historical commentary ('Replaces X', 'Used to be Y') that documents the migration path.Addresses every concern in docs/ARCHITECTURE_CRITIQUE.md except items 3 (repo_match auto-pair), 8 (taskgroup idempotency — partial, see roles/analyze), and 9 (telemetry). Live verified on 2026-05-21 against mta-openshift-mta.apps.anaeem.na-launch.com — pre-fix state had 43/52 hub tasks Failed; smoking-gun analyzer payload bugs are fixed and the v4 per-app architecture is wired end-to-end. Per-PR breakdown in CHANGELOG.md (v3.0.1, v3.1.0, v4.0.0-pre1, v4.0.0-pre2). Highlights: * PR-1 (v3.0.1) — Phase A smoking-gun fix - Stopped fabricating Application.binary='mvn://...' (the analyzer pod has no Maven repo access, so binary-mode taskgroups built on these URIs exit 1 on the addon's 'Fetching application' step). - Removed data.mode.artifact='' from taskgroup payloads. - Added GET-after-PUT verify-until loop in pair_one.yml before submitting the source-mode taskgroup (fixes the 10/10 'Application repository not defined' failures observed). - playbooks/cleanup-mta-stale-binary.yml: idempotent one-off that PUTs binary='' on every Application whose current binary starts with / (ran live; cleared 25 stale filesystem-path entries). * PR-2 (v3.1.0) — roles/analyze/ + facts trim - Extracted analyzer Taskgroup lifecycle into a reusable role with idempotency (skip if a same-named taskgroup has Pending/Running/Ready inner tasks), polling on tasks[].state (NOT taskgroup.state — the hub leaves the latter at Ready indefinitely, verified live), and Application tagging (analyzer:done / failed / skipped). Sibling status tags are evicted on re-tag so multiple runs don't pile up. - Wired into publish_to_mta and pair_apps via include_role. - Deleted orphaned submit_analyzer.yml. - publish_facts.yml trimmed from ~60 keys to ~44 (dropped static-analysis-derivable keys: maven.*, deployment.maven_*, deployment.manifest, deployment.web_xml, deployment.dependencies, *_count derivables, host.satellite_packages). Runtime-only facts (JVM heap/agents/classpath, listening ports, JDBC URLs, env vars, keystores, secrets) are kept — the Konveyor analyzer cannot see live-VM state. * PR-7 (v4.0.0-pre1) — Per-app fan-out via AAP - playbooks/per-app-analyze.yml: single-application entry point. - playbooks/fan-out.yml: controller that reads discovered_apps[] + pair_apps_repos[] from workflow artifacts and POSTs one launch per app to /api/controller/v2/job_templates/{id}/launch/. Throttle controls concurrency; supports fan_out_dry_run for testing. - meta/aap/job-templates/jt-perapp-analyze.yml and jt-e-fanout.yml. - Rewired migration-discovery-e2e workflow: scan_fleet -> review_gate -> pair_apps -> fan_out (JT-C and JT-D no longer in DAG; per-app jobs tracked in AAP Jobs view since AAP cannot represent runtime-spawned nodes). - containerize.yml --tags pair now set_stats-emits pair_apps_repos[] so fan-out can forward repo_url into each per-app launch. * PR-3 (v4.0.0-pre2) — roles/fetch_binary/ + bucket upload - Per-VM role pulls non-exploded WAR/EAR/JAR files off source hosts into a staging dir during scan (NOT at fan-out time — VMs may be offline by then). - publish_to_mta/upload_binary.yml multipart-POSTs each staged file to /hub/applications/{id}/bucket/<slug>/<file> and PUTs Application.binary = 'taskgroup:./<slug>/<file>'. Promotes the analyzer queue entry from mode=none to mode=binary inline so the analyzer runs against a real artifact. Apps with a paired repo stay at mode=source (deeper); the binary remains uploaded for re-analysis later. - Size cap (fetch_binary_max_size_mb=200) skips bulky fat-jars. * AUTH — hub Identity for source-mode clone - pair_apps/ensure_identity.yml idempotently creates/refreshes a hub Identity (kind: source) wrapping the gitea PAT (pair_apps_gitea_pat or GIT_TOKEN env). pair_one.yml merges its id into Application.identities. Fixes the 'git clone exit 128' the analyzer pod hit for legacy-jboss-sample on 2026-05-21. * PR-4 (v4.0.0-pre2) — roles/render_manifests/ Jinja templates - 7 deterministic K8s manifests (deployment, service, route, configmap, externalsecret, networkpolicy, kustomization) driven by a small per-app struct built from discovery facts. Two runs of the same scan produce byte-identical YAML. ConfigMap filters sensitive env keys; NetworkPolicy derives egress rules from discovered external_connections. * PR-5 (v4.0.0-pre2) — generate_artifacts to Containerfile-only - System prompt rewritten to request only {containerfile, notes}. - parse_response.yml writes only the Containerfile; legacy manifests/ kustomization keys are silently ignored with a debug warning. - K8s manifest authoring is fully delegated to render_manifests. * WIRE — per-app-analyze.yml stubs to real includes - Replaced TODO debug stubs with: fetch_facts_for_app (thin rehydration) -> render_manifests -> generate_artifacts (Containerfile) -> validate_artifacts. publish_bundle remains stubbed for PR-6. Live test results (2026-05-21): - Phase A cleanup: 25/25 stale binaries cleared, 0 remaining. - Phase B end-to-end: tag bootstrap, skip path, source-mode queue+poll on legacy-jboss-sample. Detected Failed inner task (taskgroup stayed Ready) and tagged analyzer:failed. The git clone exit 128 unblocked by the AUTH fix above. - Phase C dry-run: fan-out planner emits correct per-app launches. - Phase D render: all 7 templates render byte-identically; YAML parses. - 2 Jinja bugs found and fixed during testing (per-app-analyze.yml had !r Python repr syntax and a > 0 | ternary operator-precedence issue). Not done in this rollup (deferred to follow-ups): - PR-6 (gitea PR per app, roles/publish_bundle/) — stubbed. - Critique items 3 (auto-pair), 6 (tool-use loop), 9 (telemetry), 10 (scan cache). - The legacy fleet-level path is still reachable for one-release-cycle rollback safety; PR-8 (flag flip + legacy code removal) is the cleanup PR. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>Closes the last functional v4 loop. Every per-app job now lands its bundle as a gitea PR; engineer review is the unit of human approval. * PR-6 — roles/publish_bundle/ - Per-app role: idempotently ensures the target gitea repo exists (POST /api/v1/user/repos, tolerates 409), shallow-clones it, checks out (or creates) bundle/<slug>, copies the rendered manifests/<slug>/ tree in, commits if anything changed, force-with-lease pushes, opens or updates a PR. - Re-runs leave a 🔄 comment on the existing PR so reviewers know to look again. No commit, no comment when git diff --cached shows no change. - pr_body.md.j2 — PR body with bundle contents table, MTA cross-link, /revalidate instructions. - PAT sourced from publish_bundle_gitea_pat / GIT_TOKEN env. No PAT → role skips with debug message (CI-friendly). * PR-9 — eda/rulebooks/pr-comment-revalidate.yml - EDA webhook on :5001/endpoint, gitea issue_comment events. - Matches /revalidate on PRs titled "Migration bundle: <slug>", extracts slug from the title, fires JT-perapp-analyze with revalidate_only: true. * revalidate_only flag in playbooks/per-app-analyze.yml - When true, skips pair_apps + analyze + render_manifests + generate_artifacts so engineer-edited manifests/Containerfile are not clobbered. - Before validate, shallow-clones the existing bundle/<slug> branch into the local manifests dir so validate_artifacts runs against the engineer's edits. publish_bundle then re-pushes any validation outputs (e.g. updated VALIDATION_FAILED.txt). * playbooks/per-app-analyze.yml - Replaced the publish_bundle TODO stub with a real include_role call. End-to-end per-app pipeline now closes: scan → MTA → pair → analyze → render → Containerfile → validate → PR opened in gitea. Wire-up (one-time, not in this commit): - Create the gitea webhook on owner/migration-targets pointing at the EDA endpoint URL (see rulebook docblock for the exact event filter). - Register pr-comment-revalidate.yml as an EDA Activation in AAP. Still deferred: - PR-8 — delete the legacy single-pass code path (do this after v4 runs a release cycle clean). - Critique items 3 (auto-pair), 6 (tool-use loop), 9 (telemetry), 10 (scan cache). - Live-test of bucket upload (PR-3), Identity attach (AUTH), and the full per-app job loop — needs AAP apply of the JT YAMLs first. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>