apme/v5-deeper-marker-find #8
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "apme/v5-deeper-marker-find"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Two v5 bugs surfaced when validating against fresh WebSphere VMs: 1. roles/discover/tasks/gather/filesystem.yml — the marker-file find used -maxdepth 4 from each scan_dir root. WebSphere Liberty's ws-server.jar sits at /opt/ibm/wlp/bin/tools (depth 5 from /opt) and traditional WAS's WAS.product sits at /opt/IBM/WebSphere/AppServer/properties/version (depth 6). Both were silently dropped, leaving the websphere / websphere_liberty detectors with only the systemd-service method matching — 1 method → low confidence → no fact file emitted. Bumped to maxdepth 8. 2. playbooks/utils/apply-v5-to-aap.yml — two fail_msg strings used the Jinja expression {{ project_name!r if false else project_name }} which relies on Python's !r repr filter. Modern Ansible (2.20+) lazy-parses templates and chokes on !r before the 'if false' short-circuit kicks in. Replaced with plain quoted expansion. Also added a localhost_inventory_name variable (default 'Localhost') so installs with a differently-named utility inventory (e.g. 'Demo Inventory') don't need to create a new inventory just to satisfy apply-v5-to-aap.yml's hardcoded GET.Found while validating WebSphere detection. Across the gather phase (processes.yml, filesystem.yml, java_envs.yml, assemble.yml) the construct `(X.stdout_lines | string)` was used to coerce a possibly- None register output to something the next pipe could chew on. In modern Jinja, `list | string` returns the Python repr — a string like `"['line1', 'line2']"` — and the downstream `[1:]` slices the string, `| list` iterates *characters*, `map('regex_replace', ...)` runs once per character, and `for line in ...` iterates characters too. Net effect: discovered_marker_files, discovered_processes, discovered_process_commands, discovered_ports (in the parse loop), discovered_java_environments (parse loop), and environment (in assemble.yml) all degraded to silent garbage on every host. The detector then matched almost nothing via the process / filesystem- marker / package paths, leaving every host at 'low' confidence unless the systemd-service or shallow-directory check happened to fire on its own. Concretely surfaced on websphere-liberty and websphere-traditional VMs in test-fleet: marker_files was a list of literal chars like ['[', "'", '/', 'o', 'p', 't', ...], so detector regex_search for '/opt/ibm/wlp' against single chars produced zero hits. Fix is mechanical — drop the '| string' coercion; `X.stdout_lines | default([])` already handles the missing-register case correctly (returns [] not None).Same '(X.stdout_lines | string)' bug found in 8 more sites across build_analysis, java/{log_config,keystores,cron,network}, war/introspect, and extract_single. With detection now reaching medium/high confidence on WebSphere VMs (post earlier fixes), the deep-inspect tasks finally executed and surfaced this latent bug — 'dict object has no attribute stdout_lines' raised when iterating a mangled character-array as a list. Also tightened the conditional in extract_single.yml lines 128-129: the previous '(X.stdout_lines | string) is defined' check evaluated the dotted expression before the defined-check, which races. Replaced with a plain 'X is defined' guard plus '| default([])' on stdout_lines.AAP runner pods are ephemeral — scan_fleet writes <slug>.facts.json to its pod's /tmp, then dies. fan_out spawns N per_app jobs that land on *different* runner pods and can't see the file. The discover_output_dir extra_var only renames the (still pod-local) path; it doesn't make /tmp shared. Workaround: fan_out already has the full fact data in the discovered_apps workflow artifact (the set_stats publish from roles/discover/tasks/emit_fact_file.yml). Now passes the whole app dict inline via the per-launch extra_var app_fact_inline. per_app.yml grows two new tasks that materialize that inline payload to {{ discover_output_dir }}/{{ slug }}.facts.json before the stat check fires. When app_fact_inline isn't set (e.g. a hand-launched JT-98 run from a developer laptop where discover and per_app share /tmp), the materialization is skipped and the original 'discover must run first' guard still applies.Pairs with the prior fan_out commit. fan_out now passes the full app dict as the app_fact_inline extra_var; per_app.yml writes it to {{ discover_output_dir }}/{{ slug }}.facts.json before the stat check fires. When app_fact_inline isn't set (laptop run with shared /tmp), the materialization is skipped and the 'discover must run first' guard still applies.Container render derived _rm_struct.slug from app.id (e.g. 'websphere_liberty') while emit_fact_file.yml writes app.slug as '<hostname>-<app_id>' ('websphere-liberty-websphere-liberty', regex-cleaned). Deploy then looks at publish_bundle_source_dir/<app_slug>/ — the host-prefixed form — and 404s. Use app.slug (fall back to app.id) so both render and deploy agree on the directory name. Same fix on image_ref so the produced image name is unique-per-host.git rev-parse on a missing ref prints the ref string itself on stderr ('origin/bundle/<slug>') and returns 128 — but the previous script captured stdout (which was empty) into REMOTE_SHA, then the '|| echo ""' covered the exit. Net result: REMOTE_SHA was empty for missing refs but SOMETIMES the ref name leaked through, and force-with-lease then choked with 'cannot parse expected object name'. Fix: * git rev-parse --verify -q refs/remotes/origin/<branch> — atomic contract: prints the SHA on success, prints nothing & exits non-zero on missing ref. -q suppresses stderr. * Use 'git push -u' on the brand-new-branch path so the local branch starts tracking origin/<branch> after the first push (the next run's fetch + rev-parse then works on the fast path). Restored no_log on the calling task.llm/context.yml line 35 set _manifest_dir using app.id ('websphere_liberty') while the K8s render path uses app.slug ('daytrader-liberty-websphere-liberty' — hostname-prefixed). The LLM-authored Containerfile got written to the app.id dir; deploy copied from the app.slug dir and dropped only the 5 K8s manifests + VALIDATED.txt — no Containerfile in the bundle. Mirrored the app.slug fallback pattern from render.yml so both write to the same per-app directory.emit_fact_file.yml computed slug (= hostname-appid, regex-cleaned) only at fact-file write time. Downstream consumers that loop over detected_applications with loop_var=app (containerize's llm_containerfile.yml is the main one) didn't see app.slug — it was synthesized later in read_fact.yml's reconstructed app dict, but the loop variable shadowed that. Net effect: the loop body's '{{ app.slug | default(app.id) }}' fell through to app.id, and the LLM Containerfile landed in /tmp/discovery-reports/manifests/<app.id>/ (e.g. 'tomcat') instead of .../manifests/<hostname-appid>/ where deploy reads. Two parallel app hosts with the same app.id (e.g. tomcat-petclinic and tomcat-roller both detect 'tomcat') would even race-write the same dir. Compute slug at detect_single.yml using the same '<inventory_hostname>-<id>' formula emit_fact_file uses, so every consumer that reads detected_applications now has slug from the first moment.