Skip to content

fix(ci): Build deb/rpm packages in the release workflow and drop promtail - #23680

Merged
jnewbigin merged 3 commits into
mainfrom
jnewbigin/backfill-packaging-fix-main
Aug 3, 2026
Merged

fix(ci): Build deb/rpm packages in the release workflow and drop promtail#23680
jnewbigin merged 3 commits into
mainfrom
jnewbigin/backfill-packaging-fix-main

Conversation

@jnewbigin

@jnewbigin jnewbigin commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

What

Backport of #23598 to main: restore .deb/.rpm package production in the release build, and finish removing promtail from packaging.

Why

When the release build moved off loki-build-image (which bundled nfpm) to a plain golang:1.26.5 container, install_workflow_dependencies.sh never installed nfpm. tools/packaging/nfpm.sh then failed to find nfpm and, lacking set -e, swallowed the error. End result was that releases silently shipped without any .deb/.rpm assets.

A fix was made on the release-3.7.x branch. This ports the fix to main

Changes

  • .github/jsonnetfile.json + .lock.json: bump vendored grafana/loki-release a02bbf9479fc5eab, which adds go install github.com/goreleaser/nfpm/v2/cmd/nfpm@v2.45.2 to install_dist_dependencies().
  • Regenerated the vendored install script and workflow YAMLs via make release-workflows. The workflow changes are purely the lib-ref hash bump.
  • tools/packaging/nfpm.sh: add set -euo pipefail so a missing tool fails loudly instead of silently
  • and drop promtail from the packaged components
  • Delete tools/packaging/verify-{deb,rpm}-install.sh. Orphaned scripts that exercised the promtail package.

Testing / validation

  • Verified the regenerated workflow YAMLs differ only by the a02bbf9479fc5eab lib-ref hash (no GO_VERSION or other churn).
  • Validated the packaging output by regenerating packages from the published release binaries and byte-comparing against known-good releases: v3.7.2 and v3.6.11 both reproduce content-identical .deb/.rpm (18/18 each) with the pinned nfpm.
  • shellcheck clean on nfpm.sh.

Notes

  • SHA256SUMS does not (and never did) include the OS package files
  • The missing-package releases (v3.6.12/13, v3.7.3/4) have been backfilled separately.
…tail

When the release build moved off loki-build-image (which bundled nfpm) to a
plain golang container, install_workflow_dependencies.sh never
installed nfpm. tools/packaging/nfpm.sh then failed to find nfpm and,
lacking `set -e`, swallowed the error - so releases silently shipped
without .deb/.rpm packages (v3.6.12, v3.6.13, v3.7.3, v3.7.4).

This PR:
- bumps the vendored loki-release to a ref that installs nfpm@v2.45.2
- hardens nfpm.sh with `set -euo pipefail`
- drops promtail from the packaged components (no longer built for release)
- removes the orphaned verify-{deb,rpm}-install.sh scripts that only exercised the removed promtail package.
@jnewbigin
jnewbigin requested a review from a team as a code owner July 31, 2026 04:19
@jnewbigin

Copy link
Copy Markdown
Contributor Author

Rebase after #23686 is merged

@shantanualsi shantanualsi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@jnewbigin
jnewbigin merged commit 7061803 into main Aug 3, 2026
90 checks passed
@jnewbigin
jnewbigin deleted the jnewbigin/backfill-packaging-fix-main branch August 3, 2026 06:03
@loki-gh-app

loki-gh-app Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

The backport to release-3.6.x failed:

error cherry-picking: error running git cherry-pick: error running command 'git -c user.name=grafanabot -c user.email=bot@grafana.com cherry-pick -x 7061803f016ac3eeb5b9d1acadd33643e6b8c68a'
error: exit status 1
stdout: Auto-merging .github/jsonnetfile.json
CONFLICT (content): Merge conflict in .github/jsonnetfile.json
Auto-merging .github/jsonnetfile.lock.json
CONFLICT (content): Merge conflict in .github/jsonnetfile.lock.json
Auto-merging .github/vendor/github.com/grafana/loki-release/workflows/validate.libsonnet
Auto-merging .github/workflows/check.yml
CONFLICT (content): Merge conflict in .github/workflows/check.yml
Auto-merging .github/workflows/images.yml
CONFLICT (content): Merge conflict in .github/workflows/images.yml
Auto-merging .github/workflows/minor-release-pr.yml
CONFLICT (content): Merge conflict in .github/workflows/minor-release-pr.yml
Auto-merging .github/workflows/patch-release-pr.yml
CONFLICT (content): Merge conflict in .github/workflows/patch-release-pr.yml
Auto-merging .github/workflows/release.yml
CONFLICT (content): Merge conflict in .github/workflows/release.yml
Auto-merging tools/packaging/nfpm.sh
CONFLICT (modify/delete): tools/packaging/verify-deb-install.sh deleted in 7061803f01 (fix(ci): Build deb/rpm packages in the release workflow and drop promtail (#23680)) and modified in HEAD.  Version HEAD of tools/packaging/verify-deb-install.sh left in tree.
CONFLICT (modify/delete): tools/packaging/verify-rpm-install.sh deleted in 7061803f01 (fix(ci): Build deb/rpm packages in the release workflow and drop promtail (#23680)) and modified in HEAD.  Version HEAD of tools/packaging/verify-rpm-install.sh left in tree.

stderr: error: could not apply 7061803f01... fix(ci): Build deb/rpm packages in the release workflow and drop promtail (#23680)
hint: After resolving the conflicts, mark them with
hint: "git add/rm <pathspec>", then run
hint: "git cherry-pick --continue".
hint: You can instead skip this commit with "git cherry-pick --skip".
hint: To abort and get back to the state before "git cherry-pick",
hint: run "git cherry-pick --abort".
hint: Disable this message with "git config set advice.mergeConflict false"

To backport manually, run these commands in your terminal:

git fetch
git switch --create backport-23680-to-release-3.6.x origin/release-3.6.x
git cherry-pick -x 7061803f016ac3eeb5b9d1acadd33643e6b8c68a

Resolve the conflicts, then add the changes and run git cherry-pick --continue:

git add . && git cherry-pick --continue

If you have the GitHub CLI installed:

git push --set-upstream origin backport-23680-to-release-3.6.x
PR_BODY=$(gh pr view 23680 --json body --template 'Backport 7061803f016ac3eeb5b9d1acadd33643e6b8c68a from #23680{{ "\n\n---\n\n" }}{{ index . "body" }}')
echo "${PR_BODY}" | gh pr create --title 'fix(ci): Build deb/rpm packages in the release workflow and drop promtail [release-3.6.x]' --body-file - --label 'backport' --label '' --base release-3.6.x --web

Or, if you don't have the GitHub CLI installed (we recommend you install it!):

git push --set-upstream origin backport-23680-to-release-3.6.x

And open a pull request where the base branch is release-3.6.x and the compare/head branch is backport-23680-to-release-3.6.x

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

2 participants