Skip to content

fix(limits): Read the consumer group and topic from the ingest-limits config - #17831

Merged
periklis merged 10 commits into
mainfrom
use-cfg-consumer-group
Jun 2, 2025
Merged

fix(limits): Read the consumer group and topic from the ingest-limits config#17831
periklis merged 10 commits into
mainfrom
use-cfg-consumer-group

Conversation

@periklis

@periklis periklis commented May 27, 2025

Copy link
Copy Markdown
Collaborator

What this PR does / why we need it:
This pull request enables a separate configuration of the topic and consumer-group for the ingest-limits service. This enables to separate consumers from other components like partition-ingesters.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • Title matches the required conventional commits format, see here
    • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR
@periklis
periklis force-pushed the use-cfg-consumer-group branch from 648566c to bce467e Compare May 27, 2025 12:04
@periklis
periklis marked this pull request as ready for review May 27, 2025 12:04
@periklis
periklis requested a review from a team as a code owner May 27, 2025 12:04
@periklis periklis self-assigned this May 27, 2025
Comment thread pkg/limits/service.go Outdated
Comment thread pkg/limits/service.go Outdated
@pull-request-size pull-request-size Bot added size/L and removed size/S labels May 28, 2025
@github-actions

github-actions Bot commented May 28, 2025

Copy link
Copy Markdown
Contributor

💻 Deploy preview deleted.

@grobinson-grafana

Copy link
Copy Markdown
Contributor

Looks good but I think you also need to remove this function:

// MetadataTopic returns the metadata topic name for the given topic.
func MetadataTopic(topic string) string {
	return topic + ".metadata"
}

and update its usage in different places like the stream generator.

@pull-request-size pull-request-size Bot added size/M and removed size/L labels May 29, 2025
Comment thread pkg/loki/loki.go
@periklis periklis changed the title fix(limits): Read the consumer group from the config May 29, 2025
Comment thread tools/stream-generator/generator/kafka.go Outdated
Comment thread pkg/limits/config.go
// Make a copy of the streams from the request. We will prune this slice
// each time we receive the responses from a zone.
streams := make([]*proto.StreamMetadata, 0, len(req.Streams))
for _, stream := range req.Streams {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@periklis
periklis enabled auto-merge (squash) June 2, 2025 08:00
@periklis
periklis merged commit 60c7631 into main Jun 2, 2025
@periklis
periklis deleted the use-cfg-consumer-group branch June 2, 2025 08:06
@periklis periklis added type/bug Somehing is not working as expected backportk187 backport k257 and removed backportk187 labels Jun 2, 2025
loki-gh-app Bot pushed a commit that referenced this pull request Jun 2, 2025
felix0102 pushed a commit to felix0102/loki that referenced this pull request Aug 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport k257 size/M type/bug Somehing is not working as expected

2 participants