fix(objstore): Ruler not starting and incorrect mkdir path when Thanos Store is enabled - #16555
Merged
Merged
Conversation
ringerc
force-pushed
the
fix-thanos-store-ruler-config
branch
from
March 5, 2025 03:45
334c5bb to
2ab01c1
Compare
ringerc
referenced
this pull request
Mar 7, 2025
Contributor
Author
|
Anyone? |
Contributor
|
@ringerc i'll have a look at this today. ty! |
The ruler will fail to start in "all" or "backend" startup target with message "Ruler storage is not configured; ruler will not be started" if Loki is configured to use the Thanos object store. The ruler configuration should be read from the 'ruler_storage' top level config key when thanos object store is enabled (-use-thanos-objstore=true or storage_config.use_thanos_objstore: true in the config file). A bug in the ruler-storage module startup check meant that the ruler.storage block that's supposed to be ignored in thanos storage mode was still being used to check whether the ruler was configured. So a placeholder ruler.storage block had to be provided to get the ruler to start, even though it would be ignored by ruler itself. Fix, so the ruler.storage block is completely ignored and can be entirely removed when thanos client is in use.
When the ruler-storage module starts, it tries to mkdir the local storage directory. This was not updated to use the path from the ruler_storage block when the thanos object store is enabled, and was still using the otherwise-ignored ruler.storage block instead. Fix so it properly uses ruler_storage.filesystem.dir or ruler_storage.local.directory when in thanos storage mode, ignoring the non-thanos-store ruler.storage.local.directory.
Emit a warning if ruler.storage is defined but ignored because
object_store.use_thanos_client: true causes the ruler_storage block
to be used instead. And vice versa when the ruler_storage config only
used with the thanos client is present, but the thanos client is not
enabled.
This is particularly important for ruler, because the thanos object
store client setting affects which ruler storage block is used even
when ruler is just using the local file system as a store.
The new warning looks like this:
level=warn msg="ruler.storage exists and is not empty, but will be ignored in favour of ruler_storage because storage_config.use_thanos_objstore is true."
ringerc
force-pushed
the
fix-thanos-store-ruler-config
branch
from
April 15, 2025 23:25
a159757 to
dd4ef05
Compare
Contributor
Author
|
Thanks @ashwanthgoli . I've rebased it. This was incredibly confusing when I was converting to the Thanos object store config so it'd be good to have the bug fix landed. |
Contributor
Author
|
Anybody willing to review and merge this bug fix? |
ashwanthgoli
added
type/bug
Contributor
|
@ringerc extremely sorry for letting this go unattended for so long. Thank you for the thorough investigation and fixing it! ❤️ |
ashwanthgoli
changed the title
Contributor
|
@ringerc can you pull in the latest changes from main? its missing some recent CI changes. Happy to handle it if you don't mind |
Contributor
|
This seems to be failing |
Contributor
Author
|
Thanks very much |
This was referenced Jun 30, 2025
This was referenced Jul 21, 2025
This was referenced Aug 5, 2025
|
Will this get backported to 3.5? I'm not seeing this code in the v3.5.8 release. |
Contributor
|
@jameshartig This PR will be part of the Loki 3.6 release. |
felix0102
pushed a commit
to felix0102/loki
that referenced
this pull request
Aug 2, 2026
…s Store is enabled (grafana#16555)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
As discussed in #16543, loki does not completely switch from using the
ruler_storageconfiguration block when-use-thanos-objstore=trueorstorage_config.use_thanos_objstore: trueis set.It still uses the non-thanos-store config block
ruler.storageto:mkdirfor local rules storage during startup; andallandbackendtargetsFix both issues, so
ruler.storageis completely ignored when in thanos mode.Also add a warning when one of the config blocks is being overridden, but is non-empty, like:
and the inverse.
See also related docs change #16553
Fixes bug introduced in 8bca2e7 (#11713) per #16543 (comment)
This should be backported to v3.3.x and v3.4.x, possibly except the warning, since it's present in all releases containing the above commit.
Which issue(s) this PR fixes:
Fixes #16543
Special notes for your reviewer:
Checklist
CONTRIBUTING.mdguide (required)featPRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.docs/sources/setup/upgrade/_index.mddeprecated-config.yamlanddeleted-config.yamlfiles respectively in thetools/deprecated-config-checkerdirectory. Example PR