[长期更新]victoriaMetrics常见配置或问题

[长期更新]victoriaMetrics常见配置或问题

Posted by 梁远鹏 on 2024-01-07 | 阅读 |,阅读约 2 分钟

TOC

说明

本文主要收集常见场景下 victoriaMetrics 配置以及遇到的一些常见问题,欢迎对本文进行投稿你认为好的场景配置或问题.

victoriaMetrics 从已有数据中启动报错 FATAL: incomplete vmrestore run; run vmrestore again or remove lock file

在我的 victoriaMetrics pod 通过备份好的数据启动时报错了.

lan@lan:~/lan/$ k logs vmetrics-lan
Defaulted container "vmetrics" out of: vmetrics, tools, restore (init)
2024-01-07T03:41:07.619Z        info    VictoriaMetrics/lib/logger/flag.go:12   build version: victoria-metrics-20231212-224619-tags-v1.96.0-0-g304fe0565
2024-01-07T03:41:07.620Z        info    VictoriaMetrics/lib/logger/flag.go:13   command-line flags
2024-01-07T03:41:07.620Z        info    VictoriaMetrics/app/victoria-metrics/main.go:70 starting VictoriaMetrics at ":8428"...
2024-01-07T03:41:07.620Z        info    VictoriaMetrics/app/vmstorage/main.go:108       opening storage at "victoria-metrics-data" with -retentionPeriod=1
2024-01-07T03:41:07.620Z        panic   VictoriaMetrics/lib/storage/storage.go:191      FATAL: incomplete vmrestore run; run vmrestore again or remove lock file "/victoria-metrics-data/restore-in-progress"
panic: FATAL: incomplete vmrestore run; run vmrestore again or remove lock file "/victoria-metrics-data/restore-in-progress"

goroutine 1 [running]:
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger.logMessage({0xbd5869, 0x5}, {0xc00007eb80, 0x75}, 0x1?)
        github.com/VictoriaMetrics/VictoriaMetrics/lib/logger/logger.go:301 +0xa91
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger.logLevelSkipframes(0x1, {0xbd5869, 0x5}, {0xc147ba?, 0xc0000ee2c0?}, {0xc0002116f0?, 0xbc10c0?, 0xc0000c40d0?})
        github.com/VictoriaMetrics/VictoriaMetrics/lib/logger/logger.go:141 +0x1a5
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger.logLevel(...)
        github.com/VictoriaMetrics/VictoriaMetrics/lib/logger/logger.go:133
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger.Panicf(...)
        github.com/VictoriaMetrics/VictoriaMetrics/lib/logger/logger.go:129
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.MustOpenStorage({0xbdf9a0?, 0xbd5302?}, 0x983fd98910000, 0x0, 0x0)
        github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/storage.go:191 +0x52d
github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage.Init(0xd915d0)
        github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage/main.go:111 +0x51e
main.main()
        github.com/VictoriaMetrics/VictoriaMetrics/app/victoria-metrics/main.go:74 +0x30e

一番检查后发现,备份数据是空的,但是 vmrestore 没有报错,只是写了一个文件锁,然后永远无法结束.

vmrestore 命令是这样的:

vmrestore-prod -src=fs:///tmp/data -storageDataPath=/opt/lan/vmetrics

其中 /tmp/data 是我下载好备份数据后解压数据的一个目录,但是备份的数据有问题导致解压后这个目录是空的,没有任何内容,但 vmrestore-prod 命令也没有任何错误提示… 因此导致数据恢复一直是进行中的状态,而 victoria-metrics 检测到数据目录有文件锁,也就是 /victoria-metrics-data/restore-in-progress,因此它认为目前还在进行数据恢复,所以一直启动失败.

最后解决是按照提示把/victoria-metrics-data/restore-in-progress删除掉,接着把正确的备份数据准备好就可以了.

vmbackup 备份数据到 s3

基本上按照官网文档操作就可以走通整个流程了, 官网地址:https://docs.victoriametrics.com/vmbackup/

下面是一个使用 vmbackup 命令将数据备份到 minio 的主要命令和配置:

vmbackup-prod -dst=s3://tmp/data -snapshot.createURL=http://localhost:8428/snapshot/create -storageDataPath=/opt/lan/vmetrics -s3TLSInsecureSkipVerify -customS3Endpoint=http://localhost:9000  -credsFilePath=credentials
[default]
aws_access_key_id=minioadmin
aws_secret_access_key=minioadmin

微信公众号

扫描下面的二维码关注我们的微信公众号,第一时间查看最新内容。同时也可以关注我的Github,看看我都在了解什么技术,在页面底部可以找到我的Github。

wechat-qrcode