Programming, robotics, traveling

28 Oct 2021

How to update Prometheus's configuration?

Hi! So, about prometheus configuration updating on the fly.

Update configution file

The one problem is the deleting configuration file and changing inode file data.

Upload file to remote server and update conf file there:

import paramiko

hostname = "example.com"
username = "user

ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(hostname=hostname, username=username)
sftp = ssh.open_sftp()

# https://docs.paramiko.org/en/stable/api/sftp.html#paramiko.sftp_client.SFTPClient.file
with sftp.file(target_file, 'w+') as output, open(local_file, 'r') as input:
    while True:
        try:
            buffer = input.read(5000) # 5KB
            if buffer == '':
                print("Uploading was finished")
                break
            output.write(buffer)
        except Exception as ex:
            print(str(ex))
            break

sftp.close()
ssh.close()

Using sftp.put(local_file, target_file) will cause deleting the file.

Also you can use next bash construction:

cat input_file > target_file; rm input_file

Reload configuration

Prometheus has two options to do that:

  1. Use Lifecycle API

    Bash example:

    curl -s -XPOST http://localhost:9090/-/reload
    

    Python example:

    import request
    request = requests.post("http://localhost:9090/-/reload")
    if request.status_code != 200:
        sys.exit(1)
    
  2. Send HUP signal to Prometheus

    Bash example:

    # apt install psmisc
    killall -HUP prometheus
    
    # apt install procps
    kill -SIGHUP $(pgrep prometheus)
    

    Python example:

    import paramiko
    import time
    
    hostname = "example.com"
    username = "user
    
    ssh = paramiko.SSHClient()
    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    ssh.connect(hostname=hostname, username=username)
    stdin, stdout, stderr = ssh.exec_command("kill -SIGHUP $(pgrep prometheus)")
    
    # https://stackoverflow.com/questions/60037299/attributeerror-nonetype-object-has-no-attribute-time-paramiko
    time.sleep(1)
    ssh.close()
    
    

Source: prometheus/issues/1572.