How to update Prometheus's configuration?

Hi! So, about prometheus configuration updating on the fly.

Update configuration file

The one problem is the deleting configuration file and changing inode file data.

Upload file to remote server and update conf file there:

import paramiko

hostname = "example.com"
username = "user

ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(hostname=hostname, username=username)
sftp = ssh.open_sftp()

# https://docs.paramiko.org/en/stable/api/sftp.html#paramiko.sftp_client.SFTPClient.file
with sftp.file(target_file, 'w+') as output, open(local_file, 'r') as input:
    while True:
        try:
            buffer = input.read(5000) # 5KB
            if buffer == '':
                print("Uploading was finished")
                break
            output.write(buffer)
        except Exception as ex:
            print(str(ex))
            break

sftp.close()
ssh.close()

Using sftp.put(local_file, target_file) will cause deleting the file.

Also you can use next bash construction:

cat input_file > target_file; rm input_file

Reload configuration

Prometheus has two options to do that:

1. Use Lifecycle API

   Bash example:

   curl -s -XPOST http://localhost:9090/-/reload
   

   Python example:

   import request
   request = requests.post("http://localhost:9090/-/reload")
   if request.status_code != 200:
       sys.exit(1)
   

2. Send HUP signal to Prometheus

   Bash example:

   # apt install psmisc
   killall -HUP prometheus
   
   # apt install procps
   kill -SIGHUP $(pgrep prometheus)
   

   Python example:

   import paramiko
   import time
   
   hostname = "example.com"
   username = "user
   
   ssh = paramiko.SSHClient()
   ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
   ssh.connect(hostname=hostname, username=username)
   stdin, stdout, stderr = ssh.exec_command("kill -SIGHUP $(pgrep prometheus)")
   
   # https://stackoverflow.com/questions/60037299/attributeerror-nonetype-object-has-no-attribute-time-paramiko
   time.sleep(1)
   ssh.close()
   
   

Source: prometheus/issues/1572.