Home > Software engineering >  Why is Barman not compressing WAL files
Why is Barman not compressing WAL files

Time:10-17

Our Barman install doesn't seem to be compressing the WAL files. The wal file sizes are the same as they are on the Postgresql server. We've got streaming_archiver turned on, and in barman show-server compression is set to pigz, but wal_compression shows as off. Barman version is 2.12.

Thanks in advance.

barman show-server v2db:

Server v2db:
        active: True
        archive_timeout: 0
        archiver: False
        archiver_batch_size: 0
        backup_directory: /var/lib/barman/v2db
        backup_method: rsync
        backup_options: BackupOptions({'concurrent_backup'})
        bandwidth_limit: None
        barman_home: /var/lib/barman
        barman_lock_directory: /var/lib/barman
        basebackup_retry_sleep: 30
        basebackup_retry_times: 3
        basebackups_directory: /var/lib/barman/v2db/base
        check_timeout: 30
        checkpoint_timeout: 300
        compression: pigz
        config_file: /etc/postgresql/12/main/postgresql.conf
        connection_error: None
        conninfo: host=**********
        create_slot: manual
        current_lsn: 50/35C7FF00
        current_size: 49459828350
        current_xlog: 000000010000005000000035
        custom_compression_filter: None
        custom_decompression_filter: None
        data_checksums: off
        data_directory: /var/lib/postgresql/12/main
        description: Backups from DB
        disabled: False
        errors_directory: /var/lib/barman/v2db/errors
        has_backup_privileges: True
        hba_file: /etc/postgresql/12/main/pg_hba.conf
        hot_standby: on
        ident_file: /etc/postgresql/12/main/pg_ident.conf
        immediate_checkpoint: False
        included_files: ['/etc/postgresql/12/main/conf.d/9999-override.conf']
        incoming_wals_directory: /var/lib/barman/v2db/incoming
        is_in_recovery: False
        is_superuser: True
        last_backup_maximum_age: 7 days (latest backup: 2 hours, 52 minutes, 55 seconds )
        max_incoming_wals_queue: None
        max_replication_slots: 10
        max_wal_senders: 10
        minimum_redundancy: 3
        msg_list: []
        name: v2db
        network_compression: False
        parallel_jobs: 1
        passive_node: False
        path_prefix: /usr/lib/postgresql/12/bin
        pg_receivexlog_compatible: True
        pg_receivexlog_installed: True
        pg_receivexlog_path: /usr/lib/postgresql/12/bin/pg_receivewal
        pg_receivexlog_supports_slots: True
        pg_receivexlog_synchronous: False
        pg_receivexlog_version: 12.6
        pgespresso_installed: False
        post_archive_retry_script: None
        post_archive_script: None
        post_backup_retry_script: None
        post_backup_script: None
        post_delete_retry_script: None
        post_delete_script: None
        post_recovery_retry_script: None
        post_recovery_script: None
        post_wal_delete_retry_script: None
        post_wal_delete_script: None
        postgres_systemid: 6902948791374441963
        pre_archive_retry_script: None
        pre_archive_script: None
        pre_backup_retry_script: None
        pre_backup_script: None
        pre_delete_retry_script: None
        pre_delete_script: None
        pre_recovery_retry_script: None
        pre_recovery_script: None
        pre_wal_delete_retry_script: None
        pre_wal_delete_script: None
        primary_ssh_command: None
        recovery_options: RecoveryOptions({'get-wal'})
        replication_slot: Record(slot_name='backup_pgsql_masterbarman', active=True, restart_lsn='50/35000000')
        replication_slot_support: True
        retention_policy: RECOVERY WINDOW OF 4 WEEKS
        retention_policy_mode: auto
        reuse_backup: link
        server_txt_version: 12.12
        slot_name: backup_pgsql_masterbarman
        ssh_command: ssh -q postgres@******
        streaming: True
        streaming_archiver: True
        streaming_archiver_batch_size: 0
        streaming_archiver_name: barman_receive_wal
        streaming_backup_name: barman_streaming_backup
        streaming_conninfo: host=********
        streaming_supported: True
        streaming_systemid: 6902948791374441963
        streaming_wals_directory: /var/lib/barman/v2db/streaming
        synchronous_standby_names: ['']
        tablespace_bandwidth_limit: None
        timeline: 1
        wal_compression: off
        wal_keep_segments: 0
        wal_level: replica
        wal_retention_policy: MAIN
        wals_directory: /var/lib/barman/v2db/wals
        xlog_segment_size: 16777216
        xlogpos: 50/35C7FF00

barman.conf:

[barman]
barman_home = /var/lib/barman
barman_user = barman

log_file = /var/log/barman/barman.log

; rsync or postgres
backup_method = rsync
parallel_jobs = 1

; Incremental backup support: possible values are off (default), link or copy
reuse_backup = link

; Identify the standard behavior for backup operations: possible values are
; exclusive_backup (default), concurrent_backup
backup_options = concurrent_backup

; Immediate checkpoint for backup command - default false
immediate_checkpoint = False

; Global bandwidth limit in KBPS - default 0 (meaning no limit)
;bandwidth_limit = 4000

; Default compression level: possible values are None (default), bzip2, gzip, pigz, pygzip or pybzip2
compression = pigz

; Enable network compression for data transfers - default false
network_compression = False

; Number of retries of data copy during base backup after an error - default 0
basebackup_retry_times = 3

; Number of seconds of wait after a failed copy, before retrying - default 30
basebackup_retry_sleep = 30

; Minimum number of required backups (redundancy) - default 0
minimum_redundancy = 3

; Global retention policy (REDUNDANCY or RECOVERY WINDOW) - default empty
retention_policy = RECOVERY WINDOW OF 4 WEEKS

; Recovery options (influences recovery.conf written by barman recover)
recovery_options = 'get-wal'

; Time frame that must contain the latest backup date.
; If the latest backup is older than the time frame, barman check
; command will report an error to the user.
; If empty, the latest backup is always considered valid.
; Syntax for this option is: "i (DAYS | WEEKS | MONTHS)" where i is an
; integer > 0 which identifies the number of days | weeks | months of
; validity of the latest backup for this check. Also known as 'smelly backup'.
last_backup_maximum_age = 1 WEEK

configuration_files_directory = /etc/barman.d

path_prefix = /usr/lib/postgresql/12/bin

server.conf:

[v2db]
description = "Backups from DB"

ssh_command = "ssh -q postgres@******"
conninfo = "********** "
streaming_conninfo = "************** "
streaming_archiver = on
slot_name = "backup_pgsql_masterbarman"

active = true

CodePudding user response:

Parameter wal_compression is fetched from your PostgreSQL server configuration. It means you have configured wal_compression = off on your database server. Check in psql with:

show wal_compression;

You can check more info about this parameter on: https://postgresqlco.nf/doc/en/param/wal_compression/

Anyway, your WAL compression on barman side should work as It seems you have configured compression correctly. Make sure you have installed required OS library for pigz on barman server.

P.S. You will not see compressed WAL size during streaming operation. WAL log will be compressed after complete stream operation. But you can check already compressed WAL logs in wal directory.

Command to check your configured wals_directory:

barman show-server <server_name_here> | grep "wals_directory"
  • Related