This is the best practices guide for setting up an FTP Server for use with ClustrixDB Fast Backup and Restore, including how to avoid bottlenecks and optimizing for maximum throughput.
ClustrixDB implements fast backup and restore as a binary backup mechanism which works at the row level. Each ClustrixDB node sends its data directly to the FTP backup target in parallel, eliminating bottlenecks and allowing backup speed to scale with cluster size. Similarly for restore, the initiating node coordinates with other participating nodes in parallel to read from the dump file and restore replicas.
For more information on Fast Backup and Restore see: ClustrixDB Fast Backup and Restore.
Choosing an FTP Server
- We recommend using vsftpd or proftpd running on Linux. These FTP softwares have undergone the most testing by Clustrix QA and are used by the majority of ClustrixDB users.
Configuring the FTP Server
- Configure the FTP server to allow many concurrent connections from the Clustrix user account otherwise it is possible to bottleneck the backup/restore.
- We recommend using a dedicated FTP server for ClustrixDB backup/restore. If that is not feasible we recommend taking steps to ensure the FTP server isn't being used for other operations during the backup/restore process.
- The FTP server should be used on passive mode as ClustrixDB Fast Backup/Restore does not support active mode.
sufficient I/O and storage capacity. Fast I/O is essential to handle
the concurrency of the backup/restore process:
- Using a hardware RAID controller with RAID10 is recommended to increase read and write I/O performance.
- If I/O becomes a bottleneck, the backup/restore will take longer to complete. Since the backup/restore process pins BigC (our garbage collector), if it runs too long it could introduce latencies in the database or cause the disks to become full.
Use 10Gb or bonded 1Gb ethernet connectivity between the FTP server and the Ethernet switch connected to your ClustrixDB nodes in order to achieve maximum network bandwidth.
Configuring ClustrixDB Fast Backup and Restore
- Whenever possible, execute backups and restores during off-peak hours.
- Consider tweaking the value of backup_backup_concurrency global variable to a higher value to increase concurrency and backup speed. This variable default of '0' creates connections equal to the number of nodes in the cluster. This variable determines the number of tables that the system will backup in parallel. If increasing backup_backup_concurrency to a higher value introduces system sluggishness then it is recommended to revert back to the default (or other lower) value.
more nodes are added to your ClustrixDB installation, more concurrency
is also added, and more I/O will be required for your fast backup and
See more on how to run ClustrixDB Fast Backup and Restore.