Backup Exec 2014 handles removable media in a completely different way from the initial release of Backup Exec 2012 and BE 2010 R3 or earlier versions. This blog is an attempt to explain the operation and describe a scenario support have seen with removable media. I have based all information provided by a combination of training and developer supplied details, as well as testing I have performed myself to validate the information. Some of the behaviour documented is under review and may change in the future.
A few definitions to start things off:
- RDX, this is a specific type of removable media technology, however for simplicity I will use RDX to represent all types in the following notes
- Backup set, in the context of this document a backup set represents the backup of one resource on a server from one instance of running a backup job. As C: D: , System State, Mailbox Database etc are all different resources it can be seen that one backup job could create multiple backup sets and each time a job runs further sets are created.
- DLM (Data Lifecycle Management) this is a term for the processing and handling of disk based media introduced in Backup Exec 2012 and is separate from the tape (or legacy disk handling) mechanisms which depended on media set control
- BKF, a media file containing all or part of a non-GRT backup set
- IMG, a media folder containing a complete GRT enabled backup set
- SDR, Simplified Disaster Recovery
The following points outline some key points about DLM handling against RDX (some of the points also apply to standard disk storage backups)
1) RDX media actually means BKF or IMG and these relate directly to Backup Sets and can be seen in Backup Job Logs
2) the FLDRnnnnnn media object seen in the BE console represents each RDX cartridge but in reality should be considered to be a media container as it does not directly relate to one backup set.
3) There is no concept of APPEND to BKF or IMG as the process to use existing space on an RDX cartridge for a new backup set actually creates new BKF or IMG it does not add to an existing BKF or IMG
4) Overwrite is also misleading as again you do not overwrite a whole cartridge, the DLM process also does not re-use existing IMG or BKF files.
5) The process to remove expired backups sets and the associated on disk media is known as a RECLAIM
6) For RDX, a reclaim operation for BKF (non-GRT) media starts at the beginning of the creation of a BKF creating backup set. If a given job would create multiple backup sets, then multiple reclaim operations will be triggered during the timespan of the job. However if a GRT (IMG creating) backup set begins then the reclaim against BKF will not take place.
7) For RDX, a reclaim operation for IMG (GRT) media starts at the beginning of the job ONLY. This means that if an IMG is also part of a Last Recovery Chain it will be held until a further job runs.
8) The default 4 hour reclaim trigger that runs against standard disk storage and deduplication devices does not apply to RDX.
9) RDX cartridges do not try to reclaim when they fill up, as the cartridge gets ejected instead.
10) Once a reclaim operation starts the process looks for ALL expired media within the RDX cartridge to reclaim, it does not look just for a single backup set and also does not look for sets pertaining to only the previous history of the same job.
11) Expired backup sets will not be reclaimed if they form part of the Last Recovery Chain for a given resource, unless the setting for “Allow Backup Exec to delete all expired backup set” is enabled.
12) If multiple, separate job definitions are created to backup the same resource, this will create multiple Last Recovery Chains, one per resource against each unique job definition and not just one Last Recovery Chain against the last job that ran for the resource.
13) If the backup job is SDR enabled then all of the critical SDR backups sets must be expired and also not be part of Last Recovery Chain for them to be reclaimed. As per the GRT/IMG sets this means a further job needs to run to reclaim these sets
14) The System State Backup (which is also needed for SDR) is always the last backup set against a specific server in terms of ordering within the job.
15) If sets are expired but cannot be reclaimed they remain visible in the Storage section of the in the console with an Expired status.
16) No reclaim can be performed against RDX cartridges that are not inserted/mounted in a drive
17) An individual RDX cartridge (FLDRnnnnn media container) that has been disconnected (ejected) from the media server for an extended time will be set to “Limited to Read Only” operations after a default of 30days. This timing can be adjusted or disabled and the setting can also be manually enabled. Once enabled no DLM activity can take place on the cartridge.
18) Incremental or differential sets can also be dependent on existing backup sets and block the reclaim of expired media, although as GRT enabled incremental or differential sets need to be online at the same time as the previous sets in the chain, this scenario is less likely to be seen when using RDX and swapping cartridges from day-to-day. Although if the full and incremental or differential sets are all on one cartridge then this might be seen as it does extend the Last Recovery Chain.
Note: Points 6, & 7 are subject to review as whilst the behaviour as as intended, hindsight shows it may not be as desired. (See TECH226993)
Based on some of the above we recently experienced the following scenario within Tech Support:
Requirement was to backup one server using a different RDX carriage for each day of the week, re-using the cartridge on the same day the following week.
The amount of data being backed up per day could potentially amount to more than 50% of cartridge size.
SDR backups were required.
There was at least one GRT enabled set involved on the server concerned
A separate job definition had been configured for each day of the week.
The retention was set to less than 7 days
The first week’s jobs all ran correctly, from the second week onwards disk space issues started to occur. Investigation showed that in this scenario the RDX cartridge would need enough space for the following:
- One copy of every backup set in the job
- A second copy of every backup set the job that was marked as SDR critical (at least C: and System State, but potentially might require EFI partitions and further volumes). This is needed because the system state is always last and means the earlier SDR sets will not be reclaimed during the job (in fact the SDR sets will be kept until a 3rd backup job to the same cartridge is run)
- A second copy of every GRT enabled set
- A second copy of the backup set currently subject to backup activity within the job. This is needed because until the current backup of the same set finishes, the Last Recovery Chain locks the previous one. If calculating before the backup job starts, then base the disk space requirement on the biggest non-SDR set in the job.
- Potentially a second copy of the last non-GRT set (depending on whether or not only GRT sets followed it)
Depending on the backup data quantity on a day-to-day basis one cartridge did not always have enough space even though it easily could hold one complete backup of the server concerned.
This problem could be avoided in multiple ways (there are pros and cons to them all):
- Use larger RDX cartridges - would recommended for simplicity more than twice the biggest expected data size for one night’s backup cycle. As per above if the backup contains non-SDR data as well as SDR data, then it probably won’t always need twice the size but it is a simple rule of thumb.
- Use one job definition that repeats each day and contains weekly or monthly stages as needed, instead of one job definition per day. This would have the effect of only maintaining one Last Recovery Chain across all the cartridges used and then at the very start of the next job, all the expired backup sets on the available cartridge would be reclaimed (assuming the cartridge had been changed so did not now own the Last Recovery Chain)
- Enable the “Allow Backup Exec to delete all expired backup sets” option. This would mean that at the start of each night’s job all the expired sets should go from the cartridge. Care would need to be taken to make sure cartridges are swapped out properly and an understanding of when you might want to manually set Limit to Read Only might be needed.
- Disable SDR in the jobs (definitely NOT recommended but it is an option). This would allow things like C: drive to be reclaimed as the second backup set in the job starts, thus performing a rolling clean up of disk space against all backup sets during the job operation.
Notes:
Duplicating Backup sets from RDX to something else may change the expiration dependencies (this article does not currently discuss the possible effects of duplicate jobs)