Amazon Simple Storage Service (S3) Migrator Facts in Enterprise Vault

In the blog, I will introduce details about Amazon S3 cloud migrator feature & its usage in Enterprise Vault. This will help you to understand architecture, feature details, how to use it as secondary migrator in Enterprise Vault & if any problem, how to start troubleshooting it.  

In Enterprise vault world, what is use of Amazon S3 cloud migrator?

• Customer can use Amazon S3 as a secondary storage location in the cloud to store data which is CAB files, created by the Enterprise Vault file collection software, which in turn contains *.DVS* files.
• In Enterprise vault 11, the feature name has been changed from Symantec Enterprise Vault Cloud Storage Connector (Amazon S3; Rackspace or AT&T Synaptic) to Symantec Enterprise Vault Cloud Storage Secondary Migrator.

Details of Feature Usage

• Enables Migration of archived data to Amazon S3 cloud storage.
• Enables Retrieval of archived data from Amazon S3 cloud storage.
• Enables Expiration of archived data from Amazon S3 cloud storage.

What’s the minimum requirement?

• The Enterprise Vault 10.0.1 or later installation supports migration to Amazon S3 storage
• Amazon S3 account
• At least one Amazon S3 bucket to store data.

Some interesting facts\issues highlighted via this article:

• Amazon Simple Storage Service (S3) store Multi Region support: Enterprise Vault started with support for Amazon Simple Storage Service (S3) by store the buckets storage based in US region. Enhancing this capability further, now with Enterprise Vault 10.0.4 onwards, customer can now choose other geographical regions as well where Amazon Simple Storage Service stores the buckets that they create using the Administration Console (VAC). A different Amazon S3 store region is an option to customers to optimize latency, minimize costs, or address regulatory requirements. Objects stored in one Amazon S3 store region never leave that region unless customer explicitly transfer them to another Amazon S3 store region.

• With Enterprise Vault 10.0.4, list of Amazon S3 store regions supported via Enterprise Vault are: US Standard, US West (Oregon), US West (Northern California), EU (Ireland), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo), South America (Sao Paulo). The Advanced tab of the vault store partition properties includes the Amazon bucket region parameter, which provides various region options.

• With Enterprise Vault 10.0.4 we support Amazon S3 instances across the world using both HTTP and HTTPS

Oh that’s great, what’s the architecture?

• The architecture is based on time tested Symantec’s OST framework. Well in Simple terms, here is how EV will communicate with Amazon S3 for storing, retrieval and expiration:

Backend of this architecture

• Answer is: Enterprise Vault Storage Streamer API:

The Enterprise Vault Streamer exposes interfaces that benefit the communication of Enterprise Vault with third party Storage System Vendors components that store, retrieve and delete Enterprise Vault content stream to and from a storage system.

As shown in the pictorial representation, the Vendor’s Streamer Plugin implements all the interface methods exposed by EV Streamer API. This facilitates Enterprise Vault, which in turn loads the Vendor’s Streamer Plugin DLL and calls appropriate method to read and write data to the Vendor’s Storage System. The Streamer storage could be presented as a primary or secondary storage in the Enterprise Vault Administration Console.

Worth noting here: There are other devices that also make use of EV Streamer API. Please check Compatibility Guide for supported devices as well as related blogs on Streamer API devices.

Troubleshooting Tips?

• Verify the correctness of Amazon S3 configuration parameters and/or Server connectivity issues by clicking the ‘Test’ button on Partition’s Advanced tab.
• Set LogLevel parameter to ‘Everything’ for verbose logging of Amazon S3 OST Plugin.
• Set Dtrace on StorageFileWatch.exe to monitor migration activity (data upload and empty cab deletion).
• Set Dtrace on EVStgOfflineOps.exe to monitor retrieval or restore activity.
• Set Dtrace on StorageManagment.exe to validates the configured settings and monitor ‘Test’ button functionality.
• For further refinement, set Dtrace filter on “OST Streamer” keyword to view Amazon S3 OST Plugin specific log messages.

Troubleshooting Scenarios & Solutions:

• Scenario 1 – Amazon Region: Enterprise Vault version 10.0.4 onwards, when a customer choose a regions where Amazon Simple Storage Service stores the buckets, it should immediately reflect in the configuration page as well as in the configuration file (perf file). Just in case, if the new region changes, to store the buckets, are not reflected (for example from US region to Sydney region in the Advance configuration tab on EV), it may result in bucket to be created in US region only.

Solution: To solve such issues, one needs to make sure that the right configuration is entered in Enterprise Vault (Advanced tab of the vault store partition properties). Second, make sure you have right account\credential for the new region. Third, check the integrity and location of bucket by logging into Amazon S3 account via browser and verifying bucket and location.

• Scenario 2 – CURL Proxy Settings:  Many customers prefer to use and configure proxy with Amazon S3 in Enterprise Vault. The CURL proxy settings (with or without SSL) can be entered in Enterprise Vault in Advanced tab of the Vault Store Partition properties. If there are any issues in this, the way customer can troubleshoot is by narrowing it down if this is an EV configuration issue or in general network issue. In browser, one can type https://s3.amazonaws.com and check if it is hitting correctly with same proxy settings (as entered in the Advanced tab of the vault store partition properties). If results vary in both cases, then it may be a pointer as which part the issue is.

For example, in case if you get HTTP error 400 (For example something like this in dtrace:  EV:M     OST Streamer: [TID:7584] [Plugin] <= Recv header, 0000000026 bytes (0x0000001a)|0000: HTTP/1.1 400 Bad Request|), it’s worth to check who is throwing HTTP error 400 – configured proxy or the Amazon cloud (you can use wireshark or similar tool to find that).

Second, it’s worth checking which proxy type is used – .

What’s not supported?

• As of now, Enterprise Vault current implementation of Amazon S3 plugin, does not support SSE (Server side encryption).

Need more information? – Look for these Technote

• Configuration of Amazon S3 Migrator in enterprise vault - Here is a comprehensive document to describe the configuration settings, explanation and initial troubleshooting details: http://www.symantec.com/business/support/index?page=content&id=DOC5387

• Proxy issues: http://www.symantec.com/business/support/index?page=content&id=TECH206024

• Amazon S3 Migrator Configuration issue when clicking on the test button on the Advanced tab of Vault Store Partition: http://www.symantec.com/business/support/index?page=content&id=TECH201640

http://www.symantec.com/business/support/index?page=content&id=TECH206024 

Both these issues related to configuration, has been addressed in 10.0.4

• 10.0.4 updates: Regarding the issues\features addressed in 10.0.4, you can go through following technote: http://www.symantec.com/business/support/index?page=content&id=TECH200691

• Multi Region Support for Amazon Simple Storage Service (S3) store: Technote http://www.symantec.com/business/support/index?page=content&id=TECH203542

Hope you found this information useful. Comment if you want to know anything further on this topic. . Thanks for reading this blog and sharing your valuable feedback. 

GLOSSARY