How to Replicate XML Files from Oracle to AWS S3

Question

How can I replicate XML files from an Oracle Linux server to AWS S3 bucket using HVR?

I need to replicate audit logs (XML files) that reside on Oracle Linux server to AWS Redshit/S3 bucket. Does HVR support moving files to (Amazon) AWS S3 bucket?

Environment

HVR 5

Answer

HVR is indeed capable of moving files between different locations, including S3.

The configuration is really straightforward. All you need is a simple channel setup with a "source" file type location (this would be your Linux server) and an S3 target location, with Capture and Integrate actions defined on them, respectively.

An example of a setting up a source remote file location on server orahost with an HVR agent.

NOTE: If the agent is not installed on your source machine, you can use SFTP to access your files.

After setting up the channel, HVR will simply copy any files it finds in the directory of the file location to the target location. You can control which files you want HVR to handle using the /Pattern and /IgnorePattern parameters of the Capture action.

FAQ

Question 1

We are estimating 10K to 14K audit files per day (all together ~15 to 20GB). Can HVR handle this many files? In other words, do we need any additional resources to be added to the HVR hub side?

Answer

It should not be an issue if you have enough bandwidth between the source and your hub, and the hub and AWS. If you are expecting the number of files to be replicated being on the higher side, having a source location with an HVR agent installed is a recommended best practice. It provides compression and direct read access with little overhead to the files, which is beneficial to minimize latency on the capture side.

Question 2

We already have an agent (HVR remote listener agent) installed on source database servers as we already have Oracle tables replication setup from both these database servers. Will the same agent work for XML files replication from these database servers to send over to the S3 target location in AWS?

Answer

Yes, you can use the same agent.

NOTE: The scope of this KB is only to describe how to move the files to an existing AWS Redshift/S3 instance.

For additional information, see AWS documentation for configuring Redshift.

NOTE: Security groups, keys, and SSL certificates need to be in place for seamless communication.