Why Did Target Kafka Time Change After Adding a Target Agent Machine?

Question

After I switched an Integrate job from running on an HVR Hub in EST to a new target agent machine running in UTC, timestamp columns in the target no longer include the expected time zone offset.

For example:

How does HVR handle time zone conversion for timestamp data, and why did the Kafka target value change?

Environment

• HVR 6
• Context: Oracle source, HVR Hub, HVR Agent, Kafka, PostgreSQL or S3 target
• Migration event: HVR Hub server migration and HVR Agent introduction

Answer

We do not usually change date or time values that do not include time zone information. We replicate these values as they appear in the source, unless the target system requires the values to be stored as UTC epoch timestamps.

Kafka stores date and time fields as Unix timestamps in milliseconds since the epoch, in UTC. During integration, we must therefore convert incoming date and time values to UTC. We perform this conversion by using the time zone of the machine where the Integrate process runs.

Why the Kafka timestamp changed

In the original setup, the HVR Hub runs in EST and serves as the target agent. When you introduce a new target agent configured in UTC, we use that agent's time zone for all date and time conversions:

If you set the TZ environment variable in the target hub or target agent, it changes the target Kafka date and time. For example, if the target agent HVR service has TZ set to EST in the startup process:

If the target agent HVR service has TZ set to PST in the startup process, we convert timestamps using the PST offset, which changes the target Kafka time: