Logging
Logging enables observability and debugging for connector execution.
Required import
# For enabling Logs in your connector code
from fivetran_connector_sdk import Logging as log
Logging levels
| Level | Purpose |
|---|---|
| FINE | Debugging and development. Used only during debugging. |
| INFO | General status updates. |
| WARNING | Potential non-critical issues. |
| SEVERE | Critical errors. |
Example
log.fine("Debugging data transformation.")
log.info("Connector started successfully.")
log.warning("Response time slower than expected.")
log.severe("Failed to connect to data source.")
log.severe("Failed to connect to the data source.", e) # Error can be logged using this parameter in severe method.
Logging methods except severe accepts only one argument. Errors can be logged using a second argument with the severe log level.
Exception handling
Python Runtime Errors implemented in connector.py file, when raised after deployment, appear in the dashboard as an error and trigger an alert.
Do not use exit() to terminate Python code because connector can become stuck. Instead, raise error or catch exception and display error using log.severe(), which is then displayed in sync logs.
To fail sync by throwing exception, use any RuntimeError. This will trigger an error in the dashboard with the title Python code throwing error.
raise RuntimeError(f"Value not expected. Value: {value}")To handle error, use try-catch and log it with
log.severe("log_message", e). This handles error and continues processing remaining code. Use this only if this exception is not impacting your sync in a critical manner.try: // Some Code catch exception as e: log.severe("Value not expected.", e)
Notes
- INFO logs are limited to 1500/min. Any logs that exceed this rate are dropped.
- Avoid excessive logging. For example, avoid placing sync log event for each record, as it can increase log volume and cause logs to be discarded due to rate limits. You can check
weather example, which uses info and fine-level logging inupdate()method for reference. - FINE logs are only generated when running
fivetran debugcommand during debugging. Leaving FINE logs in code is safe, but they don't appear in production logs.
Accessing Connector SDK logs
You can access your Connector SDK connections' log events by:
- Navigating to the Connector SDK logs tab of your connection details page in the Fivetran dashboard.
- Using the Fivetran Platform Connector. Your connections' logs are available in the
CONNECTOR_SDK_LOGtable within the associated Fivetran destination. - Using external log services, previously configured for your destination.
Connector SDK logs provide in-depth event data, including timestamps, log levels, and messages. We prefix all Fivetran process logs with the Fivetran-Platform identifier, and logs from the fivetran-connector-sdk library are prefixed with the Fivetran-Connector-sdk identifier.
Local tester logs flow
When running Fivetran's Local Tester, fivetran debug logs are printed from two asynchronous sources: the logs from your connector.py code and the logs from the Local Tester.
- Local tester logs appear asynchronously, prefixed with
Fivetran-Tester-Process. - Library logs are prefixed with
Fivetran-Connector-sdk. - Long syncs print progress summaries every 5 minutes.
Local tester logs details
Expand to see the Local tester details
The Fivetran Local Tester methods are not blocking calls, so your `connector.py` code continues to execute after we enqueue the instruction. Because the execution of your connector's code and Fivetran’s Local Tester are not synchronous, the logs related to the Local Tester are also not synchronous. The logs from your `connector.py` code are printed immediately whereas the logs from the Local Tester are printed when the corresponding operations have been executed by the tester. To help differentiate the Local Tester and Library logs from your connection logs, the Local Tester logs are prefixed with the `Fivetran-Tester-Process` identifier, and logs from the `fivetran-connector-sdk library` prefixed with the `Fivetran-Connector-sdk` identifier.Furthermore, for connection tests with a large volume of data, the Local Tester process might take a long time to complete, longer than the connector.py code takes to complete. For this, a periodic log displaying counts of different operations completed is printed every 5 min.
See the following example of such a log:
Dec 19, 2024 01:41:52 AM Fivetran-Tester-Process: INFO: Sync Progress
Operation | Call Count
----------------|------------
Upserts | 4
Updates | 0
Deletes | 0
Truncates | 0
SchemaChanges | 2
Checkpoints | 3
Log example
Expand to see log example
Dec 18, 2024 10:04:19 PM Fivetran-Tester-Process: INFO: Configuration:
{}
Dec 18, 2024 10:04:19 PM Fivetran-Tester-Process: INFO: Previous state:
{}
Dec 18, 2024 10:04:19 PM Fivetran-Tester-Process: INFO: [SchemaChange]: tester.company
Dec 18, 2024 10:04:19 PM Fivetran-Tester-Process: INFO: [CreateTable]: tester.company
Dec 18, 2024 10:04:19 PM Fivetran-Tester-Process: INFO: [SchemaChange]: tester.department
Dec 18, 2024 10:04:19 PM Fivetran-Tester-Process: INFO: [CreateTable]: tester.department
Dec 18, 2024 10:04:19 PM WARNING: Example: Common Patterns For Connectors - Cursors - Multiple Tables With Cursors
Dec 18, 2024 10:04:19 PM INFO: Upserting
Dec 18, 2024 10:04:30 PM INFO: Checkpointing
Dec 18, 2024 10:04:35 PM INFO: Upserting
Dec 18, 2024 10:04:35 PM Fivetran-Tester-Process: INFO: Checkpoint: {"company_cursor": "2024-08-14T01:00:00Z"}
Dec 18, 2024 10:04:45 PM INFO: Checkpointing
Dec 18, 2024 10:04:55 PM INFO: Upserting
Dec 18, 2024 10:04:55 PM Fivetran-Tester-Process: INFO: Checkpoint: {"company_cursor": "2024-08-14T01:00:00Z", "department_cursor": {"1": "2024-08-14T01:00:00Z"}}
Dec 18, 2024 10:05:05 PM INFO: Checkpointing
Dec 18, 2024 10:05:15 PM INFO: Upserting
Dec 18, 2024 10:05:15 PM Fivetran-Tester-Process: INFO: Checkpoint: {"company_cursor": "2024-08-14T01:00:00Z", "department_cursor": {"1": "2024-08-14T02:00:00Z"}}
Dec 18, 2024 10:05:25 PM INFO: Checkpointing
Dec 18, 2024 10:05:25 PM Fivetran-Tester-Process: INFO: Sync Progress
Operation | Call Count
--------------------+------------
Upserts | 4
Updates | 0
Deletes | 0
Truncates | 0
SchemaChanges | 2
Checkpoints | 3
Dec 18, 2024 10:05:35 PM INFO: Upserting
Dec 18, 2024 10:05:35 PM Fivetran-Tester-Process: INFO: Checkpoint: {"company_cursor": "2024-08-14T01:00:00Z", "department_cursor": {"1": "2024-08-14T03:00:00Z"}}
Dec 18, 2024 10:05:45 PM INFO: Checkpointing
Dec 18, 2024 10:05:50 PM INFO: Upserting
Dec 18, 2024 10:05:50 PM Fivetran-Tester-Process: INFO: Checkpoint: {"company_cursor": "2024-08-14T02:01:00Z", "department_cursor": {"1": "2024-08-14T03:00:00Z"}}
Dec 18, 2024 10:06:00 PM INFO: Checkpointing
Dec 18, 2024 10:06:10 PM INFO: Upserting
Dec 18, 2024 10:06:10 PM Fivetran-Tester-Process: INFO: Checkpoint: {"company_cursor": "2024-08-14T02:01:00Z", "department_cursor": {"1": "2024-08-14T03:00:00Z", "2": "2024-08-14T01:00:00Z"}}
Dec 18, 2024 10:06:20 PM INFO: Checkpointing
Dec 18, 2024 10:06:30 PM INFO: Upserting
Dec 18, 2024 10:06:30 PM Fivetran-Tester-Process: INFO: Checkpoint: {"company_cursor": "2024-08-14T02:01:00Z", "department_cursor": {"1": "2024-08-14T03:00:00Z", "2": "2024-08-14T02:00:00Z"}}
Dec 18, 2024 10:06:30 PM Fivetran-Tester-Process: INFO: Sync Progress
Operation | Call Count
--------------------+------------
Upserts | 7
Updates | 0
Deletes | 0
Truncates | 0
SchemaChanges | 2
Checkpoints | 7
Dec 18, 2024 10:06:40 PM INFO: Checkpointing
Dec 18, 2024 10:06:50 PM Fivetran-Tester-Process: INFO: Checkpoint: {"company_cursor": "2024-08-14T02:01:00Z", "department_cursor": {"1": "2024-08-14T03:00:00Z", "2": "2024-08-14T03:00:00Z"}}
Dec 18, 2024 10:06:50 PM Fivetran-Tester-Process: INFO: Sync Progress
Operation | Call Count
--------------------+------------
Upserts | 16
Updates | 0
Deletes | 0
Truncates | 0
SchemaChanges | 2
Checkpoints | 16
Dec 18, 2024 10:06:50 PM Fivetran-Tester-Process: INFO: Sync SUCCEEDED