Record & Replay Regression Testing Public Preview

Record-and-replay testing helps you check that changes to your connector don't unintentionally change the data it delivers to the destination. Once you've debugged the connector and are satisfied with its output, record a run. Later, after changing the code, fixing a bug, or upgrading a dependency, replay the recording to compare the new output with the approved result and catch any regressions in that scenario.

Use it with the Connector SDK local tester (fivetran debug).

Use the @action decorator to mark source-reading functions, such as API requests or database queries. Record mode saves their inputs and results, together with the expected destination output and connector state, in a fixture. Replay mode returns the recorded source data instead of calling the live source, then compares the new connector output with the approved result. If the data your connector delivers to the destination or the connector state differs from the recording, we report a regression.

We recommend recording a scenario after your connector is ready and works as expected. Then replay it each time you refactor the connector, upgrade a dependency, change how you transform data, or add a feature, to confirm that the data your connector delivers stays the same.

How record-replay regression testing works

The recording is the "golden file" to compare subsequent replays against to find regressions. Make sure to record after you are satisfied with your connector code and you want to preserve its output for particular scenarios.

1. Mark the functions that read from your source with the @action decorator.
2. Record a scenario with fivetran debug --record <scenario>. A scenario is a named recording of one connector run, for example multi_page_pagination. Each scenario protects one code path, and you can record as many as you need. The connector runs against the live source. We save every @action call with its method arguments and result, and the warehouse.db and state.json files the run produced.
3. Replay the scenario with fivetran debug --replay <scenario>. The connector runs again from the same starting point where the recording started. Each @action call returns the recorded result instead of calling the source.
4. We compare the replay's state.json and warehouse.db with the recorded ones. If anything differs, we detect a regression and list the differences.

Because a replay doesn't call the source, it's fast, doesn't use your API quota, and doesn't depend on source data that changes between runs.

Mark source calls with @action

Import action from fivetran_connector_sdk and decorate every function that reads from your source, such as an API request or a database query. Only the calls you decorate are recorded.

import requests

from fivetran_connector_sdk import Connector, Operations as op, action


@action("customers_page", match=lambda page: {"page": page})
def fetch_customers_page(page):
    response = requests.get(
        "https://api.example.com/customers",
        params={"page": page, "per_page": 10},
        timeout=30,
    )
    response.raise_for_status()
    return response.json()


def schema(configuration: dict):
    return [{"table": "customer", "primary_key": ["id"]}]


def update(configuration: dict, state: dict):
    page = state.get("last_page", 0) + 1
    while True:
        payload = fetch_customers_page(page)
        if not payload["data"]:
            break
        for item in payload["data"]:
            yield op.upsert(table="customer", data=item)
        if page >= payload["total_pages"]:
            break
        page += 1
    yield op.checkpoint(state={"last_page": page})


connector = Connector(update=update, schema=schema)

When you aren't recording or replaying, @action calls your function unchanged. In a deployed connection, it has no effect, so you can leave it in your production code.

Use decorator parameters

The @action decorator has two parameters:

Choose what identifies a call with match

During a replay, we look up the recorded result by the call's name and its matched arguments. If a call has an argument that changes on every run, such as a timestamp or a request ID, the replay can't find the recorded call. Use match to return only the arguments that identify the call.

import uuid

import requests
from fivetran_connector_sdk import action


@action("orders", match=lambda since, request_id: {"since": since})
def fetch_orders(since, request_id):
    response = requests.get(
        "https://api.example.com/orders",
        params={"since": since},
        headers={"X-Request-Id": request_id},
    )
    return response.json()


# request_id is different on every run, so it's left out of match
orders = fetch_orders("2026-01-01", str(uuid.uuid4()))

match receives the same arguments as the decorated function, including default values for arguments the call leaves out. It returns a dictionary of the arguments to store and match on. It behaves the same whether the call passes arguments by position or by keyword.

The arguments that match returns must also tell your calls apart. If several calls share the same name and the same matched arguments, for example when you poll the same endpoint twice, a replay can't distinguish them and returns the first recorded result for each. If the source returned different results for those calls, the replay then differs from the recording. To avoid this, include an argument that differs between the calls, such as a page number or cursor.

Record a scenario

A scenario is a named recording. A scenario name can contain only letters, digits, hyphens, and underscores. Give each scenario a descriptive name, such as multi_page_pagination or incremental_with_deletes. Record one scenario for each code path you want to protect.

Run the following command from your project folder:

fivetran debug --record <scenario> [--max-records <number>]

For example, to record a scenario named basic with the default limit of 100 write operations, run:

fivetran debug --record basic

--max-records is optional. It limits how many write operations (op.upsert, op.update, op.delete) your connector executes while the sync runs. The limit is a single total across all tables, not a limit for each table. After the limit, we stop the sync, and the recording contains only what ran up to that point. If you omit it, the limit is 100 write operations. For more information, see Limit the size of a recording with --max-records.

To record a scenario named basic with a limit of 50 write operations, run:

fivetran debug --record basic --max-records 50

If your connector needs configuration, pass it the same way you do for a normal fivetran debug run. For more information, see Local testing.

(Optional) Overwrite an existing recording

A scenario exists when fixtures/<scenario>/record_data/fixture.jsonl exists. When you record it again, we ask you to confirm, because recording clears all existing data for that scenario.

You can't use --yes and --non-interactive together.

(Optional) Limit the size of a recording with --max-records

A recording of a full historical sync can be large. Use --max-records to stop the recording after a set number of write operations.

fivetran debug --record basic --max-records 50

--max-records must be a positive integer, and you can use it only with --record. If you omit it, the limit is 100. Keep scenarios small, because a small recording is faster to replay and easier to review.

How --max-records works

• We count every op.upsert, op.update, or op.delete call. The count is one total for the whole sync, not a limit for each table, so --max-records 50 allows 50 write operations across all tables together.
• The first N write operations go through. The next write stops your update function, and we treat that as a successful end of the sync.
• We save N in the first line of fixture.jsonl as capped_count. A replay reads it and applies the same limit, so it stops at the same write as the recording. You can't pass --max-records to --replay.
• After the limit is reached, new @action calls also stop, so a worker thread that is still running can't make more source calls.

What happens during a record run

1. If the scenario already exists, we ask before overwriting it. See Overwrite an existing recording.
2. We take a copy of your project's files/state.json and files/warehouse.db as the starting point of the scenario. If there's no state.json, we start with an empty {}.
3. We run your connector against the live source, starting from that state. The run writes to the scenario folder, fixtures/<scenario>/ in your project, not to your files folder, so your local debug data isn't changed.
4. Each @action call is appended to fixtures/<scenario>/record_data/fixture.jsonl as it happens.
5. When the run finishes, we keep the output in fixtures/<scenario>/ as the reference for replays. For the full layout, see What a recording contains.

If the run succeeds but your connector made no @action calls, we keep the output and log a warning that the recording is empty. In that case, a replay has nothing to answer from.

We lock the scenario folder while a record or replay run uses it. If you start a second run of the same scenario, it fails right away with a message that another run is in progress. Wait for the first run to finish, then try again.

What a recording contains

We save each recording in fixtures/<scenario>/ in your project:

fixtures/<scenario>/
├── record_start/
│   ├── warehouse.db
│   └── state.json
├── record_data/
│   └── fixture.jsonl
└── record_end/
    ├── warehouse.db
    └── state.json

The first line of fixture.jsonl is a header with the format version and the --max-records limit. Each following line holds one call:

{"__fixture_header__": {"fixture_version": "1.0", "capped_count": 50}}
{"interaction": "customers_page", "arguments": {"page": 1}, "result": {"data": [], "total_pages": 1}}

Credential masking in recordings

Before we save a call, we replace the value of any argument whose name ends in api_key, api-key, apikey, token, secret, password, credential, configuration, or configurations with a sha256: digest. Names are matched without regard to case. The digest keeps calls with different credentials distinct without storing the credential.

Data must be JSON-serializable

We store each call in fixture.jsonl as JSON. The arguments (or the dictionary that match returns) and the result can contain only dictionaries, lists, strings, numbers, booleans, and None. If an argument or result isn't JSON-serializable, such as bytes or a custom object, the recording fails with an error that names the interaction. To fix it:

• For a result, convert the value inside the @action function before you return it. For example, return response.json() instead of the response object.
• For an argument, use match to return only JSON-safe values.

Replay a scenario

After you change your connector code, replay the scenario to check that the change didn't alter the data your connector delivers. Run the following command from your project folder:

fivetran debug --replay <scenario>

For example, to replay the scenario named basic:

fivetran debug --replay basic

Run record and replay from Python code

You don't need the fivetran command to record or replay. If you run your connector from a script or an IDE, pass the same options to connector.debug(). This is the same approach as Option 2 in Local testing.

if __name__ == "__main__":
    # Record scenario "basic", stopping after 50 write operations.
    connector.debug(configuration={}, record="basic", max_records=50)

    # Replay scenario "basic".
    # connector.debug(configuration={}, replay="basic")

Pass project_path if you run the script from outside the project folder. If the replay doesn't match, debug() logs the differences and exits with code 1, so you can use it in a test script or a CI job.

Run regression tests in CI

Replay needs no access to your source, so you can run it in any CI job. Commit your fixtures folder after you review it, install the Connector SDK, and run one replay for each scenario:

fivetran debug --replay basic
fivetran debug --replay incremental

A non-zero exit code means the replay failed. Check the logs to tell output differences from recording, configuration, or runtime errors. If the output differs, record the scenario again only when you intend that change.

What happens during a replay

1. We recreate .fixtures_replay/<scenario>/ from record_start, and add .fixtures_replay/ to your .gitignore file if it isn't already there. We delete and rebuild the folder on every replay, and you shouldn't commit it.
2. We apply the --max-records limit saved in the recording.
3. We run your connector from the recorded starting state. Each @action call returns the recorded result. The source isn't called.
4. We compare the replay's state.json and warehouse.db with record_end.
5. The command succeeds only if both files match. Your project's files folder and the recording are never changed.

If your connector makes an @action call that isn't in the recording, the replay fails and tells you to record the scenario again. When more than one recorded call has the same name and arguments, the replay returns the first one. This includes a function with no arguments, because nothing tells its calls apart, so every call to it returns the first recorded result.

What's compared in regression testing

State

We compare state.json key by key. We report each key that's missing from the replay, unexpected in the replay, or has a different value.

Warehouse data

We compare the two warehouse.db files table by table and row by row. We open both files read-only, so the comparison never changes them.

We follow these rules:

• We match rows by primary key. If a table has no primary key, we match rows by the _fivetran_id column.
• We never compare _fivetran_synced and _fivetran_index, because their values change between runs.
• We compare values exactly. NULL equals only NULL. An empty string, a different capital letter, extra whitespace, or a small difference in a floating-point number counts as a difference.
• We compare nested values, such as lists, structs, maps, and JSON, as a whole.
• If a column's data type differs, we report it once and don't compare its values. If a primary key column's type differs, we don't compare the rows of that table.

Read the output

Replay warehouse.db does not match the recording:
table 'tester.customer': row {'id': '024ea5e9-b604-4342-a496-f5255fd03bbc'} column 'name': record_end='Helen Lewis DDS' replay='Helen Lewis'
table 'tester.customer': column 'job' present in recorded data but missing during replay
2 mismatch(es) reported

We print at most 100 findings, and the last line shows the total, for example 120 mismatch(es) reported (showing first 100).

If the differences are a change you intended, such as a new column or a new transformation, record the scenario again, and commit the new recording with your code change. If you didn't intend to change the output, the differences are a regression. Fix your code and replay again.

Examples

This example records and replays a connector that reads customers from the Fivetran API Playground, with its source calls decorated with @action. Use one of the runnable examples in the regression_test folder:

• cli_example runs record and replay from the command line.
• ide_example runs them from your IDE.

Troubleshooting