Google Cloud Storage API Configuration
For more information about authentication, accessing the Fivetran REST API, and request and response formats, see Getting Started with the Fivetran REST API.
Request
POST https://api.fivetran.com/v1/connections
{ "group_id": "group_id", "service": "gcs", "trust_certificates": true, "trust_fingerprints": true, "run_setup_tests": true, "paused": false, "pause_after_trial": false, "sync_frequency": 1440, "data_delay_sensitivity": "LOW", "data_delay_threshold": 0, "daily_sync_time": "14:00", "schedule_type": "auto", "connect_card_config": { "redirect_uri": "https://your.site/path", "hide_setup_guide": true, "all_fields": true }, "proxy_agent_id": "proxy_agent_id", "private_link_id": "private_link_id", "networking_method": "Directly", "hybrid_deployment_agent_id": "hybrid_deployment_agent_id", "destination_configuration": { "virtual_warehouse": "virtual_warehouse" }, "destination_schema_names": "FIVETRAN_NAMING", "external_secrets_manager_id": "esm_id", "data_checks": true, "config": { "append_file_option": "upsert_file_with_primary_keys | upsert_file | append_file", "auth_type": "FIVETRAN_SERVICE_ACCOUNT | CUSTOM_SERVICE_ACCOUNT", "bucket": "bucket_name", "compression": "zip | gz | tar_gz | tar | infer | gzip | uncompressed | bz2 | tar_bz2", "delimiter": "string", "empty_header": true, "escape_char": "string", "extraction_pattern": "(?<table>\\w+)/.*", "file_mapping_method": "EXTRACT_TABLES | DEFINE_PER_TABLE", "file_type": "tsv | xml | csv | spreadsheet | json | jsonl | avro | parquet", "files": [ { "archive_pattern": "string", "file_pattern": "string", "selected_range": "'Sheet1'!A1", "table_name": "string" } ], "json_delivery_mode": "Packed | Unpacked", "line_separator": ";", "non_standard_escape_char": true, "null_sequence": "string", "on_error": "fail | skip", "pgp_pass_phrase": "<pass-phrase-used>", "pgp_secret_key": "<secret-key-used>", "prefix": "files/exports", "quote_char": "string", "quote_character_enabled": true, "secret_key": "secret_key", "selected_range": "'Sheet1'!A1", "signer_public_key": "<signer-public-key-used>", "skip_after": 0, "skip_before": 0, "sync_inconsistent_rows": true, "use_pgp_encryption_options": true, "schema": "schema_name", "table_group_name": "string" } }
Config parameters
| Name | Description |
|---|---|
append_file_option | If you know that the source completely over-writes the same file with new data, you can append the changes instead of upserting based on filename and line number. |
auth_type | Authorization type. Required for storage bucket authentication. |
bucket | The name of the GCS bucket. |
compression | The secrets that should be passed to the function at runtime. |
delimiter | You can specify your the delimiter that your CSVs use here. Fivetran generally tries to infer the delimiter, but in some cases this is impossible. |
empty_header | Optional. If your CSV generating software doesn't provide header line for the documents, Fivetran can generate the generic column names and sync data rows with them. |
escape_char | If your CSV generator follows non-standard rules for escaping quotation marks, you can set the escape character here. |
extraction_pattern | Required when file_mapping_method = EXTRACT_TABLES. A regular expression with a named capture group (?<table>...) to extract table names from file paths. |
file_mapping_method | The method used to map files to destination tables. Use DEFINE_PER_TABLE to manually configure each table with its own file pattern, or EXTRACT_TABLES to automatically discover tables based on a single extraction pattern. |
file_type | If your files are saved with improper extensions, you can force them to by synced as the selected file type. |
files | Required when file_mapping_method = DEFINE_PER_TABLE. Mapped file name patterns to a destination table. |
files[0].archive_pattern | All Files inside of compressed archives with filenames matching this regular expression will be synced. |
files[0].file_pattern | All files in your search path matching this regular expression will be synced per table. |
files[0].selected_range | Cell reference that will be used to sync all contiguous data starting from the top-left cell in all the spreadsheets matching the name. Cell reference is in the '<sheetName>'!<startColumnName><startRowNumber> format. |
files[0].table_name | All files matching the above pattern will be synced to this table. |
json_delivery_mode | Control how your JSON data is delivered into your destination |
line_separator | You can specify the custom line separator for your CSV files. The line separator is used in files to separate one row from the next. |
non_standard_escape_char | Use this if your CSV generator uses non-standard ways of escaping characters. Default value: false. |
null_sequence | If your CSVs use a special value indicating null, you can specify it here. |
on_error | If you know that your files contain some errors, you can choose to have poorly formatted lines skipped. We recommend leaving the value as fail unless you are certain that you have undesirable, malformed data. |
pgp_pass_phrase | The PGP passphrase used to create the key. Must be populated if use_pgp_encryption_options is set to true. |
pgp_secret_key | The contents of your PGP secret key file. Must be populated if use_pgp_encryption_options is set to true. |
prefix | The connection will look for files in this folder path. Choose the lowest common folder in a folder hierarchy that includes all the files you want to sync. This defines a specific location where Fivetran scans for files and helps ensure optimal performance. |
quote_char | You can specify the enclosing character used in your CSVs here. Fivetran will consider " as the quote character if this field is empty and quote_character_enabled is set to true. |
quote_character_enabled | If you don’t want to use an enclosing character while processing the files, set the value to false. By default, the value is set to true and is considered " as quote character. |
secret_key | Your JSON Private Key. Used to authorize service account. Required if you use a Custom Service Account to authenticate the storage bucket. |
selected_range | Cell reference that will be used to sync all contiguous data starting from the top-left cell in all the spreadsheets matching the name. Cell reference is in the '<sheetName>'!<startColumnName><startRowName> format. |
signer_public_key | The contents of the signer's public key file. This is optional — if you don't provide the key, we decrypt the files without verifying their signatures. |
skip_after | We will skip over the number of lines specified at the end so as to not introduce aberrant data into your destination. |
skip_before | We will skip over the number of lines specified before syncing data. |
sync_inconsistent_rows | When enabled, Fivetran syncs rows with fewer or more columns than the header row. For rows with fewer columns, Fivetran adds NULL values for the missing columns. For rows with extra columns, Fivetran names them column_<position>. When disabled, Fivetran handles inconsistent rows according to your error-handling settings. |
use_pgp_encryption_options | Set to true if files are encrypted using PGP in the GCS bucket. Default value: false. |
schema (required) | Destination schema name. Schema name is permanent and cannot be changed after connection creation |
table_group_name (required) | Combined with the schema to form the Fivetran connection name <schema>.<table_group_name>. Lets you create multiple Merge Mode connections per schema. |
Authorization
There are two ways to authorize this connector type:
- By specifying the values for the
bucket,auth_typeandsecret_keyparameters in the request.POST https://api.fivetran.com/v1/connections{ "service": "gcs", "group_id": "group_id", "config": { "bucket": "bucket_name", "auth_type": "FIVETRAN_SERVICE_ACCOUNT | CUSTOM_SERVICE_ACCOUNT", "secret_key": "secret_key" } } By using the Connect Card or the Fivetran dashboard.