Context Layer Setup Guide Public Preview

Fivetran uses gemini-embedding-001 for BigQuery. The model must use the project's default Cloud resource connection in the same location as the AGENTS dataset. Creating a remote model through CONNECTION DEFAULT as an administrator creates or configures that connection and grants its Google-managed service account the required Vertex AI access.

The AGENTS dataset must already exist. Run indexing once to create it, or create it in the same location as the destination data before continuing.

Replace <project_id> and <fivetran_worker_service_account> below. Run the commands as an administrator who can configure the BigQuery default connection and update project IAM. Ensure that the BigQuery Connection API and Vertex AI API are enabled in the project.

1. Enable the required APIs

gcloud services enable \
  bigqueryconnection.googleapis.com \
  aiplatform.googleapis.com \
  --project=<project_id>

2. Bootstrap the default connection and remote model

bq query --project_id=<project_id> --nouse_legacy_sql \
  'CREATE MODEL IF NOT EXISTS `<project_id>.AGENTS.ai_ready__emb__gemini_embedding_001`
   REMOTE WITH CONNECTION DEFAULT
   OPTIONS (ENDPOINT = "gemini-embedding-001")'

This is a safe no-op after the model and default connection have been configured successfully. The administrator running it typically needs BigQuery Admin and Project IAM Admin permissions; see BigQuery connection management for the exact permissions.

The initial model is a bootstrap vehicle for configuring the connection. The indexing pipeline owns its lifecycle after setup.

3. Let the Fivetran worker use the connection

Grant the Fivetran worker service account BigQuery Connection User. This project-level command lets the worker use Cloud resource connections in the project:

gcloud projects add-iam-policy-binding <project_id> \
  --member="serviceAccount:<fivetran_worker_service_account>" \
  --role="roles/bigquery.connectionUser" \
  --condition=None

The worker also needs its normal destination permissions to create and inspect models in AGENTS. It does not need permission to read or change project IAM. After this bootstrap, the pipeline refers to the system-managed default connection explicitly and creates the model if it is missing.

For background, see BigQuery default connections and BigQuery text embedding generation.

Fivetran uses OpenAI text-embedding-3-large. A customer administrator must expose that model through a Databricks external Model Serving endpoint so ai_query() can embed queries without routing them through Fivetran. The workspace must support Unity Catalog, serverless compute, AI Search, and ai_query().

Replace <secret_scope> and <secret_key> below. The endpoint must be named ai_ready__emb__text_embedding_3_large.

1. Store the OpenAI API key

databricks secrets create-scope <secret_scope>
databricks secrets put-secret <secret_scope> <secret_key>

2. Create the external embedding endpoint

Run this in a Databricks notebook authenticated as an administrator:

%pip install "mlflow[genai]>=2.9.0"

import mlflow.deployments

client = mlflow.deployments.get_deploy_client("databricks")
client.create_endpoint(
    name="ai_ready__emb__text_embedding_3_large",
    config={
        "served_entities": [{
            "name": "ai_ready__emb__text_embedding_3_large",
            "external_model": {
                "name": "text-embedding-3-large",
                "provider": "openai",
                "task": "llm/v1/embeddings",
                "openai_config": {
                    "openai_api_key": "{{secrets/<secret_scope>/<secret_key>}}"
                },
            },
        }],
    },
)

Grant the Fivetran destination principal CAN QUERY on this endpoint. It also needs its normal Unity Catalog write privileges, CREATE TABLE on each target schema, and permission to create or manage the fivetran_ai_search AI Search endpoint. The pipeline enables Change Data Feed on each vector table and creates a triggered Delta Sync index over the customer-provided embeddings.

For background, see Databricks external OpenAI model endpoints, Databricks ai_query, and Databricks AI Search setup.

Fivetran generates Snowflake document embeddings outside the warehouse with text-embedding-3-large. To let warehouse-connected agents embed query text with the same model, a Snowflake administrator must create customer-owned external-access objects. Creating only the external-access integration is insufficient because Snowflake requires it to reference an existing network rule and secret.

Replace <destination_database>, <destination_schema>, <destination_warehouse>, and <fivetran_destination_role> below. Supply the OpenAI API key through an approved secret-management workflow.

1. Create the network rule and secret

Run with a role that can create network rules and secrets in the AGENTS schema:

USE DATABASE <destination_database>;
CREATE SCHEMA IF NOT EXISTS AGENTS;

CREATE OR REPLACE NETWORK RULE AGENTS.FIVETRAN_OPENAI_NETWORK_RULE
  MODE = EGRESS
  TYPE = HOST_PORT
  VALUE_LIST = ('api.openai.com');

CREATE OR REPLACE SECRET AGENTS.OPENAI_API_KEY
  TYPE = GENERIC_STRING
  SECRET_STRING = '<customer-owned OpenAI API key>';

2. Create the external-access integration

Run with a role that has the account-level CREATE INTEGRATION privilege:

CREATE OR REPLACE EXTERNAL ACCESS INTEGRATION FIVETRAN_OPENAI_EXTERNAL_ACCESS_INTEGRATION
  ALLOWED_NETWORK_RULES = (<destination_database>.AGENTS.FIVETRAN_OPENAI_NETWORK_RULE)
  ALLOWED_AUTHENTICATION_SECRETS = (<destination_database>.AGENTS.OPENAI_API_KEY)
  ENABLED = TRUE;

3. Grant the destination role access

The role used by the Fivetran Snowflake destination must be able to create the UDF, read the secret, and use the integration:

GRANT USAGE ON DATABASE <destination_database> TO ROLE <fivetran_destination_role>;
GRANT USAGE ON SCHEMA <destination_database>.AGENTS TO ROLE <fivetran_destination_role>;
GRANT CREATE FUNCTION ON SCHEMA <destination_database>.AGENTS TO ROLE <fivetran_destination_role>;
GRANT CREATE CORTEX SEARCH SERVICE ON SCHEMA <destination_database>.<destination_schema>
  TO ROLE <fivetran_destination_role>;
GRANT USAGE ON WAREHOUSE <destination_warehouse> TO ROLE <fivetran_destination_role>;
GRANT READ ON SECRET <destination_database>.AGENTS.OPENAI_API_KEY TO ROLE <fivetran_destination_role>;
GRANT USAGE ON INTEGRATION FIVETRAN_OPENAI_EXTERNAL_ACCESS_INTEGRATION
  TO ROLE <fivetran_destination_role>;

If agents query Snowflake with a different role, grant that role USAGE on the database and schema. After the next indexing run creates the function, also grant function usage:

GRANT USAGE ON FUNCTION
  <destination_database>.AGENTS.ai_ready__emb__text_embedding_3_large(VARCHAR)
  TO ROLE <agent_query_role>;

For background, see Snowflake external network access setup, Snowflake CREATE SECRET, and OpenAI embeddings API.