# Web Scraper schema ERD

Table and field reference for the Fivetran Web Scraper connector.

## Table & Field Reference

### account_information

| Column | Type | Keys |
| --- | --- | --- |
| email | String | PK |
| first_name | String |  |
| last_name | String |  |
| page_credits | Int |  |

### sitemap

| Column | Type | Keys |
| --- | --- | --- |
| id | Long | PK |
| cron_day | String |  |
| cron_hour | Int |  |
| cron_minute | Int |  |
| cron_month | String |  |
| cron_timezone | String |  |
| cron_weekday | String |  |
| driver | String |  |
| name | String |  |
| page_load_delay | Int |  |
| proxy | Int |  |
| request_interval | Int |  |
| scheduler_enabled | Boolean |  |

### scraping_job

| Column | Type | Keys |
| --- | --- | --- |
| id | Long | PK |
| sitemap_id | Long | FK |
| custom_id | String |  |
| driver | String |  |
| jobs_empty | Int |  |
| jobs_executed | Int |  |
| jobs_failed | Int |  |
| jobs_scheduled | Int |  |
| page_load_delay | Int |  |
| request_interval | Int |  |
| scheduled | Int |  |
| scraping_duration | Int |  |
| status | String |  |
| stored_record_count | Int |  |
| test_run | Int |  |
| time_created | Instant |  |

### data_quality

| Column | Type | Keys |
| --- | --- | --- |
| id | Long | PK, FK |
| max_empty_pages_percent_expected | Int |  |
| max_empty_pages_percent_got | Int |  |
| max_empty_pages_percent_success | Boolean |  |
| max_failed_pages_percent_expected | Int |  |
| max_failed_pages_percent_got | Int |  |
| max_failed_pages_percent_success | Boolean |  |
| min_column_records_category_link_expected | Int |  |
| min_column_records_category_link_got | Int |  |
| min_column_records_category_link_href_expected | Int |  |
| min_column_records_category_link_href_got | Int |  |
| min_column_records_category_link_href_success | Boolean |  |
| min_column_records_category_link_success | Boolean |  |
| min_column_records_description_expected | Int |  |
| min_column_records_description_got | Int |  |
| min_column_records_description_success | Boolean |  |
| min_column_records_name_expected | Int |  |
| min_column_records_name_got | Int |  |
| min_column_records_name_success | Boolean |  |
| min_column_records_pagination_expected | Int |  |
| min_column_records_pagination_got | Int |  |
| min_column_records_pagination_href_expected | Int |  |
| min_column_records_pagination_href_got | Int |  |
| min_column_records_pagination_href_success | Boolean |  |
| min_column_records_pagination_success | Boolean |  |
| min_column_records_price_expected | Int |  |
| min_column_records_price_got | Int |  |
| min_column_records_price_success | Boolean |  |
| min_column_records_product_link_expected | Int |  |
| min_column_records_product_link_got | Int |  |
| min_column_records_product_link_href_expected | Int |  |
| min_column_records_product_link_href_got | Int |  |
| min_column_records_product_link_href_success | Boolean |  |
| min_column_records_product_link_success | Boolean |  |
| min_column_records_reviews_expected | Int |  |
| min_column_records_reviews_got | Int |  |
| min_column_records_reviews_success | Boolean |  |
| min_column_records_subcategory_link_expected | Int |  |
| min_column_records_subcategory_link_got | Int |  |
| min_column_records_subcategory_link_href_expected | Int |  |
| min_column_records_subcategory_link_href_got | Int |  |
| min_column_records_subcategory_link_href_success | Boolean |  |
| min_column_records_subcategory_link_success | Boolean |  |
| min_record_count_expected | Int |  |
| min_record_count_got | Int |  |
| min_record_count_success | Boolean |  |
| overall_data_quality_success | Boolean |  |

### problematic_url

| Column | Type | Keys |
| --- | --- | --- |
| url | String | PK |
| scraping_job_id | Long | PK, FK |
| type | String |  |

## Relationships

- scraping_job(sitemap_id) → sitemap(id)
- data_quality(id) → scraping_job(id)
- problematic_url(scraping_job_id) → scraping_job(id)