Web Scraper

Explore the interactive ERD

Table & Field Reference

account_information

Column	Type	Keys
`email`	String	PK
`first_name`	String
`last_name`	String
`page_credits`	Int

sitemap

Column	Type	Keys
`id`	Long	PK
`cron_day`	String
`cron_hour`	Int
`cron_minute`	Int
`cron_month`	String
`cron_timezone`	String
`cron_weekday`	String
`driver`	String
`name`	String
`page_load_delay`	Int
`proxy`	Int
`request_interval`	Int
`scheduler_enabled`	Boolean

scraping_job

Column	Type	Keys
`id`	Long	PK
`sitemap_id`	Long	FK
`custom_id`	String
`driver`	String
`jobs_empty`	Int
`jobs_executed`	Int
`jobs_failed`	Int
`jobs_scheduled`	Int
`page_load_delay`	Int
`request_interval`	Int
`scheduled`	Int
`scraping_duration`	Int
`status`	String
`stored_record_count`	Int
`test_run`	Int
`time_created`	Instant

data_quality

Column	Type	Keys
`id`	Long	PKFK
`max_empty_pages_percent_expected`	Int
`max_empty_pages_percent_got`	Int
`max_empty_pages_percent_success`	Boolean
`max_failed_pages_percent_expected`	Int
`max_failed_pages_percent_got`	Int
`max_failed_pages_percent_success`	Boolean
`min_column_records_category_link_expected`	Int
`min_column_records_category_link_got`	Int
`min_column_records_category_link_href_expected`	Int
`min_column_records_category_link_href_got`	Int
`min_column_records_category_link_href_success`	Boolean
`min_column_records_category_link_success`	Boolean
`min_column_records_description_expected`	Int
`min_column_records_description_got`	Int
`min_column_records_description_success`	Boolean
`min_column_records_name_expected`	Int
`min_column_records_name_got`	Int
`min_column_records_name_success`	Boolean
`min_column_records_pagination_expected`	Int
`min_column_records_pagination_got`	Int
`min_column_records_pagination_href_expected`	Int
`min_column_records_pagination_href_got`	Int
`min_column_records_pagination_href_success`	Boolean
`min_column_records_pagination_success`	Boolean
`min_column_records_price_expected`	Int
`min_column_records_price_got`	Int
`min_column_records_price_success`	Boolean
`min_column_records_product_link_expected`	Int
`min_column_records_product_link_got`	Int
`min_column_records_product_link_href_expected`	Int
`min_column_records_product_link_href_got`	Int
`min_column_records_product_link_href_success`	Boolean
`min_column_records_product_link_success`	Boolean
`min_column_records_reviews_expected`	Int
`min_column_records_reviews_got`	Int
`min_column_records_reviews_success`	Boolean
`min_column_records_subcategory_link_expected`	Int
`min_column_records_subcategory_link_got`	Int
`min_column_records_subcategory_link_href_expected`	Int
`min_column_records_subcategory_link_href_got`	Int
`min_column_records_subcategory_link_href_success`	Boolean
`min_column_records_subcategory_link_success`	Boolean
`min_record_count_expected`	Int
`min_record_count_got`	Int
`min_record_count_success`	Boolean
`overall_data_quality_success`	Boolean

problematic_url

Column	Type	Keys
`url`	String	PK
`scraping_job_id`	Long	PKFK
`type`	String