Seed URLs List
Domain Processor Control
Running
Background Processor:
Active and monitoring for new domains
Check Interval:
30
seconds
Domain Processor: This service automatically
monitors the TempSeedUrl table for new domains. When a new domain
is detected, it validates the website, checks if it's active, uses
the region from TempSeedUrl, analyzes JavaScript usage, and
inserts valid domains into the GenericWebsite table.
Pending domains: 3233
Auto-restart enabled - processor will automatically start when new domains are added.
Pending domains: 3233
Auto-restart enabled - processor will automatically start when new domains are added.
Total Websites
23067
Crawled
604
JS Heavy
12276
Total PDFs
28443726
Generic Website Data
Status:
Region:
Date Range:
| Domain URL | Status | Region | Spider Name | Crawled | Last Crawl Ended | Total PDFs Found | JS Heavy | Actions |
|---|---|---|---|---|---|---|---|---|
|
https://www.avantorsciences.com/no/en/
|
DONE | NO | avantorsciences_spider | Yes |
2026-01-07
10:17:45
|
3738
|
No | |
|
https://www.3m.com.pe/3M/es_PE/inicio/mapa-del-sitio/
|
DONE | ES | 3m_spider | Yes |
2026-01-08
07:49:30
|
113
|
No | |
|
https://unite.ai
Query: "Sikkerhetsdatablad" "01/18/24"
|
NEW | AI | Not Set | No | Never | 0 | No | |
|
https://petreader.net
Query: "Sikkerhetsdatablad" "19/08/25"
|
NEW | IS | Not Set | No | Never | 0 | No | |
|
https://amartha.com
Query: "Sicherheitsdatenblatt" "20/01/25"
|
NEW | ID | Not Set | No | Never | 0 | Yes | |
|
https://msds.deco.akzonobel.com/
|
DONE | NL | akzonobel_spider | Yes |
2026-01-08
16:29:02
|
14675
|
No | |
|
https://www.3m.com.ni/3M/es_NI/inicio/mapa-del-sitio/
|
DONE | ES | 3m_spider | Yes |
2026-01-08
14:26:52
|
88
|
No |
Showing 23061
to 23067
of 23067 websites
Show: