Crawling job information from the career pages of multiple websites

[pranjalm04]

Hey guys, I am trying to builda crawler which will smartly navigate through the urls which takes to job listing page and later on the individual job page which has all the information regarding a specific job posting. My question is how do i configure crawler or do a work around so that it know if the page is a job listingpage. Are there any strategies inbuilt in this module, which can identify it
Once it identifies the job listing page i want to go on individual url for the job posting and then extract data using llm in structured format. Since i am doing this for almost more than 400 websites i want a generalized solution