Here's what I want to achieve:
class Hello(Spider): #some stuff def parse(self, response): #get a list of url of cities using pickle and store in a list #Now for each city url I have to get list of monuments (using selenium) which is achieved by the below loops for c in cities: #get the list of monuments using selenium and iterate through each monument url contained in the division divs = sel.xpath('some xpath/div') for div in divs: monument_url=''.join(div.xpath('some xpath')) #For each monument url get the response and scrape the information yield Request(monument_url, self.parse_monument) def parse_monument(self, response): #scrape some information and return to the loop(i.e. return to "for div in divs:")
Now what's happening is:
1. I get the list of all the monuments in all the city before yield statement is executed.
2. Whenever yield statement is executed it goes to the parse_monument function and doesn't return back to the loop, only scraping the list of monument that is present in the first city.
Is there any way to do this? Is there any way to get the response object that request method passes to the parse_monument without going to the parse_monument method so that I can use selectors to select elements that I require from the response?
Thank You !!