I have to call a crawler from another python file, for which I use :
def crawl_koovs(): spider = SomeSpider() settings = get_project_settings() crawler = Crawler(settings) crawler.signals.connect(reactor.stop, signal=signals.spider_closed) crawler.configure() crawler.crawl(spider) crawler.start() log.start() reactor.run()
On running this, I get the error:
exceptions.ValueError: signal only works in main thread
The only workaround I could find is to use
which I don't want to use as I want to call this method multiple times and want reactor to be stopped before the next call. What can I do to make this work (maybe force the crawler to start in the same 'main' thread)?