Home > front end >  passing table name to pipeline scrapy python
passing table name to pipeline scrapy python

Time:01-27

I have different spiders that scrape similar values and I want to store the scraped values in different slite3 tables. I can do this by using a different pipeline for each spider but, since the only thing that changes is the table name, would it be possible to pass somehow the table name from the spider to the pipeline? This is the code I use to create the table:

def open_spider(self, spider):
        self.connection = sqlite3.connect("database.db")
        self.c = self.connection.cursor()
        try:
            self.c.execute('''
                CREATE TABLE IF NOT EXISTS table_name(
                    value1 TEXT,
                    value2 TEXT,
                    value3 TEXT
                )
            ''')
            self.connection.commit()

CodePudding user response:

yes , something like this?

def open_spider(self, spider):
        self.connection = sqlite3.connect("database.db")
        self.c = self.connection.cursor()
        querystring = f"
                CREATE TABLE IF NOT EXISTS {spider.name} (
                    value1 TEXT,
                    value2 TEXT,
                    value3 TEXT
                )
            "
        try:
            self.c.execute(querystring)
            self.connection.commit()
  •  Tags:  
  • Related