Python2.7Step-by-step tutorial of installing Scrapy framework

Due to the requirements of the graduation project, a large amount of data needs to be scraped from the website, and using the Scrapy framework can make this process much simpler, as Scrapy is an application framework written specifically for crawling website data and extracting structured data. Therefore, I embarked on my journey to install the Scrapy framework. It can be said that this process was not very pleasant, with various errors popping up, but in the end, I finally installed the Scrapy framework. Below is a summary of my Scrapy framework installation.

1. Install python2.7

Since Scrapy does not support Python3.0, so I uninstalled Python3.0, and then reinstalled Python2.7(python2.7The installation package), when installing Python2.7There will be an option to automatically set environment variables, it is recommended to select this option here to save the trouble of adding environment variables later. If we add environment variables ourselves later, we can add these two statements to the system environment variable path according to our actual installation path.

After configuring the environment, we test whether we have successfully installed and configured the environment. Just enter python –version in cmd, and then the correct python version will be displayed. If it does not display, try restarting cmd.

2. Install pywin32

After installing and configuring python2.7After that, we cannot directly install Scrapy, we first need to install several tools that Scrapy depends on. Next, install pywin32(pywin32The installation package), you can just keep clicking next during the software installation.

3. Install pip

pip is actually a very convenient online software installation tool, similar to easy install. Now we are installing pip, so we can use the pip install command for software installation in the future. First, we need to download get-pip.py(get-The pip.py file) After the download is complete, we install it under cmd, first switch to the directory where the file is located, and then enter python get-After the installation of pip.py, you can install it, but the most troublesome problem appeared. Since this file will automatically write some configuration files to our C disk user file directory, and my user directory is a Chinese name, it will cause encoding exceptions.

After searching for some information, the problem was finally solved. To solve the problem of Chinese path, we only need to set the path in D:\python27\Lib\site-A new directory named
The file sitecustomize.py can be created under the packages directory.

Then enter the following code in the file:

import sys
reload(sys)
sys.setdefaultencoding('gbk')

Of course, it is not necessary to set the encoding to gbk, it should be set according to the encoding of your computer. In this way, our Chinese path problem is solved, and after restarting cmd and executing the pip installation command, pip is installed smoothly. After the installation is complete, we can carry out a check, by entering pip –version in cmd, if the correct version number is displayed, it indicates that it is correct.

4.Install lxml

After installing pip, I thought I could easily use the command pip install lxml to install it, but an unexpected event occurred. Since pip needs vc2008environment, and it can only be2008the2013All of them do not work. In order to use this tool, we can only turn to DuDu (Baidu), and finally find the solution. It turns out that Microsoft has provided VcForPython (VCForPython installation package) for us, so we don't need to install VC2008After that, we can use pip install lxml to install it.

5.Install pyOpenSSL

When installing this tool, we can use the statement pip install pyOpenSSL to install it, and the installation speed depends on the internet speed.

6.Install Scrapy

After experiencing a series of ups and downs, we finally can install the Scrapy framework. After entering the command pip install Scrapy in the cmd, we can look forward to enjoying the joy of success. After the installation is complete, we can enter Scrapy in the cmd to check whether it is truly installed successfully.

Finally installed, still a bit of a sense of achievement.

That's all for this article. Hope it will be helpful to your study, and also hope everyone will support the Yelling Tutorial more.

Declaration: The content of this article is from the Internet, the copyright belongs to the original author. The content is contributed and uploaded by Internet users spontaneously. This website does not own the copyright, has not been manually edited, and does not assume any relevant legal liability. If you find any suspected copyright content, please send an email to: notice#w3Please report via email to codebox.com (replace # with @) when reporting, and provide relevant evidence. Once verified, this site will immediately delete the suspected infringing content.