Pipeline is the persistent and offline process part of crawler.