gdom - DOM Traversing and Scraping using GraphQL

gdom: DOM Traversing and Scraping using GraphQL   2016-02-27 22:57:44 发布
您的评价:
     
0.0
收藏     0收藏
文件夹
标签
(多个标签用逗号分隔)

GDOM

GDOM is the next generation of web-parsing, powered by GraphQL syntax and the Graphene framework.

Install it typing in your console:

pip install gdom

DEMO: Try GDOM online

Usage

You can either do gdom --test to start a test server for testing queries or

gdom QUERY_FILE

This command will write in the standard output (or other output if specified via --output) the resulting JSON.

Your QUERY_FILE could look similar to this:

{
  page(url:"http://news.ycombinator.com") {
    items: query(selector:"tr.athing") {
      rank: text(selector:"td span.rank")
      title: text(selector:"td.title a")
      sitebit: text(selector:"span.comhead a")
      url: attr(selector:"td.title a", name:"href")
      attrs: next {
         score: text(selector:"span.score")
         user: text(selector:"a:eq(0)")
         comments: text(selector:"a:eq(2)")
      }
    }
  }
}

Advanced usage

If you want to generalize your gdom query to any page, just rewrite your query file adding the $page var. So should look to something like this:

query ($page: String) {
  page(url:$page) {
    # ...
  }
}

And then, query it like:

gdom QUERY_FILE http://news.ycombinator.com

项目地址: https://github.com/syrusakbary/gdom

扩展阅读

Python Web 爬虫汇总
Python 用于网站抓取 登录 发布的模块介绍
Scrapy 示例 - Web 爬虫框架
Python的网页爬虫&文本处理&科学计&机器学习&数据挖掘工具集
Go语言(golang)开源项目大全

为您推荐

30分钟快速掌握Boostrap
struts2上传图片并剪切
JQuery经典总结
CSS选择器
PostgreSQL 数据库的简单操作PHP类

更多

gdom: DOM Traversing and Scraping using GraphQL
软件开发
相关文档  — 更多
相关经验  — 更多
相关讨论  — 更多