natas_hw
/
sempre
mirror of https://github.com/percyliang/sempre
0
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
sempre/tables/wikipedia-scripts
History
Panupong Pasupat f203345ffd Updated the Wikipedia scraping scripts 2017-08-28 17:37:07 -07:00
..
weblib Updated the Wikipedia scraping scripts 2017-08-28 17:37:07 -07:00
README Added more files related to tables experiments 2016-06-04 16:15:28 -07:00
convert-all-to-html Added more files related to tables experiments 2016-06-04 16:15:28 -07:00
convert-to-html Added more files related to tables experiments 2016-06-04 16:15:28 -07:00
decode-html Added more files related to tables experiments 2016-06-04 16:15:28 -07:00
display.css Added more files related to tables experiments 2016-06-04 16:15:28 -07:00
display.html Added more files related to tables experiments 2016-06-04 16:15:28 -07:00
display.js Added more files related to tables experiments 2016-06-04 16:15:28 -07:00
download-wikipedia-pages.py Updated the Wikipedia scraping scripts 2017-08-28 17:37:07 -07:00
dump.py Added more files related to tables experiments 2016-06-04 16:15:28 -07:00
extract.py Added more files related to tables experiments 2016-06-04 16:15:28 -07:00
find-good-tables.py Updated the Wikipedia scraping scripts 2017-08-28 17:37:07 -07:00
format.py Added more files related to tables experiments 2016-06-04 16:15:28 -07:00
get-wikipedia-pages.py Updated the Wikipedia scraping scripts 2017-08-28 17:37:07 -07:00
process-exemplars.sh Added more files related to tables experiments 2016-06-04 16:15:28 -07:00
table-to-csv.py Updated the Wikipedia scraping scripts 2017-08-28 17:37:07 -07:00

README 

Tools for extracting data from Wikipedia
========================================
See the docstring of each file for descriptions

Extracting categories
---------------------
Requires SQL dumps of category data

- dump.py
    * Dump a field from an SQL dump
- extract.py
    * Extract categories and their members (requires multiple steps)
- format.py
    * Reformat Wikipedia article names

Extracting articles
-------------------
Requires Wikipedia dump (bz2)

- get-wikipedia-pages.py
    * Get wikipedia pages from the dump
- download-wikipedia-pages.py
    * Download wikipedia pages using the metadata from get-wikipedia-pages.py
- display.*
    * Display the downloaded pages 
- find-good-tables.py
    * Find pages with information-rich tables

Other utilities
---------------

- decode-html
    * Decode HTML entities (&lt; becomes <)
- convert-to-html / convert-all-to-html
    * Use pandoc to convert Markup to HTML (not that reliable)
- table-to-csv.py
    * Does what it says