sempre

History

Panupong Pasupat 506314c5e6 Table stuff from master branch		2016-04-09 23:13:14 -07:00
..
weblib	Table stuff from master branch	2016-04-09 23:13:14 -07:00
README	Table stuff from master branch	2016-04-09 23:13:14 -07:00
convert-all-to-html	Table stuff from master branch	2016-04-09 23:13:14 -07:00
convert-to-html	Table stuff from master branch	2016-04-09 23:13:14 -07:00
decode-html	Table stuff from master branch	2016-04-09 23:13:14 -07:00
display.css	Table stuff from master branch	2016-04-09 23:13:14 -07:00
display.html	Table stuff from master branch	2016-04-09 23:13:14 -07:00
display.js	Table stuff from master branch	2016-04-09 23:13:14 -07:00
download-wikipedia-pages.py	Table stuff from master branch	2016-04-09 23:13:14 -07:00
dump.py	Table stuff from master branch	2016-04-09 23:13:14 -07:00
extract.py	Table stuff from master branch	2016-04-09 23:13:14 -07:00
find-good-tables.py	Table stuff from master branch	2016-04-09 23:13:14 -07:00
format.py	Table stuff from master branch	2016-04-09 23:13:14 -07:00
get-wikipedia-pages.py	Table stuff from master branch	2016-04-09 23:13:14 -07:00
process-exemplars.sh	Table stuff from master branch	2016-04-09 23:13:14 -07:00
table-to-csv.py	Table stuff from master branch	2016-04-09 23:13:14 -07:00

README

Tools for extracting data from Wikipedia
========================================
See the docstring of each file for descriptions

Extracting categories
---------------------
Requires SQL dumps of category data

- dump.py
    * Dump a field from an SQL dump
- extract.py
    * Extract categories and their members (requires multiple steps)
- format.py
    * Reformat Wikipedia article names

Extracting articles
-------------------
Requires Wikipedia dump (bz2)

- get-wikipedia-pages.py
    * Get wikipedia pages from the dump
- download-wikipedia-pages.py
    * Download wikipedia pages using the metadata from get-wikipedia-pages.py
- display.*
    * Display the downloaded pages 
- find-good-tables.py
    * Find pages with information-rich tables

Other utilities
---------------

- decode-html
    * Decode HTML entities (&lt; becomes <)
- convert-to-html / convert-all-to-html
    * Use pandoc to convert Markup to HTML (not that reliable)
- table-to-csv.py
    * Does what it says