4.7 KiB

Raw Permalink Blame History

Compositional Semantic Parsing on Semi-Structured Tables

This README file describes the tables mode which accompanies the paper:

Panupong Pasupat, Percy Liang.
Compositional Semantic Parsing on Semi-Structured Tables
Association for Computational Linguistics (ACL), 2015.

Further descriptions and experiments can be viewed on the CodaLab website:

https://www.codalab.org/worksheets/0xf26cd79d4d734287868923ad1067cf4c/

Running the code

Download the dependencies and the dataset:

 ./pull-dependencies core
 ./pull-dependencies corenlp
 ./pull-dependencies tables
 ./pull-dependencies tables-data

The dataset lives in lib/data/WikiTableQuestions/

Compile the source:
```
 ant tables
```

This will produce JAR files in the libsempre directory.

The following command train and test on 100 development examples:

 ./run @mode=tables @data=u-1 @feat=all @train=1 -maxExamples train:100 dev:100

The command should take less than 30 minutes.

To train on the complete development set, remove -maxExamples train:100 dev:100
The command above uses u-1 (80:20 split of the development data). Other available sets include u-2, ..., u-5 (four other development splits) and test (actual train-test split).

Other options

Macro Grammar (Experimental)

Macro grammar can be used to significantly speed up the parser. To turn on macro grammar, run the following:

./run @mode=tables @data=u-1 @feat=more @parser=cprune @grammar=extended @fuzzy=editdist-fuzzy @train=1

Please refer to the following paper for more information:

Yuchen Zhang, Panupong Pasupat, Percy Liang.
Macro Grammars and Holistic Triggering for Efficient Semantic Parsing
Empirical Methods on Natural Language Processing (EMNLP), 2017.

Currently the module does not support model saving, and testing has to be done on the official test set. These features will be added in the future.

Official evaluation

The official evaluation script in the WikiTableQuestions dataset is slightly more lenient than the SEMPRE one (tables.TableValueEvaluator). In particular, the SEMPRE evaluator enforces that the type of the predicted denotation must match the correct answer type, while the official one allows type conversion.

To get the official number of a trained model, run

./pull-dependencies tables-cprune
./run @mode=tables @data=u-1 @feat=all @train=0 -Derivation.showValues -Builder.inParamsPath path/to/params

(Change the @data other options to match the ones used during training.)

This should produce an execution directory (in state/execs/ by default) with a log file in it. Then run

./tables/log-parsers/get-predictions.py path/to/log > predictions
./lib/data/WikiTableQuestions/evaluator.py -t ./lib/data/WikiTableQuestions/tagged-data predictions

Other usages

To launch the interactive shell, use:

./run @mode=tables -interactive

Apart from the usual shell commands, the additional command context can load the context graph for execution. For example, use:

(context (graph tables.TableKnowledgeGraph csv/204-csv/590.csv))

to load lib/data/tables/csv/204-csv/590.csv

The table can also be viewed in pretty-printed format by calling

./tables/view 204 590

./tables/view csv/204-csv/590.csv

Generating `.tagged` Files

The WikiTableQuestions dataset contains CoreNLP tagged files. Here is how they were generated.

The classes responsible for generating .tagged files are TaggedDatasetGenerator and TaggedTableGenerator in edu.stanford.nlp.sempre.tables.serialize.

Suppose the LispTree .examples files is at my-data.examples, and the CSV table files all have paths of the form table-base-directory/csv/___-csv/___.csv where ___ are numbers. (The path pattern is currently hard-coded.)

Please see the WikiTableQuestions dataset package for how the LispTree .examples file should be formatted (these files are stored in data/), and how the CSV table files are stored (these are stored in csv/; only .csv files are needed; .tsv, .html, and .table are not needed).

For dataset .tagged files:

./run @mode=tables @class=tag-data \
    -dataset.inpaths anygroupname:my-data.examples \
    -baseCSVDir table-base-directory @useTaggedFile=0

This will invoke TaggedDatasetGenerator. The results should be in states/exec/___.exec/___.tagged.

For table .tagged files:

./run @mode=tables @class=tag-table \
    -baseCSVDir table-base-directory @useTaggedFile=0

This will invoke TaggedTableGenerator. The results should be in states/exec/___.exec/tagged/...

4.7 KiB Raw Permalink Blame History