natas_hw
/
sempre
mirror of https://github.com/percyliang/sempre
0
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

readme updates, efficiency improvements, etc

This commit is contained in:
Sida Wang 2017-04-06 23:22:40 -07:00
parent 65fd5122ca
commit 8d3d170bd9
15 changed files with 2027 additions and 230 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

104
interactive/README.md
View File

@ -1,43 +1,68 @@
# README
This `interactive` package is the code for our paper
*Naturalizing a programming language through interaction* (ACL 2017)
## Running an experiment
1) Start the server
./interactive/run @mode=voxelurn -server
2) Query the server with existing data
./interactive/run @mode=simulator @server=local @sandbox=none @task=freebig
3) Run analysis script to get results
./interactive/run @mode=analyze
4) (Optional) clean up
./interactive/run @mode=backup # save previous data logs
./interactive/run @mode=trash # deletes previous data logs
## Running the server for voxelurn
1) Start the sempre server
./interactive/run @mode=voxelurn -server
2) Start the client server
./interactive/run @mode=community
2) (Optionally) Query the server with existing data and use previous definitions
./interactive/run @mode=simulator @server=local @sandbox=none @task=freebigdef
Voxelurn is a language interface to a voxel world.
This server handles commands used to learn from definitions, and other interactive queries.
In this setting, the system begin with the dependency-based action language (`dal.grammar`), and gradually expand the language through interacting with it users.
## Tests
## Running the Voxelurn server
There are many units for interactive learning
1. Start the server
./interactive/run @mode=voxelurn -server -interactive
things in the core language such as `add red left`, `repeat 3 [select left]` should work.
2. Feed the server existing definitions, which should take less than 2 minutes.
./interactive/run @mode=simulator @server=local @sandbox=none @task=freebigdef -maxQueries 2496
try `add green monster` now.
### Interacting with the server
There are 3 ways to interactive with the server
* The visual way is to use our client at `https://github.com/sidaw/shrdlurn`, which has a more detailed [README.md](https://github.com/sidaw/shrdlurn/blob/master/README.md) for the client.
After `npm install` and taking care of dependencies, run `npm start local` to start a client that queries localhost. Try `[add dancer; front 5] 3 times`
* Hit `Ctrl-D` on the server terminal, and type `add red top`, or `add green monster`
* On a browser, type `http://localhost:8410/sempre?q=(:q add green monster)`
## Experiments in ACL2017
1. Start the server
./interactive/run @mode=voxelurn -server -interactive
2. Feed the server all the query logs
./interactive/run @mode=simulator @server=local @sandbox=none @task=freebig -maxQueries 103876
decrease maxQuery for shorter plots
3. Analyze the results and produce some numbers and graphs
./interactive/run @mode=voxelurn
## Client server (optional)
This server helps with client side logging, leaderboard, authentication etc. basically anything that is not directly parsing.
cd interactive
python community-server/install-deps.py
export SEMPRE_JWT_SECRET=sdlfdsaklafsl
export SLACK_SECRET=somekeyyougetfromslack
python community-server/server.py --port 8403
## Misc.
There are some unit tests
./interactive/run @mode=test
@ -45,6 +70,13 @@ To specify a specific test class and verbosity
./interactive/run @mode=test @class=DALExecutorTest -verbose 5
Test in interactive mode
Clean up or backup data
./interactive/run @mode=voxelurn -interactive
./interactive/run @mode=backup # save previous data logs
./interactive/run @mode=trash # deletes previous data logs
Data, in .gz can be found in queries.
* `./interactive/queries/freebuildbig-0206.def.json.gz`
has 2495 definitions combining just over 10k utterances.
* `./interactive/queries/freebuildbig-0206.json.gz` has 103875 queries made during the main experiment.

16
interactive/analyze_definitions.ipynb
View File

@ -16,7 +16,7 @@
},
{
"cell_type": "code",
"execution_count": 45,
"execution_count": 85,
"metadata": {
"collapsed": false,
"scrolled": false
@ -26,15 +26,15 @@
"name": "stdout",
"output_type": "stream",
"text": [
"898\n",
"log.file\tstate/execs/898.exec/log\n",
"995\n",
"log.file\tstate/execs/995.exec/log\n",
"Simulator.reqParams\tgrammar=1&cite=1&learn=1&logging=0\n",
"Simulator.logFiles\t./interactive/queries/freebuildbig-0206.def.json.gz\n",
"2495 lines in plotInfo\n",
"[u'java.util.concurrent.TimeoutException', u'java.util.concurrent.TimeoutException', u'java.util.concurrent.TimeoutException', u'java.util.concurrent.TimeoutException', u'java.util.concurrent.TimeoutException', u'java.util.concurrent.TimeoutException', u'java.util.concurrent.TimeoutException', u'java.util.concurrent.TimeoutException', u'java.util.concurrent.TimeoutException', u'java.util.concurrent.TimeoutException']\n",
"errors:80, stats.uncaught_error:10\n",
"{'total_rules': 2874, 'total_failed': 724, 'total_def_queries': 2405, 'total_body': 15498}\n",
"failpercent: 0.0467\n"
"2000 lines in plotInfo\n",
"[]\n",
"errors:66, stats.uncaught_error:0\n",
"{'total_rules': 2320, 'total_failed': 447, 'total_def_queries': 1911, 'total_body': 10973}\n",
"failpercent: 0.0407\n"
]
}
],

4
interactive/community-server/install-deps.py
View File

@ -3,8 +3,6 @@
"""
Installs the needed dependencies for the community-server.
This should be called only from ./pull-dependencies community-server
Ideally, you would use virtualenv.
"""
@ -12,4 +10,4 @@ import pip
print("Installing community-server dependencies...")
pip.main(['install', '-r', 'community-server/requirements.txt'])
pip.main(['install', '-r', 'requirements.txt'])

1913
interactive/generate_plots.ipynb
View File

File diff suppressed because it is too large Load Diff

17
interactive/run
View File

@ -101,18 +101,6 @@ addMode('test', 'Run unit tests for interactive stuff', lambda { |e|
nil)
})
addMode('community-dep', 'dependencies for the community server', lambda { |e| l(
l("python ./#{$path}/community-server/install-deps.py"),
nil)})
addMode('community', 'start the community server', lambda { |e| l(
l("python ./#{$path}/community-server/server.py --port 8403"),
nil)})
addMode('analyze', 'analyze the latest run', lambda { |e| l(
l("python ./#{$path}/analyze.py --port 8403"),
nil)})
addMode('simulator', 'run the simulator', lambda { |e| l(
# rlwrap,
header('core,interactive'),
@ -177,13 +165,15 @@ addMode('voxelurn', 'interactive semantic parsing in a VoxelWorld', lambda { |e|
o('Parser.callSetEvaluation', false),
o('Parser.coarsePrune', 'true'),
o('Parser.beamSize', 50),
o('Parser.beamSize', 100),
o('BeamFloatingParser.maxNewTreesPerSpan', 5001),
o('BeamFloatingParser.maxNonterminals', 5),
o('ParserState.customExpectedCounts', 'None'),
selo(0, 'BeamFloatingParser.floatStrategy', 'Never', 'NoParse', 'Always'),
o('BeamFloatingParser.trackedCats', 'Number', 'Numbers', 'Color', 'Direction', 'Set', 'Sets', 'Action', 'Actions'),
o('Derivation.derivComparator', 'AnchorPriorityScoreComparator'),
o('Params.l1Reg', 'nonlazy'),
o('Params.l1RegCoeff', 0.0001),
@ -213,6 +203,7 @@ addMode('voxelurn', 'interactive semantic parsing in a VoxelWorld', lambda { |e|
o('GrammarInducer.useBestPacking', true),
o('GrammarInducer.useSimplePacking', true),
o('GrammarInducer.maxNonterminals', 2),
o('Derivation.showTypes', false),
o('Derivation.showValues', false),

25
src/edu/stanford/nlp/sempre/Derivation.java
View File

@ -28,6 +28,8 @@ public class Derivation implements SemanticFn.Callable, HasScore {
public boolean showCat = false;
@Option(gloss = "When executing, show formulae (for debugging)")
public boolean showExecutions = false;
@Option(gloss = "Pick the comparator used to sort derivations")
public String derivComparator = "ScoredDerivationComparator";
}
public static Options opts = new Options();
@ -112,7 +114,9 @@ public class Derivation implements SemanticFn.Callable, HasScore {
// we can break ties consistently for reproducible results.
long creationIndex;
public static long numCreated = 0; // Incremented for each derivation we create.
public static final Comparator<Derivation> derivScoreComparator = new ScoredDerivationComparator();
@SuppressWarnings("unchecked")
public static final Comparator<Derivation> derivScoreComparator =
(Comparator<Derivation>)Utils.newInstanceHard(SempreUtils.resolveClassName("Derivation$" + opts.derivComparator));
public static final List<Derivation> emptyList = Collections.emptyList();
@ -453,6 +457,25 @@ public class Derivation implements SemanticFn.Callable, HasScore {
return 0;
}
}
//Used to compare derivations by score, prioritizing the fully anchored.
public static class AnchorPriorityScoreComparator implements Comparator<Derivation> {
@Override
public int compare(Derivation deriv1, Derivation deriv2) {
boolean deriv1Core = deriv1.allAnchored();
boolean deriv2Core = deriv2.allAnchored();
if (deriv1Core && !deriv2Core) return -1;
if (deriv2Core && !deriv1Core) return +1;
if (deriv1.score > deriv2.score) return -1;
if (deriv1.score < deriv2.score) return +1;
// Ensure reproducible randomness
if (deriv1.creationIndex < deriv2.creationIndex) return -1;
if (deriv1.creationIndex > deriv2.creationIndex) return +1;
return 0;
}
}
// for debugging
public void printDerivationRecursively() {

2
src/edu/stanford/nlp/sempre/Main.java
View File

@ -30,7 +30,7 @@ public class Main implements Runnable {
Master master = createMaster(masterType, builder);
if (server)
master.runServer();
else if (interactive)
if (interactive)
master.runInteractivePrompt();
}
}

6
src/edu/stanford/nlp/sempre/ParserState.java
View File

@ -130,15 +130,15 @@ public abstract class ParserState {
for (Derivation deriv : derivations)
deriv.score += Parser.opts.derivationScoreRandom.nextDouble() * Parser.opts.derivationScoreNoise;
}
Derivation.sortByScore(derivations);
// Print out information
if (Parser.opts.verbose >= 3) {
LogInfo.begin_track("ParserState.pruneCell(%s): %d derivations", cellDescription, derivations.size());
for (Derivation deriv : derivations) {
LogInfo.logs("%s(%s,%s): %s %s, [score=%s]", deriv.cat, deriv.start, deriv.end, deriv.formula,
deriv.canonicalUtterance, deriv.score);
LogInfo.logs("%s(%s,%s): %s %s, [score=%s] allAnchored: %s", deriv.cat, deriv.start, deriv.end, deriv.formula,
deriv.canonicalUtterance, deriv.score, deriv.allAnchored());
}
LogInfo.end_track();
}

48
src/edu/stanford/nlp/sempre/RuleSource.java
View File

@ -6,6 +6,7 @@ import java.util.List;
import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.annotation.JsonInclude;
import com.fasterxml.jackson.annotation.JsonProperty;
/**
* Tracking where the rule comes from in the grammar induction process.
*
@ -15,24 +16,33 @@ import com.fasterxml.jackson.annotation.JsonProperty;
@JsonIgnoreProperties(ignoreUnknown = true)
@JsonInclude(JsonInclude.Include.NON_NULL)
public class RuleSource {
@JsonProperty public String uid;
@JsonProperty public LocalDateTime time;
@JsonProperty public String head;
@JsonProperty public List<String> body;
@JsonProperty
public String uid;
@JsonProperty
public LocalDateTime time;
@JsonProperty
public String head;
@JsonProperty
public List<String> body;
@JsonProperty
public int cite = 0;
@JsonProperty
public int self = 0;
@JsonProperty
public boolean align = false;
@JsonProperty
public String alignInfo = "";
public RuleSource(String uid, String head, List<String> body) {
this.uid = uid;
this.head = head;
this.body = body;
this.time = LocalDateTime.now();
}
public String toJson() {
return Json.writeValueAsStringHard(this);
}
@JsonProperty public int cite = 0;
@JsonProperty public int self = 0;
@JsonProperty public boolean align = false;
@JsonProperty public String alignInfo = "";
public RuleSource(String uid, String head, List<String> body) {
this.uid = uid;
this.head = head;
this.body = body;
this.time = LocalDateTime.now();
}
public String toJson() { return Json.writeValueAsStringHard(this); }
}

6
src/edu/stanford/nlp/sempre/Trie.java
View File

@ -2,6 +2,8 @@ package edu.stanford.nlp.sempre;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.LinkedHashMap;
import java.util.LinkedHashSet;
import java.util.HashMap;
import java.util.Map;
import java.util.Set;
@ -12,9 +14,9 @@ import java.util.Set;
*/
public class Trie {
public ArrayList<Rule> rules = new ArrayList<>();
Map<String, Trie> children = new HashMap<>();
Map<String, Trie> children = new LinkedHashMap<>();
// Set of LHS categories of all rules in this subtree
public Set<String> cats = new HashSet<>();
public Set<String> cats = new LinkedHashSet<>();
public Trie next(String item) { return children.get(item); }

63
src/edu/stanford/nlp/sempre/interactive/BeamFloatingParser.java
View File

@ -27,11 +27,13 @@ import edu.stanford.nlp.sempre.ParserState;
import edu.stanford.nlp.sempre.Rule;
import edu.stanford.nlp.sempre.SemanticFn;
import edu.stanford.nlp.sempre.Trie;
import fig.basic.Evaluation;
import fig.basic.IOUtils;
import fig.basic.IntRef;
import fig.basic.LogInfo;
import fig.basic.Option;
import fig.basic.SetUtils;
import fig.basic.StopWatch;
import fig.basic.StopWatchSet;
import fig.exec.Execution;
@ -53,6 +55,8 @@ public class BeamFloatingParser extends Parser {
public FloatStrategy floatStrategy = FloatStrategy.Never;
@Option(gloss = "track these categories")
public List<String> trackedCats;
@Option
public int maxNonterminals = Integer.MAX_VALUE;
}
public enum FloatStrategy {
@ -79,6 +83,22 @@ public class BeamFloatingParser extends Parser {
this.chartFillOut = IOUtils.openOutAppendEasy(Execution.getFile("chartfill"));
}
// for grammar induction, just need the formula, do not execute
public BeamFloatingParserState justParse(Params params, Example ex, boolean computeExpectedCounts) {
// Parse
StopWatch watch = new StopWatch();
watch.start();
BeamFloatingParserState state = new BeamFloatingParserState(this, params, ex);
state.infer();
watch.stop();
state.parseTime = watch.getCurrTimeLong();
ex.predDerivations = state.predDerivations;
Derivation.sortByScore(ex.predDerivations);
// Clean up temporary state used during parsing
return state;
}
@Override
public synchronized void addRule(Rule rule) {
if (!rule.isCatUnary()) {
@ -126,15 +146,25 @@ class BeamFloatingParserState extends ChartParserState {
private final BeamFloatingParser parser;
private final BeamFloatingParserState coarseState; // Used to prune
private final boolean execute;
public List<Derivation> chartList;
public BeamFloatingParserState(BeamFloatingParser parser, Params params, Example ex) {
super(parser, params, ex, false);
this.parser = parser;
this.mode = Mode.full;
this.coarseState = null;
this.execute = false;
}
public BeamFloatingParserState(BeamFloatingParser parser, Params params, Example ex, boolean computeExpectedCounts,
Mode mode, BeamFloatingParserState coarseState) {
super(parser, params, ex, computeExpectedCounts);
this.parser = parser;
this.mode = mode;
this.coarseState = coarseState;
this.execute = true;
}
@Override
@ -182,6 +212,16 @@ class BeamFloatingParserState extends ChartParserState {
else
parseFloat = false;
if (mode == Mode.full) {
// Compute gradient with respect to the predicted derivations
if (this.execute)
ensureExecuted();
if (computeExpectedCounts) {
expectedCounts = new HashMap<>();
ParserState.computeExpectedCounts(predDerivations, expectedCounts);
}
}
/* If Beam Parser failed to find derivations, try a floating parser */
if (parseFloat) {
/*
@ -220,15 +260,6 @@ class BeamFloatingParserState extends ChartParserState {
}
}
}
if (mode == Mode.full) {
// Compute gradient with respect to the predicted derivations
ensureExecuted();
if (computeExpectedCounts) {
expectedCounts = new HashMap<>();
ParserState.computeExpectedCounts(predDerivations, expectedCounts);
}
}
}
private List<Derivation> collectChart() {
@ -296,10 +327,12 @@ class BeamFloatingParserState extends ChartParserState {
String cell = cellString(cat, start, end);
if (cellsPruned.contains(cell))
return;
cellsPruned.add(cell);
pruneCell(cell, derivations);
}
private boolean canBeRoot(int start, int end) {return start==0 && end==numTokens;};
// Apply all unary rules with RHS category.
// Before applying each unary rule (rule.lhs -> rhsCat), we can prune the cell
// of rhsCat
@ -308,17 +341,17 @@ class BeamFloatingParserState extends ChartParserState {
for (Rule rule : parser.getCatUnaryRules()) {
if (!coarseAllows(rule.lhs, start, end))
continue;
if (rule.lhs.equals(Rule.rootCat) && !canBeRoot(start, end))
continue;
String rhsCat = rule.rhs.get(0);
List<Derivation> derivations = chart[start][end].get(rhsCat);
if (Parser.opts.verbose >= 5)
LogInfo.logs("applyCatUnaryRules %s %s %s %s", start, end, rule, chart[start][end]);
if (derivations == null)
continue;
pruneCell(cellsPruned, rhsCat, start, end, derivations); // Prune before
// applying rules
// to eliminate
// cruft!
// Prune before applying rules to eliminate cruft!
pruneCell(cellsPruned, rhsCat, start, end, derivations);
for (Derivation deriv : derivations) {
applyRule(start, end, rule, Collections.singletonList(deriv));
@ -339,6 +372,8 @@ class BeamFloatingParserState extends ChartParserState {
return;
if (!coarseAllows(node, start, end))
return;
if (children.size() > BeamFloatingParser.opts.maxNonterminals)
return;
if (Parser.opts.verbose >= 5) {
LogInfo.logs("applyNonCatUnaryRules(start=%d, end=%d, i=%d, children=[%s], %s rules)", start, end, i,

27
src/edu/stanford/nlp/sempre/interactive/GrammarInducer.java
View File

@ -45,6 +45,8 @@ public class GrammarInducer {
public boolean useBestPacking = true;
@Option(gloss = "use simple packing")
public boolean useSimplePacking = true;
@Option(gloss = "maximum nonterminals in a rule")
public long maxNonterminals = 3;
}
public static Options opts = new Options();
@ -104,7 +106,7 @@ public class GrammarInducer {
List<Rule> simpleInduced = induceRules(packing, def);
for (Rule rule : simpleInduced) {
rule.addInfo("simple_packing", 1.0);
addRuleDedupByRHS(rule);
filterRule(rule);
}
if (opts.verbose > 1) {
@ -125,7 +127,7 @@ public class GrammarInducer {
for (Rule rule : induceRules(bestPacking, def)) {
if (rule.rhs.stream().allMatch(s -> Rule.isCat(s)))
continue;
addRuleDedupByRHS(rule);
filterRule(rule);
}
if (opts.verbose > 1) {
@ -141,11 +143,24 @@ public class GrammarInducer {
Set<String> RHSs = new HashSet<>();
private void addRuleDedupByRHS(Rule rule) {
if (!RHSs.contains(rule.rhs.toString())) {
inducedRules.add(rule);
RHSs.add(rule.rhs.toString());
private void filterRule(Rule rule) {
//
if (RHSs.contains(rule.rhs.toString())) {
LogInfo.logs("GrammarInducer.filterRule: already have %s", rule.rhs.toString());
return;
}
int numNT = 0;
for (String t : rule.rhs) {
if (Rule.isCat(t)) numNT++;
}
if (numNT > GrammarInducer.opts.maxNonterminals ) {
LogInfo.logs("GrammarInducer.filterRule: too many nontermnimals (max %d) %s", GrammarInducer.opts.maxNonterminals, rule.rhs.toString());
return;
}
inducedRules.add(rule);
RHSs.add(rule.rhs.toString());
}
static Map<String, List<Derivation>> makeChartMap(List<Derivation> chartList) {

15
src/edu/stanford/nlp/sempre/interactive/InteractiveMaster.java
View File

@ -88,7 +88,7 @@ public class InteractiveMaster extends Master {
Response response = new Response();
if (line.startsWith("(:"))
handleCommand(session, line, response);
else if (line.startsWith("(") && opts.allowRegularCommands)
else if (line.startsWith("(") && opts.allowRegularCommands || session.id.equals("stdin"))
super.processQuery(session, line);
else
handleCommand(session, String.format("(:q \"%s\")", line), response);
@ -153,9 +153,7 @@ public class InteractiveMaster extends Master {
response.ex = ex;
// Parse!
ParserState state;
state = builder.parser.parse(builder.params, ex, true);
state.ensureExecuted();
((BeamFloatingParser)builder.parser).justParse(builder.params, ex, false);
int rank = -1;
Derivation match = null;
@ -277,17 +275,14 @@ public class InteractiveMaster extends Master {
if (isNonsense(exHead))
throw BadInteractionException.nonSenseDefinition(head);
boolean coarsePrune = Parser.opts.coarsePrune;
Parser.opts.coarsePrune = false;
BeamFloatingParserState state = (BeamFloatingParserState) parser.parse(params, exHead, true);
Parser.opts.coarsePrune = coarsePrune;
BeamFloatingParserState state = ((BeamFloatingParser)parser).justParse(params, exHead, false);
if (GrammarInducer.getParseStatus(exHead) == GrammarInducer.ParseStatus.Core)
throw BadInteractionException.headIsCore(head);
LogInfo.logs("num anchored: %d", state.chartList.size());
List<String> bodyList = InteractiveUtils.utterancefromJson(jsonDef, false);
LogInfo.logs("bodyutterance:\n %s", String.join("\n", bodyList));
LogInfo.logs("bodyutterances:\n %s", String.join("\t", bodyList));
Derivation bodyDeriv = InteractiveUtils
.combine(InteractiveUtils.derivsfromJson(jsonDef, parser, params, refResponse));
@ -303,9 +298,7 @@ public class InteractiveMaster extends Master {
rule.source = new RuleSource(session.id, head, bodyList);
}
LogInfo.logs("testing Aligner, %b, %d", opts.useAligner, bodyList.size());
if (opts.useAligner && bodyList.size() == 1) {
LogInfo.logs("testing Aligner2");
List<Rule> alignedRules = DefinitionAligner.getRules(exHead.getTokens(),
InteractiveUtils.utterancefromJson(jsonDef, true), bodyDeriv, state.chartList);
for (Rule rule : alignedRules) {

7
src/edu/stanford/nlp/sempre/interactive/InteractiveServer.java
View File

@ -174,12 +174,12 @@ public class InteractiveServer {
List<Object> items = new ArrayList<Object>();
json.put("candidates", items);
List<Derivation> allCandidates = response.getExample().getPredDerivations();
Derivation.sortByScore(allCandidates);
if (allCandidates != null) {
if (allCandidates.size() >= InteractiveServer.opts.maxCandidates) {
allCandidates = allCandidates.subList(0, InteractiveServer.opts.maxCandidates);
response.lines.add(String.format("Exceeded max options: (current: %d / max: %d) ", allCandidates.size(),
InteractiveServer.opts.maxCandidates));
allCandidates = allCandidates.subList(0, InteractiveServer.opts.maxCandidates);
}
for (Derivation deriv : allCandidates) {
@ -219,12 +219,13 @@ public class InteractiveServer {
} catch (Throwable e) {
e.printStackTrace();
message = e.toString();
response.lines.add(message);
response.lines.add(String.format("Exceeded the maximum allowed time: %ss", opts.maxExecutionTime));
response.stats.put("uncaught_error", message);
LogInfo.flush();
LogInfo.resetInfos();
} finally {
future.cancel(true);
executor.shutdown();
long endTime = System.nanoTime();
response.stats.put("time", (endTime - startTime) / 1.0e9);
}

4
src/edu/stanford/nlp/sempre/interactive/InteractiveUtils.java
View File

@ -91,8 +91,8 @@ public final class InteractiveUtils {
Example ex = b.createExample();
ex.preprocess();
// LogInfo.logs("Parsing definition: %s", ex.utterance);
parser.parse(params, ex, false);
LogInfo.logs("Parsing body: %s", ex.utterance);
((BeamFloatingParser)parser).justParse(params, ex, false);
boolean found = false;
Formula targetFormula = Formulas.fromLispTree(LispTree.proto.parseFromString(formula));