Bilingually Informed Parsing
P- TRUE FUZZY
4.4.5 Errors From Extended Parsers
We saw in the baseline results for the extended parser that the extended parsers yielded better results than the baseline parser for English, but not for Danish. The output from the standard Danish parser is not the same as the output from the extended Danish parser, although the accuracy is the same. This implies that the extended parser makes both good and bad changes compared to the standard parser. This is probably also the case for the extended English parser, although the overall accuracy is better. In the following we will investigate this further.
Total 8,681 std true ext true
Diff 512 6.00% 36.28% 40.69%
not aligned 6.91% 47.22% 36.11%
aligned to one 84.84% 35.97% 40.95%
aligned token has true head 49.10% 20.74% 66.36%
head is root 1.81% 12.50% 62.50%
head not aligned 2.26% 30.00% 40.00%
head aligned to one 79.41% 36.71% 39.32%
head aligned to true head 36.71% 7.58% 91.97%
head aligned to head
65.81% 38.10% 47.61%
of aligned token
aligned to many 8.25% 30.23% 41.86%
head is root 9.30% 25.00% 75.00%
head not aligned 0% -
-head aligned to one 69.77% 40.00% 50.00%
connected 65.12% 25.00% 50.00%
Table 4.11: Error analysis on extended Danish parsing.
Quantitative Analysis
Table 4.11 and 4.12 show some statistics from the output of the standard parser and the monolingual parser on the development data. In table 4.11 we see that when there is a difference between the standard and the ex-tended parser, it is more common that the exex-tended parser makes the cor-rect analysis than it is that the standard makes the corcor-rect analysis. This does not seem to match with the previous evaluation, which showed that the extended parser was not more accurate. The only explanation is that the main part of the difference is in non-scoring tokens (punctuation) and that the rest is too little to change the overall accuracy3.
The tables give an overview of the situations where there is a difference between the output of the standard parser and that of the extended parser.
We see that although the extended parsers are correct in more cases than the standard parsers, they actually make a lot of wrong decisions as well.
In general, the distributions over the two classes are the same for the
differ-3Evaluation with punctuation confirms this. UAS for baseline parser is 87.29 and for the extended parser it is 87.56.
4.4 Analysis 85
Total 9,464 std true ext true
Diff 901 9.52% 30.41% 44.62%
not aligned 4.44% 40.00% 30.00%
aligned to one 92.34% 29.93% 45.43%
aligned token has true head 65.14% 19.19% 65.31%
head is root 3.61% 30.00% 63.33%
head not aligned 2.16% 16.67% 38.89%
head aligned to one 90.38% 30.19% 44.95%
head aligned to true head 51.99% 15.35% 78.26%
head aligned to head
77.79% 29.57% 51.79%
of aligned token
aligned to many 3.22% 31.03% 41.38%
head is root 3.45% 100.00% 0%
head not aligned 0% -
-head aligned to one 89.66% 30.77% 46.15%
connected 79.31% 26.09% 47.83%
Table 4.12: Error analysis on extended English parsing.
ent situations we look at. There are some notable differences though. There is one situation where the standard parser is better4. This is when the de-pendent token on the source side is not aligned. In this case, the extended parser can of course not get any help from the target side, but there is no reason that is should do worse than the standard parser. These results sug-gest that the non-extended part of the model turns out worse than in the standard case.
The two other categories that differ most from the overall distribution are ’aligned to one aligned token has true head’ and ’alignedtoone -head-aligned to one - head aligned to true head’. In these categories, the extended parser is much better than the standard parser. This is not sur-prising as these are situations that are indicative of good input. In the first, the analysis of the token aligned to the source-side dependent is correct. In the second, the head is aligned to the true head of the token aligned to the source side dependent.
4Excluding the ’aligned to many’ - ’head is root’, which we exclude because there is only one example.
Although the analysis only gives a very coarse view of what is happen-ing in the extended parsers, it does show that the extended parsers work as we expect - when the input is good so is the output. Apart from this con-clusion, it is difficult from this analysis to identify situations that can help us in designing additional features.
Qualitative Analysis
In the following, we present a qualitative analysis of the errors made by the extended parser and not by the standard parser. We do this by ana-lyzing situations where the extended parsers make errors and the standard parsers do not, to investigate why this happens. We hope that this anal-ysis will help us to design features to help prevent these errors. This can be seen as a conservative way of increasing the quality of the output of the extended parser. Instead of trying to get it to make more of good changes compared to the standard parser, we focus on how to help it make fewer incorrect changes.
We will show some examples to illustrate the configurations we are dis-cussing. In these examples, the structure at the top will be the output of the extended parser and the structure at the bottom will be the extended input to the extended parser. Dependency arcs that are incorrect are drawn with dashed lines. The token with the vertical lines around it is the central token in the analysis.
Prepositions and Punctuation
Prepositions and punctuations are overrepresented when looking at the dependents that are incorrect in the output from the extended parsers, com-pared to the output from the standard parsers. These are high frequency words that often carry little meaning, and are often considered difficult to align correctly. There will often be more than one punctuation token in a sentence, which can make it difficult to pick the correct one. With respect to the prepositions, these are often part of 1-n, m-1 or n-m alignment - also making it difficult to align them correctly. Figure 4.13 shows an example where the extended parser makes an error involving a preposition. We see that ”foran” gets the wrong head. There is no clear indication why this is the case, but as said, we see an overrepresentation of prepositions and
4.4 Analysis 87 punctuation when looking at the errors.
har et forspring |foran| den siddende
has a lead over the sitting
Figure 4.13: Error from extended parser involving a preposition.
Head and Dependent Aligned to Same
Situations where the head and dependent on the source side are aligned to the same token on the target side also seem to be overrepresented when looking at the errors from the extended parser. Figure 4.14 shows an ex-ample of this. We have no really good reason why this causes errors in the extended parser.
" believes |Detective| Inspector Chr
" mener kriminalinspektør Chr.
Figure 4.14: Error from extended parser involving head and dependent aligned to the same token.
Wrong Input
Most of the errors are caused by the parser being misguided by either wrong alignments or a wrong analysis on the target side. To reduce the number of variables in our analysis we have tried redoing it with gold-standard alignments instead. The biggest source of errors after this is a
wrong analysis on the target side language. Figure 4.15 shows an example of this.
To remedy this, one would have to be able to do some kind of prediction on how likely it is that the target side analysis is correct. This is almost parsing, as it requires predicting how likely a dependency arc is, so this really reduces to making the target side better. An alternative would be to make some soft-link features that return some score to indicate how likely the target side analysis is, given the target side model. We have not pursued this idea further.
Prince Frederik |and| Prince Joachim
Kronprins Frederik og prins Joachim
Figure 4.15: Error from extended parser involving a wrong analysis on the target side.
n-1 Alignments
We could identify one other major source of errors in the extended parser - i.e. situations where the monolingual parser makes the correct analysis, and the extended parser does not. These are situations where the parser is misguided by n-1 alignments. Figure 4.16 shows an example of this.
If the possible dependent is part of such a feature, thehead1-dep1and headn-dep1-features are activated, but the information from the target sen-tence is a lot less reliable if the target-side token is also aligned to more words in the source sentence.