lundi 23 février 2015

Match an array of possible tokens and return the matched token and its position

I'm writing a sound change program in Java which should replace patterns of tokens with a replacement string only if they match the pattern string.


The pattern string can contain literal strings and/or variables using %TokenName. Such a variable references a Token class containing a List of Strings containing the possible token values. An optional anchor to specifiy the location of the pattern (^ and $ like in regex ) can preceed or succeed the pattern. All whitespace is deleted while processing the replace.


The following example should only match when first a ShortVowel token occurs, followed by a VoicelessStop, and the the string should end:



%ShortVowel %VoicelessStop $


with the folloing tokens:



ShortVowel: ɑ ɛ ɪ jɪ ɔ ə
VoicelessStop: k p t


I want the replacer to return an array of a ReplacerMatch class containing the List of Strings with the matched tokens per variable, and the start and end positions of the total match in the string to be processed. For every match in the string, such a class exists in the array.


This means the string dɛt should return



[
matches: [ɛ, t]
startPosition: 1
endPosition: 3
]


and the string drɛkɔp should return



[
matches: [ɔ, p]
startPosition: 4
endPosition: 6
]


since only matches at the end of the string are matched. The string should return an empty array.




The ReplacerMatch class is defined as follows:



public class ReplacerMatch
{
private List<String> matches;
private int startPosition;
private int endPosition;

[...]
}


Such a replace rule is defined in a Replacer class:



public class Replacer
{
enum Anchor
{
NONE,
STRING_START,
STRING_END;

public static Anchor fromString(String string)
{
if (string.startsWith("^"))
return STRING_START;
else if (string.endsWith("$"))
return STRING_END;
else
return NONE;
}
}

private String pattern;
private String replacement;
private List<Token> tokens;

[...]
}


The Token class contains the name of that token and a list of String with possible token values. These values can be of variable length.



public class Token
{
private final String name;
private final List<String> tokens;

[...]
}




So far I've written the code in the Replacer class to split the pattern string into a list of Tokens and extract the Anchor.



public ReplacerMatch[] matches(String string)
{
String pat = this.pattern;

// Get anchor
Anchor anchor = Anchor.fromString(pat);
if (anchor == Anchor.STRING_START)
pat = pat.substring(1);
else if (anchor == Anchor.STRING_END)
pat = pat.substring(0,pat.length() - 1);

// Parse variables
List<Token> vars = new ArrayList<>();
Pattern varPattern = Pattern.compile("%(\\w+)");
Matcher varMatcher = varPattern.matcher(pat);
while (varMatcher.find())
{
for (Token t : this.tokens)
{
if (t.getName().equals(varMatcher.group(1)))
{
vars.add(t);
pat = pat.replace(varMatcher.group(),"%");
varMatcher.reset(pat);
break;
}
}
// Error handling on non-existing token
}

return new ReplacerMatch[0];
}


Now I'm stuck on the matching of the variables, which seems to be quite hard or impossible with regex. Does anybody have an idea how to approach this problem?


Aucun commentaire:

Enregistrer un commentaire