i need to convert a file with a code of a program inside to an xml document as the tokenizer in a compiler would do. for example in a file with the code: if (x < 153) {let city = ”Paris”;} we would get the output: <tokens>

<keyword> if </keyword> 
<symbol> ( </symbol> 
<identifier> x </identifier> 
<symbol> &lt; </symbol> 
<integerConstant> 153 </integerConstant> 
<symbol> ) </symbol> 
<symbol> { </symbol> 
<keyword> let </keyword> 
<identifier> city </identifier> 
<symbol> = </symbol> 
<stringConstant> Paris </stringConstant> 
<symbol> ; </symbol> 
<symbol> } </symbol> 

</tokens> can anyone help me to separate the words of the code to be able to translate to xml?? thanks in advance!!

Well there is lot of source code for tokenizers in the code base you can look at, the most sophisticated (and closed to C/Java like tokenizer) is the compiler::Tokenizer