Parser.Pike.tokenize() truncates when the input contains unterminated strings or character constants.
Eg:
> Parser.Pike.split("foo'bar");
(1) Result: ({ /* 2 elements */
"foo",
"\n"
})
Eg:
> Parser.Pike.split("foo'bar");
(1) Result: ({ /* 2 elements */
"foo",
"\n"
})