External module "nlp/tokenize/docs/types/index.d"

Export assignment tokenize

• tokenize(str: string, keepWhitespace?: undefined | false | true): Array<string>

• Tokenize a string.

  Parameters

  • str: string

    input string

  • Optional keepWhitespace: undefined | false | true

    boolean indicating whether whitespace characters should be returned as part of the token array (default: false)

  Returns Array < string >

  array of tokens

  Example

  var str = 'Hello World!';
  var out = tokenize( str );
  // returns [ 'Hello', 'World', '!' ]

  Example

  var str = '';
  var out = tokenize( str );
  // returns []

  Example

  var str = 'Hello Mrs. Maple, could you call me back?';
  var out = tokenize( str );
  // returns [ 'Hello', 'Mrs.', 'Maple', ',', 'could', 'you', 'call', 'me', 'back', '?' ]