Class RegexTokenizer

java.lang.Object

org.apache.commons.text.similarity.RegexTokenizer

All Implemented Interfaces:

Function<CharSequence, CharSequence[]>, CharSequenceTokenizer<CharSequence>, Tokenizer<CharSequence, CharSequence>

final class RegexTokenizer
extends Object
implements CharSequenceTokenizer<CharSequence>

A simple word Tokenizer that utilizes a regex to find words. It applies a regex (\w)+ over the input text to extract words from a given character sequence.

Instances of this class are immutable and are safe for use by multiple concurrent threads.

Since:

1.0

Field Summary

Fields

Modifier and Type	Field	Description
(package private) static final RegexTokenizer	INSTANCE	The singleton instance.
private static final Pattern	PATTERN	The whitespace pattern.

Constructor Summary

Constructors

Constructor	Description
RegexTokenizer()

Method Summary

Modifier and Type	Method	Description
CharSequence[]	apply(CharSequence text)

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface Function

andThen, compose

Field Details

PATTERN

private static final Pattern PATTERN

The whitespace pattern.

INSTANCE

static final RegexTokenizer INSTANCE

The singleton instance.

Constructor Details

RegexTokenizer

RegexTokenizer()

Method Details

apply

public CharSequence[] apply(CharSequence text)

Specified by:

apply in interface Function<CharSequence, CharSequence[]>

Throws:

IllegalArgumentException - if the input text is blank.