| Bruce Eckel’s Thinking in Java | Contents | Prev | Next |
Although
StreamTokenizer
is not derived from
InputStream
or
OutputStream,
it works only with
InputStream
objects, so it rightfully belongs in the IO portion of the library.
StreamTokenizer
is not derived from
InputStream
or
OutputStream,
it works only with
InputStream
objects, so it rightfully belongs in the IO portion of the library.
The
StreamTokenizer
class is used to break any
InputStream
into a sequence of
“tokens,” which are bits of text delimited by whatever you choose.
For example, your tokens could be words, and then they would be delimited by
white space and punctuation.
StreamTokenizer
class is used to break any
InputStream
into a sequence of
“tokens,” which are bits of text delimited by whatever you choose.
For example, your tokens could be words, and then they would be delimited by
white space and punctuation.
Consider
a program to count the occurrence of words in a text file:
a program to count the occurrence of words in a text file:
//: SortedWordCount.java // Counts words in a file, outputs // results in sorted form. import java.io.*; import java.util.*; import c08.*; // Contains StrSortVector class Counter { private int i = 1; int read() { return i; } void increment() { i++; } } public class SortedWordCount { private FileInputStream file; private StreamTokenizer st; private Hashtable counts = new Hashtable(); SortedWordCount(String filename) throws FileNotFoundException { try { file = new FileInputStream(filename); st = new StreamTokenizer(file); st.ordinaryChar('.'); st.ordinaryChar('-'); } catch(FileNotFoundException e) { System.out.println( "Could not open " + filename); throw e; } } void cleanup() { try { file.close(); } catch(IOException e) { System.out.println( "file.close() unsuccessful"); } } void countWords() { try { while(st.nextToken() != StreamTokenizer.TT_EOF) { String s; switch(st.ttype) { case StreamTokenizer.TT_EOL: s = new String("EOL"); break; case StreamTokenizer.TT_NUMBER: s = Double.toString(st.nval); break; case StreamTokenizer.TT_WORD: s = st.sval; // Already a String break; default: // single character in ttype s = String.valueOf((char)st.ttype); } if(counts.containsKey(s)) ((Counter)counts.get(s)).increment(); else counts.put(s, new Counter()); } } catch(IOException e) { System.out.println( "st.nextToken() unsuccessful"); } } Enumeration values() { return counts.elements(); } Enumeration keys() { return counts.keys(); } Counter getCounter(String s) { return (Counter)counts.get(s); } Enumeration sortedKeys() { Enumeration e = counts.keys(); StrSortVector sv = new StrSortVector(); while(e.hasMoreElements()) sv.addElement((String)e.nextElement()); // This call forces a sort: return sv.elements(); } public static void main(String[] args) { try { SortedWordCount wc = new SortedWordCount(args[0]); wc.countWords(); Enumeration keys = wc.sortedKeys(); while(keys.hasMoreElements()) { String key = (String)keys.nextElement(); System.out.println(key + ": " + wc.getCounter(key).read()); } wc.cleanup(); } catch(Exception e) { e.printStackTrace(); } } } ///:~
It
makes sense to present these in a sorted form, but since Java 1.0
and Java 1.1
don’t have any sorting methods, that will have to be mixed in. This is
easy enough to do with a
StrSortVector.
(This was created in Chapter 8, and is part of the package created in that
chapter. Remember that the starting directory for all the subdirectories in
this book must be in your class path for the program to compile successfully.)
makes sense to present these in a sorted form, but since Java 1.0
and Java 1.1
don’t have any sorting methods, that will have to be mixed in. This is
easy enough to do with a
StrSortVector.
(This was created in Chapter 8, and is part of the package created in that
chapter. Remember that the starting directory for all the subdirectories in
this book must be in your class path for the program to compile successfully.)
To
open the file, a
FileInputStream
is used, and to turn the file into words a
StreamTokenizer
is created from the
FileInputStream.
In
StreamTokenizer,
there is a default list of separators, and you can add more with a set of
methods. Here,
ordinaryChar( )
is used to say “This character has no significance that I’m
interested in,” so the parser doesn’t include it as part of any of
the words that it creates. For example, saying
st.ordinaryChar(‘.’)
means that periods will not be included as parts of the words that are parsed.
You can find more information in the online documentation that comes with Java.
open the file, a
FileInputStream
is used, and to turn the file into words a
StreamTokenizer
is created from the
FileInputStream.
In
StreamTokenizer,
there is a default list of separators, and you can add more with a set of
methods. Here,
ordinaryChar( )
is used to say “This character has no significance that I’m
interested in,” so the parser doesn’t include it as part of any of
the words that it creates. For example, saying
st.ordinaryChar(‘.’)
means that periods will not be included as parts of the words that are parsed.
You can find more information in the online documentation that comes with Java.
In
countWords( ),
the tokens are pulled one at a time from the stream, and the
ttype
information is used to determine what to do with each token, since a token can
be an end-of-line, a number, a string, or a single character.
countWords( ),
the tokens are pulled one at a time from the stream, and the
ttype
information is used to determine what to do with each token, since a token can
be an end-of-line, a number, a string, or a single character.
Once
a token is found, the Hashtable
counts
is queried to see if it already contains the token as a key. If it does, the
corresponding
Counter
object is incremented to indicate that another instance of this word has been
found. If not, a new
Counter
is created – since the
Counter
constructor initializes its value to one, this also acts to count the word.
a token is found, the Hashtable
counts
is queried to see if it already contains the token as a key. If it does, the
corresponding
Counter
object is incremented to indicate that another instance of this word has been
found. If not, a new
Counter
is created – since the
Counter
constructor initializes its value to one, this also acts to count the word.
SortedWordCount
is not a type of
Hashtable,
so it wasn’t inherited. It performs a specific type of functionality, so
even though the
keys( )
and
values( )
methods must be re-exposed, that still doesn’t mean that inheritance
should be used since a number of
Hashtable
methods are inappropriate here. In addition, other methods like
getCounter( ),
which get the
Counter
for a particular
String,
and
sortedKeys( ),
which produces an
Enumeration,
finish the change in the shape of
SortedWordCount’s
interface.
is not a type of
Hashtable,
so it wasn’t inherited. It performs a specific type of functionality, so
even though the
keys( )
and
values( )
methods must be re-exposed, that still doesn’t mean that inheritance
should be used since a number of
Hashtable
methods are inappropriate here. In addition, other methods like
getCounter( ),
which get the
Counter
for a particular
String,
and
sortedKeys( ),
which produces an
Enumeration,
finish the change in the shape of
SortedWordCount’s
interface.
In
main( )
you can see the use of a
SortedWordCount
to open and count the words in a file – it just takes two lines of code.
Then an enumeration to a sorted list of keys (words) is extracted, and this is
used to pull out each key and associated
Count.
Note that the call to
cleanup( )
is necessary to ensure that the file is closed.
main( )
you can see the use of a
SortedWordCount
to open and count the words in a file – it just takes two lines of code.
Then an enumeration to a sorted list of keys (words) is extracted, and this is
used to pull out each key and associated
Count.
Note that the call to
cleanup( )
is necessary to ensure that the file is closed.
StringTokenizer
Although
it isn’t part of the IO library, the
StringTokenizer
has sufficiently similar functionality to
StreamTokenizer
that it will be described here.
it isn’t part of the IO library, the
StringTokenizer
has sufficiently similar functionality to
StreamTokenizer
that it will be described here.
The
StringTokenizer
returns the tokens within a string one at a time. These tokens are consecutive
characters delimited by tabs, spaces, and newlines. Thus, the tokens of the
string “Where is my cat?” are “Where”,
“is”, “my”, and “cat?” Like the
StreamTokenizer,
you
can tell the
StringTokenizer
to break up the input in any way that you want, but with
StringTokenizer
you
do this by passing a second argument to the constructor, which is a
String
of
the delimiters you wish to use. In general, if you need more sophistication,
use a
StreamTokenizer.
StringTokenizer
returns the tokens within a string one at a time. These tokens are consecutive
characters delimited by tabs, spaces, and newlines. Thus, the tokens of the
string “Where is my cat?” are “Where”,
“is”, “my”, and “cat?” Like the
StreamTokenizer,
you
can tell the
StringTokenizer
to break up the input in any way that you want, but with
StringTokenizer
you
do this by passing a second argument to the constructor, which is a
String
of
the delimiters you wish to use. In general, if you need more sophistication,
use a
StreamTokenizer.
You
ask a
StringTokenizer
object for the next token in the string using the
nextToken( )
method, which either returns the token or an empty string to indicate that no
tokens remain.
ask a
StringTokenizer
object for the next token in the string using the
nextToken( )
method, which either returns the token or an empty string to indicate that no
tokens remain.
As
an example, the following program performs a limited analysis of a sentence,
looking for key phrase sequences to indicate whether happiness or sadness is
implied.
an example, the following program performs a limited analysis of a sentence,
looking for key phrase sequences to indicate whether happiness or sadness is
implied.
//: AnalyzeSentence.java // Look for particular sequences // within sentences. import java.util.*; public class AnalyzeSentence { public static void main(String[] args) { analyze("I am happy about this"); analyze("I am not happy about this"); analyze("I am not! I am happy"); analyze("I am sad about this"); analyze("I am not sad about this"); analyze("I am not! I am sad"); analyze("Are you happy about this?"); analyze("Are you sad about this?"); analyze("It's you! I am happy"); analyze("It's you! I am sad"); } static StringTokenizer st; static void analyze(String s) { prt("nnew sentence >> " + s); boolean sad = false; st = new StringTokenizer(s); while (st.hasMoreTokens()) { String token = next(); // Look until you find one of the // two starting tokens: if(!token.equals("I") && !token.equals("Are")) continue; // Top of while loop if(token.equals("I")) { String tk2 = next(); if(!tk2.equals("am")) // Must be after I break; // Out of while loop else { String tk3 = next(); if(tk3.equals("sad")) { sad = true; break; // Out of while loop } if (tk3.equals("not")) { String tk4 = next(); if(tk4.equals("sad")) break; // Leave sad false if(tk4.equals("happy")) { sad = true; break; } } } } if(token.equals("Are")) { String tk2 = next(); if(!tk2.equals("you")) break; // Must be after Are String tk3 = next(); if(tk3.equals("sad")) sad = true; break; // Out of while loop } } if(sad) prt("Sad detected"); } static String next() { if(st.hasMoreTokens()) { String s = st.nextToken(); prt(s); return s; } else return ""; } static void prt(String s) { System.out.println(s); } } ///:~
For
each string being analyzed, a
while
loop is entered and tokens are pulled off the string. Notice the first
if
statement, which says to
continue
(go back to the beginning of the loop and start again) if the token is neither
an “I” nor an “Are.”
This
means that it will get tokens until an “I” or an “Are”
is found. You might think to use the
==
instead of the equals( )
method, but that won’t work correctly, since
==
compares handle values while
equals( )
compares contents.
each string being analyzed, a
while
loop is entered and tokens are pulled off the string. Notice the first
if
statement, which says to
continue
(go back to the beginning of the loop and start again) if the token is neither
an “I” nor an “Are.”
This
means that it will get tokens until an “I” or an “Are”
is found. You might think to use the
==
instead of the equals( )
method, but that won’t work correctly, since
==
compares handle values while
equals( )
compares contents.
The
logic of the rest of the
analyze( )
method is that the pattern that’s being searched for is “I am
sad,” “I am not happy,” or “Are you sad?” Without
the
break
statement, the code for this would be even messier than it is. You should be
aware that a typical parser (this is a primitive example of one) normally has a
table of these tokens and a piece of code that moves through the states in the
table as new tokens are read.
logic of the rest of the
analyze( )
method is that the pattern that’s being searched for is “I am
sad,” “I am not happy,” or “Are you sad?” Without
the
break
statement, the code for this would be even messier than it is. You should be
aware that a typical parser (this is a primitive example of one) normally has a
table of these tokens and a piece of code that moves through the states in the
table as new tokens are read.
You
should think of the
StringTokenizer
only as shorthand for a simple and specific kind of
StreamTokenizer.
However, if you have a
String
that you want to tokenize and
StringTokenizer
is too limited, all you have to do is turn it into a stream with
StringBufferInputStream
and then use that to create a much more powerful
StreamTokenizer.
should think of the
StringTokenizer
only as shorthand for a simple and specific kind of
StreamTokenizer.
However, if you have a
String
that you want to tokenize and
StringTokenizer
is too limited, all you have to do is turn it into a stream with
StringBufferInputStream
and then use that to create a much more powerful
StreamTokenizer.
