Multiple buffers
Previous: <Start conditions=>Startcondi> * Next: <End-of-file rules=>Endoffiler> * Up: <Top=>!Root>

#Wrap on
{fH3}Multiple input buffers{f}

Some scanners (such as those which support "include"
files) require reading from several input streams.  As
{fCode}flex{f} scanners do a large amount of buffering, one cannot
control where the next input will be read from by simply
writing a {fCode}YY\_INPUT{f} which is sensitive to the scanning
context.  {fCode}YY\_INPUT{f} is only called when the scanner reaches
the end of its buffer, which may be a long time after
scanning a statement such as an "include" which requires
switching the input source.

To negotiate these sorts of problems, {fCode}flex{f} provides a
mechanism for creating and switching between multiple
input buffers.  An input buffer is created by using:

#Wrap off
#fCode
YY\_BUFFER\_STATE yy\_create\_buffer( FILE \*file, int size )
#f
#Wrap on

which takes a {fCode}FILE{f} pointer and a size and creates a buffer
associated with the given file and large enough to hold
{fStrong}size{f} characters (when in doubt, use {fCode}YY\_BUF\_SIZE{f} for the
size).  It returns a {fCode}YY\_BUFFER\_STATE{f} handle, which may
then be passed to other routines (see below).  The
{fCode}YY\_BUFFER\_STATE{f} type is a pointer to an opaque {fCode}struct{f}
{fCode}yy\_buffer\_state{f} structure, so you may safely initialize
YY\_BUFFER\_STATE variables to {fEmphasis}((YY\_BUFFER\_STATE) 0){f} if you
wish, and also refer to the opaque structure in order to
correctly declare input buffers in source files other than
that of your scanner.  Note that the {fCode}FILE{f} pointer in the
call to {fCode}yy\_create\_buffer{f} is only used as the value of {fCode}yyin{f}
seen by {fCode}YY\_INPUT{f}; if you redefine {fCode}YY\_INPUT{f} so it no longer
uses {fCode}yyin{f}, then you can safely pass a nil {fCode}FILE{f} pointer to
{fCode}yy\_create\_buffer{f}.  You select a particular buffer to scan
from using:

#Wrap off
#fCode
void yy\_switch\_to\_buffer( YY\_BUFFER\_STATE new\_buffer )
#f
#Wrap on

switches the scanner's input buffer so subsequent tokens
will come from {fStrong}new\_buffer{f}.  Note that
{fEmphasis}yy\_switch\_to\_buffer(){f} may be used by {fEmphasis}yywrap(){f} to set
things up for continued scanning, instead of opening a new
file and pointing {fCode}yyin{f} at it.  Note also that switching
input sources via either {fEmphasis}yy\_switch\_to\_buffer(){f} or {fEmphasis}yywrap(){f}
does {fEmphasis}not{f} change the start condition.

#Wrap off
#fCode
void yy\_delete\_buffer( YY\_BUFFER\_STATE buffer )
#f
#Wrap on

is used to reclaim the storage associated with a buffer.
You can also clear the current contents of a buffer using:

#Wrap off
#fCode
void yy\_flush\_buffer( YY\_BUFFER\_STATE buffer )
#f
#Wrap on

This function discards the buffer's contents, so the next time the
scanner attempts to match a token from the buffer, it will first fill
the buffer anew using {fCode}YY\_INPUT{f}.

{fEmphasis}yy\_new\_buffer(){f} is an alias for {fEmphasis}yy\_create\_buffer(){f},
provided for compatibility with the C++ use of {fCode}new{f} and {fCode}delete{f}
for creating and destroying dynamic objects.

Finally, the {fCode}YY\_CURRENT\_BUFFER{f} macro returns a
{fCode}YY\_BUFFER\_STATE{f} handle to the current buffer.

Here is an example of using these features for writing a
scanner which expands include files (the {fEmphasis}<<EOF>>{f} feature
is discussed below):

#Wrap off
#fCode
\/\* the "incl" state is used for picking up the name
 \* of an include file
 \*\/
%x incl

%\{
\#define MAX\_INCLUDE\_DEPTH 10
YY\_BUFFER\_STATE include\_stack[MAX\_INCLUDE\_DEPTH];
int include\_stack\_ptr = 0;
%\}

%%
include             BEGIN(incl);

[a-z]+              ECHO;
[^a-z\\n]\*\\n?        ECHO;

<incl>[ \\t]\*      \/\* eat the whitespace \*\/
<incl>[^ \\t\\n]+   \{ \/\* got the include file name \*\/
        if ( include\_stack\_ptr >= MAX\_INCLUDE\_DEPTH )
            \{
            fprintf( stderr, "Includes nested too deeply" );
            exit( 1 );
            \}

        include\_stack[include\_stack\_ptr++] =
            YY\_CURRENT\_BUFFER;

        yyin = fopen( yytext, "r" );

        if ( ! yyin )
            error(  );

        yy\_switch\_to\_buffer(
            yy\_create\_buffer( yyin, YY\_BUF\_SIZE ) );

        BEGIN(INITIAL);
        \}

<<EOF>> \{
        if ( --include\_stack\_ptr < 0 )
            \{
            yyterminate();
            \}

        else
            \{
            yy\_delete\_buffer( YY\_CURRENT\_BUFFER );
            yy\_switch\_to\_buffer(
                 include\_stack[include\_stack\_ptr] );
            \}
        \}
#f
#Wrap on

Three routines are available for setting up input buffers
for scanning in-memory strings instead of files.  All of
them create a new input buffer for scanning the string,
and return a corresponding {fCode}YY\_BUFFER\_STATE{f} handle (which
you should delete with {fEmphasis}yy\_delete\_buffer(){f} when done with
it).  They also switch to the new buffer using
{fEmphasis}yy\_switch\_to\_buffer(){f}, so the next call to {fEmphasis}yylex(){f} will
start scanning the string.

#Indent +4
#Indent
{fEmphasis}yy\_scan\_string(const char \*str){f}
#Indent +4
scans a NUL-terminated string.

#Indent
{fEmphasis}yy\_scan\_bytes(const char \*bytes, int len){f}
#Indent +4
scans {fCode}len{f} bytes (including possibly NUL's) starting
at location {fStrong}bytes{f}.

#Indent

Note that both of these functions create and scan a {fEmphasis}copy{f}
of the string or bytes.  (This may be desirable, since
{fEmphasis}yylex(){f} modifies the contents of the buffer it is
scanning.) You can avoid the copy by using:

#Indent +4
#Indent
{fEmphasis}yy\_scan\_buffer(char \*base, yy\_size\_t size){f}
#Indent +4
which scans in place the buffer starting at {fStrong}base{f},
consisting of {fStrong}size{f} bytes, the last two bytes of
which {fEmphasis}must{f} be {fCode}YY\_END\_OF\_BUFFER\_CHAR{f} (ASCII NUL).
These last two bytes are not scanned; thus,
scanning consists of {fEmphasis}base[0]{f} through {fEmphasis}base[size-2]{f},
inclusive.

If you fail to set up {fStrong}base{f} in this manner (i.e.,
forget the final two {fCode}YY\_END\_OF\_BUFFER\_CHAR{f} bytes),
then {fEmphasis}yy\_scan\_buffer(){f} returns a nil pointer instead
of creating a new input buffer.

The type {fCode}yy\_size\_t{f} is an integral type to which you
can cast an integer expression reflecting the size
of the buffer.

#Indent

