flex: doubt regarding the meaning of some character classes

After browsing the flex manual I've stumbled on the patterns section 
where it is said that patterns can be expressed with the help of 
character class expressions. The patterns are as follows:

[:alnum:] [:alpha:] [:blank:]
[:cntrl:] [:digit:] [:graph:]
[:lower:] [:print:] [:punct:]
[:space:] [:upper:] [:xdigit:]

Some of those patterns appear to be straight forward but I don't have a 
clue regarding the exact meaning of a whole bunch of them. To make 
matters worse, it appears that there is absolutely no information 
covering that, not only in the flex manual but also anywhere.

So, could anyone please post the meaning of those character classes?


Thanks in advance
Rui Maciel

On 19 Oct 2008 21:02:20 GMT,



The POSIX character classes all correspond to functions of the same name
with 'is' prepended. So, the character class [:isdigit:] has a
corresponding C function isdigit(), which you can look up for the
definition.

Martien
-- 
                        | 
Martien Verbruggen      | 
                        | Curiouser and curiouser, said Alice.
                        | 

On Oct 20, 12:37 am, Martien Verbruggen < XXXX@XXXXX.COM >








Which doesn't really answer much.  The C functions are locale
dependent; is this also true of the flex character classes?  And
if so, which locale: the one when flex is run, or the one when
the generated program is run?  (I can make some guesses, but
this is the sort of thing which absolutely must be documented
for any serious program.)

James Kanze (GABI Software)             email: XXXX@XXXXX.COM 
Conseils en informatique orient objet/
                   Beratung in objektorientierter Datenverarbeitung
9 place Sard, 78210 St.-Cyr-l'ole, France, +33 (0)1 30 23 00 34

The flex info manual (that accompanies the distribution) is, I believe,
completely clear about it (the flex character classes are locale
dependent and depend on the locale in which flex is run) - see the
chapter entitled "Patterns".

Robert

It seems to say that, but when I look at the generated code, it has only
a table for POSIX codes.

-- 
Thomas E.{*filter*}ey
 http://www.**--****.com/ 
ftp://invisible-island.net

Thanks, Martien! Knowing that those character classes were defined in the 
POSIX standards did the trick. After a quick search I even stumbled on 
the following wikipedia article:

 http://www.**--****.com/ #POSIX_character_classes

It has a nifty table that lists both the expanded regex expression and 
the general description of that pattern. 


Once again thanks, Martien. Kudos!
Rui Maciel

On Oct 20, 12:42 pm, Robert Harris < XXXX@XXXXX.COM >













Yes.  That makes sense, but it does mean that if you deliver
flex source code, you need to warn your users about this,
possibly specifying that they must use the correct locale (or
setting it in your makefile).

On the other hand, the isxxx functions aren't really of much use
today, now that everything is UTF-8.  Not sure how flex handles
this issue, however.

James Kanze (GABI Software)             email: XXXX@XXXXX.COM 
Conseils en informatique orient objet/
                   Beratung in objektorientierter Datenverarbeitung
9 place Sard, 78210 St.-Cyr-l'ole, France, +33 (0)1 30 23 00 34

On Oct 20, 4:56 pm,  XXXX@XXXXX.COM  (Thomas E.{*filter*}ey)






So you ran flex in the POSIX locale.  The manual is actually
very precise about this, presenting it as a warning.

James Kanze (GABI Software)             email: XXXX@XXXXX.COM 
Conseils en informatique orient objet/
                   Beratung in objektorientierter Datenverarbeitung
9 place Sard, 78210 St.-Cyr-l'ole, France, +33 (0)1 30 23 00 34

Perhaps you can quote the relevant paragraph, so we can agree on what it says.

-- 
Thomas E.{*filter*}ey
 http://www.**--****.com/ 
ftp://invisible-island.net

hmm - "info", from "new" flex (new bugs, no improvements ;-).

I hadn't looked closely at the documentation, see that it does put
that in the "info" file.  Perhaps someone will improve that aspect,
sometime.

-- 
Thomas E.{*filter*}ey
 http://www.**--****.com/ 
ftp://invisible-island.net

On Oct 23, 5:29 pm,  XXXX@XXXXX.COM  (Thomas E.{*filter*}ey)











    A word of caution. Character classes are expanded
    immediately when seen in the flex input. This means the
    character classes are sensitive to the locale in which
    flex is executed, and the resulting scanner will not be
    sensitive to the runtime locale. This may or may not be
    desirable.

Exactly where Robert said I'd find it, in the section on
Patterns.  Immediately after the presentation of [:...:].

James Kanze (GABI Software)             email: XXXX@XXXXX.COM 
Conseils en informatique orient objet/
                   Beratung in objektorientierter Datenverarbeitung
9 place Sard, 78210 St.-Cyr-l'ole, France, +33 (0)1 30 23 00 34

....

However - back to my comment (documentation is one thing, reality
another).

The note in (flex "2.5.35" for the sake of discussion) does not
correspond to
what I'm seeing.  A simple lex with just this pattern

WORD            [[:alpha:]]([[:alnum:]])*

produces the same file whether my current locale is "C" or "en_US" or
"fr_FR".
The relevant information for character classes appears to be in a
table
beginning

static yyconst flex_int32_t yy_ec[256] =

(The same observation applies to "2.5.4a").

Thomas E.{*filter*}ey < XXXX@XXXXX.COM >
 http://www.**--****.com/ 
ftp://invisible-island.net

> produces the same file whether my current locale is "C">or
> "en_US" or "fr_FR".

I'd rank that as a bug then.

James Kanze (GABI Software)             email: XXXX@XXXXX.COM 
Conseils en informatique orient objet/
                   Beratung in objektorientierter Datenverarbeitung
9 place Sard, 78210 St.-Cyr-l'ole, France, +33 (0)1 30 23 00 34

I agree (and it would be nice to say that I reported it to the developers,
but having tried to do so with _these_ particular developers before,
I'm not inclined to do that again).

It's a trivial thing to find/fix (and it's not possible that it could
have worked as documented in flex 2.5.35 - perhaps it'll be fixed in
2.5.36, assuming that the developers read this newsgroup).

Since it's related to some work I've been contemplating for vile,
I did the fix to "old flex", and made that available here:

	 http://www.**--****.com/ 

(it's the change related to "isascii" ;-)

-- 
Thomas E.{*filter*}ey
 http://www.**--****.com/ 
ftp://invisible-island.net