Oh, wow....
I think I just found the real problem...
I was trying your proposal, and thinking about why you couldn't dupe it.
Your fix worked fine if I was using 2.3, but if I short circuited the logic
to treat it as 2.2 (I don't have 2.2 installed at the moment), it blew up
just like before... I had just about decided that __builtin__.compile() was
choking on the file, and that was the problem, so I decided to go about
proving it to me.
I dumped the buf before and after translate_tokens(), to see if inspecting
the file gave me any clues, and it did. A big fat CRLF.
I do some editing on Windows, and some on Linux (vi), as I've previously
mentioned. Most of the stuff I run (quixote-wise, anyway) is run on the
linux box.
To test my theory that __builtin__.compile() was the culprit, I decided to
try it with quixote out of the loop, on a .py file. Finding one that had a
mix of CRLFs and LFs, I did:
>>> b = open('utility.py').read()
>>> import __builtin__
>>> __builtin__.compile(b)
Traceback (most recent call last):
File "", line 1, in ?
TypeError: compile() takes at least 3 arguments (1 given)
>>> __builtin__.compile(b, 'blah', 'exec')
Traceback (most recent call last):
File "", line 1, in ?
File "blah", line 49
def dateFormat2(dt=None):
^
SyntaxError: invalid syntax
>>> import utility
>>>
It imported cleanly, but __builtin__.compile() choked at the first CRLF (at
the end of that line.)
On my Linux box, __builtin__.compile() chokes at the first CR it sees.
Whether it's a (translated) .ptl file, and even if it's a .py file.
If I import a .py file, it works fine, but if I explicity use __compile__,
it blows up.
This problem doesn't exist in my windows version of python.
(Obviously, I need to clean up some files!)
Do we need to do the __builtin__.compile() if it's a SyntaxError? Or is it
only really neccesary when we get a ParserError... I don't really
understand that part.
I tried this next:
>>> b = open('utility.py').read()
>>> import __builtin__
>>> __builtin__.compile(b, 'blah', 'exec')
Traceback (most recent call last):
File "", line 1, in ?
File "blah", line 49
def dateFormat2(dt=None):
^
SyntaxError: invalid syntax
>>> b = b.replace('\r','')
>>> __builtin__.compile(b, 'blah', 'exec')
>>>
Hm... Interesting. If we really need that __builtin__.compile(), I wonder
if we could do something like:
def parse(buf, filename=''):
buf = translate_tokens(buf, filename)
try:
return TemplateTransformer().parsesuite(buf)
except (parser.ParserError, SyntaxError):
import __builtin__
+ if os.name == 'posix': buf = buf.replace('\r', '')
try:
__builtin__.compile(buf, filename, 'exec')
except SyntaxError, exc:
# Another hack to fix the filename attribute (otherwise
# it is often "", even though we specify it).
raise SyntaxError(str(exc), (filename, exc.lineno,
exc.offset,
exc.text))
Whatcha think?
Jason
> -----Original Message-----
> From: Neil Schemenauer [mailto:nascheme@mems-exchange.org]On Behalf Of
> Neil Schemenauer
> Sent: Wednesday, January 21, 2004 3:59 PM
> To: Jason E. Sibre
> Cc: quixote-users@mems-exchange.org
> Subject: Re: [Quixote-users] Debug syntax errors in PTL
>
>
> On Thu, Jan 15, 2004 at 11:17:21PM -0600, Jason E. Sibre wrote:
> > If I have a syntax error in a PTL file, I get an [almost]
> useless exception.
> > For example, I modified a good PTL file (pages.ptl) by removing
> a quote (")
> > from line 38, and fired up python and did:
> >
> > import quixote as q
> > q.enable_ptl()
> > import pages
> >
> > ... and I get the following traceback:
> > Traceback (most recent call last):
>
> > File
> "/usr/local/lib/python2.3/site-packages/quixote/ptl_compile.py", line
> > 229, in parse
> > raise SyntaxError(str(value), (filename, lineno, offset, text))
> > File
> "/home/jsibre/programs/python/mp3/mp3player/web/pages.ptl", line 4
> > import time
> > ^
> > SyntaxError: invalid syntax (pages.ptl, line 4)
> >
> > Line 4? (It's the first non-comment line in the file.)
>
> I can't reproduce this. However, I believe you've run into a real
> problem. The history of this part of Quixote is pretty long. The
> short story is that the 'parser' module in Python <= 2.2 can raise
> SyntaxError or ParserError. ParserError is basically useless since
> it does not include a good description of the error and is missing
> the lineno.
>
> In Python >= 2.3, the 'parser' module does not raise ParserError
> when it means SyntaxError (I'm not 100% sure about this but it seems
> to be true). I don't know why things are going wrong with 2.3, but
> here's my proposed fix:
>
> if sys.hexversion >= 0x20300b1:
> def parse(buf, filename=''):
> buf = translate_tokens(buf, filename)
> return TemplateTransformer().parsesuite(buf)
>
> else:
> # The parser module in Python <= 2.2 can raise ParserError. Since
> # the ParserError exception is basically useless, we use compile()
> # to generate a better exception.
> def parse(buf, filename=''):
> buf = translate_tokens(buf, filename)
> try:
> return TemplateTransformer().parsesuite(buf)
> except (parser.ParserError, SyntaxError):
> import __builtin__
> try:
> __builtin__.compile(buf, filename, 'exec')
> except SyntaxError, exc:
> # Another hack to fix the filename attribute (otherwise
> # it is often "", even though we specify it).
> raise SyntaxError(str(exc), (filename,
> exc.lineno, exc.offset,
> exc.text))
>
>
> Everyting in the 'else' clause is for backwards compatibility only. It
> can eventually go away.
>
> Neil
>