Okay, Oleg's prior post (Jul 01, 2003) inspired me to look again at the
"Sytax errors during PTL import" problem.  So, I started digging around in
the dark and scary corners of PTL importing again, and I think I've found
something, which I'll hold up for the experts to take a look at.


First, a recap of the problem:

If I have a syntax error in a PTL file, I get an [almost] useless exception.
For example, I modified a good PTL file (pages.ptl) by removing a quote (")
from line 38, and fired up python and did:

import quixote as q
q.enable_ptl()
import pages

... and I get the following traceback:
Traceback (most recent call last):
  File "./test.py", line 6, in ?
    import pages
  File "/usr/local/lib/python2.3/ihooks.py", line 398, in import_module
    q, tail = self.find_head_package(parent, str(name))
  File "/usr/local/lib/python2.3/ihooks.py", line 434, in find_head_package
    q = self.import_it(head, qname, parent)
  File "/usr/local/lib/python2.3/ihooks.py", line 489, in import_it
    m = self.loader.load_module(fqname, stuff)
  File "/usr/local/lib/python2.3/site-packages/quixote/ptl_import.py", line
107, in load_module
    return _load_ptl(name, filename, file)
  File "/usr/local/lib/python2.3/site-packages/quixote/ptl_import.py", line
72, in _load_ptl
    code = compile_template(file, filename, output)
  File "/usr/local/lib/python2.3/site-packages/quixote/ptl_compile.py", line
292, in compile_template
    template.compile()
  File "/usr/local/lib/python2.3/compiler/pycodegen.py", line 111, in
compile
    tree = self._get_tree()
  File "/usr/local/lib/python2.3/site-packages/quixote/ptl_compile.py", line
263, in _get_tree
    tree = parse(self.source, self.filename)
  File "/usr/local/lib/python2.3/site-packages/quixote/ptl_compile.py", line
229, in parse
    raise SyntaxError(str(value), (filename, lineno, offset, text))
  File "/home/jsibre/programs/python/mp3/mp3player/web/pages.ptl", line 4
    import time
               ^
SyntaxError: invalid syntax (pages.ptl, line 4)

Line 4?  (It's the first non-comment line in the file.)

So I start working my way up the traceback...  Actually I went through the
modules first to get a feel for them, but I'm probably verbose enough
without boring you with all those details!

In ptl_compile.py, there is a function called parse, which I've pasted below
(it's not very big) with line numbers added (from it's position in the file)
to facilitate commentary.  I'm using the version from .7a2, but I did check
that this file is identical to the .7a3 version.

[ptl_compile.py]
214 def parse(buf, filename=''):
215     buf = translate_tokens(buf, filename)
216     try:
217         return TemplateTransformer().parsesuite(buf)
218     except (parser.ParserError, SyntaxError):
219         # Let Python compiler generate a better error message
220         import __builtin__, sys
221         try:
222             __builtin__.compile(buf, filename, 'exec')
223         except SyntaxError, value:
224             # Python does not use filename in the exception but stupidly
225             # uses "" instead.  Patch it up ourself.
226             lineno = value.lineno
227             offset = value.offset
228             text = value.text
229             raise SyntaxError(str(value), (filename, lineno, offset,
text))
230         raise # hmm, the Python compiler didn't raise an exception!?

Now I don't understand the finer details of what's being done here, but it
looks pretty obvious that lines 215 and 217 are the work horses, and the
rest is dealing with what to do in the event of an error, and mostly just to
plug the filename into the exception.  So, in true hack fashion, I started
playing with things, and simplified it to:


def parse(buf, filename=''):
    buf = translate_tokens(buf, filename)
    try:
        return TemplateTransformer().parsesuite(buf)
    except parser.ParserError, value:
        raise parser.ParserError(str(value), (filename, value.lineno,
value.offset, value.text))
    except SyntaxError, value:
        raise SyntaxError(str(value), (filename, value.lineno, value.offset,
value.text))

Now, my test case yields the following traceback:
Traceback (most recent call last):
  File "./test.py", line 6, in ?
    import pages
  File "/usr/local/lib/python2.3/ihooks.py", line 398, in import_module
    q, tail = self.find_head_package(parent, str(name))
  File "/usr/local/lib/python2.3/ihooks.py", line 434, in find_head_package
    q = self.import_it(head, qname, parent)
  File "/usr/local/lib/python2.3/ihooks.py", line 489, in import_it
    m = self.loader.load_module(fqname, stuff)
  File "/usr/local/lib/python2.3/site-packages/quixote/ptl_import.py", line
107, in load_module
    return _load_ptl(name, filename, file)
  File "/usr/local/lib/python2.3/site-packages/quixote/ptl_import.py", line
72, in _load_ptl
    code = compile_template(file, filename, output)
  File "/usr/local/lib/python2.3/site-packages/quixote/ptl_compile.py", line
292, in compile_template
    template.compile()
  File "/usr/local/lib/python2.3/compiler/pycodegen.py", line 111, in
compile
    tree = self._get_tree()
  File "/usr/local/lib/python2.3/site-packages/quixote/ptl_compile.py", line
263, in _get_tree
    tree = parse(self.source, self.filename)
  File "/usr/local/lib/python2.3/site-packages/quixote/ptl_compile.py", line
238, in parse
    raise SyntaxError(str(value), (filename, value.lineno, value.offset,
value.text))
  File "/home/jsibre/programs/python/mp3/mp3player/web/pages.ptl", line 38
    self.query_start = int(request.form.get("query_start))
                                                         ^
SyntaxError: EOL while scanning single-quoted string (line 38)



Hey!!  That looks WAY better!  I can actually troubleshoot with that.  Heck,
I don't have to.  It points me right at the error :)  I can fix my file in
15 seconds, instead of 15 minutes of cutting, pasting, and running, trying
to narrow the infected region down to a small enough area to visually
inspect looking for something that's outta whack.

Question to the experts is:   I don't know what that parser.ParserError is,
or know if it needs to be worried about.  Does this change have any obvious
potential to break things?  The exception message at the end of the
traceback is a bit different, but I think being given the correct line
number (by the exception string) and looking at the traceback to get the
filename, vs. being given the filename and a bogus line number (by the
exception string) is a worthwhile tradeoff.  Anything else I should know
about before mucking around in that area?

If it looks ok, the ptl_compile.py patch is attached.  I can provide an
easily used test case on request (I don't have something packagable ready at
this time, or I'd just attach it.)

My main concern is that my test is hardly comprehensive, and I may be
missing something important due to my lack of understanding of the whole
import & compile process.


Jason Sibre