<!-- MHonArc v2.4.4 -->
<!--X-Subject: Tutorial: Let's build a Compiler! - Part XVI: Unit Construction -->
<!--X-From-R13: "Xba O. Znzoreg" <wyflfvapNvk.argpbz.pbz> -->
<!--X-Date: Thu, 05 Mar 1998 04:23:28 +0000 -->
<!--X-Message-Id: 000c01bd47ee$dbcbd880$f53bd8ce@default -->
<!--X-Content-Type: text/plain -->
<!--X-Head-End-->
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<html>
<head>
<title>MUD-Dev message, Tutorial: Let's build a Compiler! - Part XVI: Unit Constructio</title>
<!-- meta name="robots" content="noindex,nofollow" -->
<link rev="made" href="mailto:jlsysinc#ix,netcom.com">
</head>
<body background="/backgrounds/paperback.gif" bgcolor="#ffffff"
text="#000000" link="#0000FF" alink="#FF0000" vlink="#006000">
<font size="+4" color="#804040">
<strong><em>MUD-Dev<br>mailing list archive</em></strong>
</font>
<br>
[ <a href="../">Other Periods</a>
| <a href="../../">Other mailing lists</a>
| <a href="/search.php3">Search</a>
]
<br clear=all><hr>
<!--X-Body-Begin-->
<!--X-User-Header-->
<!--X-User-Header-End-->
<!--X-TopPNI-->
Date:
[ <a href="msg00684.html">Previous</a>
| <a href="msg00686.html">Next</a>
]
Thread:
[ <a href="msg00686.html">Previous</a>
| <a href="msg00684.html">Next</a>
]
Index:
[ <A HREF="author.html#00685">Author</A>
| <A HREF="#00685">Date</A>
| <A HREF="thread.html#00685">Thread</A>
]
<!--X-TopPNI-End-->
<!--X-MsgBody-->
<!--X-Subject-Header-Begin-->
<H1>Tutorial: Let's build a Compiler! - Part XVI: Unit Construction</H1>
<HR>
<!--X-Subject-Header-End-->
<!--X-Head-of-Message-->
<UL>
<LI><em>To</em>: <<A HREF="mailto:mud-dev#null,net">mud-dev#null,net</A>></LI>
<LI><em>Subject</em>: Tutorial: Let's build a Compiler! - Part XVI: Unit Construction</LI>
<LI><em>From</em>: "Jon A. Lambert" <<A HREF="mailto:jlsysinc#ix,netcom.com">jlsysinc#ix,netcom.com</A>></LI>
<LI><em>Date</em>: Wed, 4 Mar 1998 23:23:53 -0500</LI>
</UL>
<!--X-Head-of-Message-End-->
<!--X-Head-Body-Sep-Begin-->
<HR>
<!--X-Head-Body-Sep-End-->
<!--X-Body-of-Message-->
<PRE>
LET'S BUILD A COMPILER!
By
Jack W. Crenshaw, Ph.D.
29 May, 1995
Part 16: UNIT CONSTRUCTION
*****************************************************************
* *
* COPYRIGHT NOTICE *
* *
* Copyright (C) 1995 Jack W. Crenshaw. All rights reserved. *
* *
*****************************************************************
INTRODUCTION
This series of tutorials promises to be perhaps one of the longest-
running mini-series in history, rivalled only by the delay in Volume IV
of Knuth. Begun in 1988, the series ran into a four-year hiatus in 1990
when the "cares of this world," changes in priorities and interests, and
the need to make a living seemed to stall it out after Installment 14.
Those of you with loads of patience were finally rewarded, in the spring
of last year, with the long-awaited Installment 15. In it, I began to
try to steer the series back on track, and in the process, to make it
easier to continue on to the goal, which is to provide you with not only
enough understanding of the difficult subject of compiler theory, but
also enough tools, in the form of canned subroutines and concepts, so
that you would be able to continue on your own and become proficient
enough to build your own parsers and translators. Because of that long
hiatus, I thought it appropriate to go back and review the concepts we
have covered so far, and to redo some of the software, as well. In the
past, we've never concerned ourselves much with the development of
production-quality software tools ... after all, I was trying to teach
(and learn) concepts, not production practice. To do that, I tended to
give you, not complete compilers or parsers, but only those snippets of
code that illustrated the particular point we were considering at the
moment.
I still believe that's a good way to learn any subject; no one wants to
have to make changes to 100,000 line programs just to try out a new
idea. But the idea of just dealing with code snippets, rather than
complete programs, also has its drawbacks in that we often seemed to be
writing the same code fragments over and over. Although repetition has
been thoroughly proven to be a good way to learn new ideas, it's also
true that one can have too much of a good thing. By the time I had
completed Installment 14 I seemed to have reached the limits of my
abilities to juggle multiple files and multiple versions of the same
software functions. Who knows, perhaps that's one reason I seemed to
have run out of gas at that point.
Fortunately, the later versions of Borland's Turbo Pascal allow us to
have our cake and eat it too. By using their concept of separately
compilable units, we can still write small subroutines and functions,
and keep our main programs and test programs small and simple. But,
once written, the code in the Pascal units will always be there for us
to use, and linking them in is totally painless and transparent.
Since, by now, most of you are programming in either C or C++, I know
what you're thinking: Borland, with their Turbo Pascal (TP), certainly
didn't invent the concept of separately compilable modules. And of
course you're right. But if you've not used TP lately, or ever, you may
not realize just how painless the whole process is. Even in C or C++,
you still have to build a make file, either manually or by telling the
compiler how to do so. You must also list, using "extern" statements or
header files, the functions you want to import. In TP, you don't even
have to do that. You need only name the units you wish to use, and all
of their procedures automatically become available.
It's not my intention to get into a language-war debate here, so I won't
pursue the subject any further. Even I no longer use Pascal on my job
... I use C at work and C++ for my articles in Embedded Systems
Programming and other magazines. Believe me, when I set out to
resurrect this series, I thought long and hard about switching both
languages and target systems to the ones that we're all using these
days, C/C++ and PC architecture, and possibly object-oriented methods as
well. In the end, I felt it would cause more confusion than the hiatus
itself has. And after all, Pascal still remains one of the best possible
languages for teaching, not to mention production programming. Finally,
TP still compiles at the speed of light, much faster than competing
C/C++ compilers. And Borland's smart linker, used in TP but not in their
C++ products, is second to none. Aside from being much faster than
Microsoft-compatible linkers, the Borland smart linker will cull unused
procedures and data items, even to the extent of trimming them out of
defined objects if they're not needed. For one of the few times in our
lives, we don't have to compromise between completeness and efficiency.
When we're writing a TP unit, we can make it as complete as we like,
including any member functions and data items we may think we will ever
need, confident that doing so will not create unwanted bloat in the
compiled and linked executable.
The point, really, is simply this: By using TP's unit mechanism, we can
have all the advantages and convenience of writing small, seemingly
stand-alone test programs, without having to constantly rewrite the
support functions that we need. Once written, the TP units sit there,
quietly waiting to do their duty and give us the support we need, when
we need it.
Using this principle, in Installment 15 I set out to minimize our
tendency to re-invent the wheel by organizing our code into separate
Turbo Pascal units, each containing different parts of the compiler. We
ended up with the following units:
* Input
* Output
* Errors
* Scanner
* Parser
* CodeGen
Each of these units serves a different function, and encapsulates
specific areas of functionality. The Input and Output units, as their
name implies, provide character stream I/O and the all-important
lookahead character upon which our predictive parser is based. The
Errors unit, of course, provides standard error handling. The Scanner
unit contains all of our boolean functions such as IsAlpha, and the
routines GetName and GetNumber, which process multi-character tokens.
The two units we'll be working with the most, and the ones that most
represent the personality of our compiler, are Parser and CodeGen.
Theoretically, the Parser unit should encapsulate all aspects of the
compiler that depend on the syntax of the compiled language (though, as
we saw last time, a small amount of this syntax spills over into
Scanner). Similarly, the code generator unit, CodeGen, contains all of
the code dependent upon the target machine. In this installment, we'll
be continuing with the development of the functions in these two all-
important units.
JUST LIKE CLASSICAL?
Before we proceed, however, I think I should clarify the relationship
between, and the functionality of these units. Those of you who are
familiar with compiler theory as taught in universities will, of course,
recognize the names, Scanner, Parser, and CodeGen, all of which are
components of a classical compiler implementation. You may be thinking
that I've abandoned my commitment to the KISS philosophy, and drifted
towards a more conventional architecture than we once had. A closer
look, however, should convince you that, while the names are similar,
the functionalities are quite different.
Together, the scanner and parser of a classical implementation comprise
the so-called "front end," and the code generator, the back end. The
front end routines process the language-dependent, syntax-related
aspects of the source language, while the code generator, or back end,
deals with the target machine-dependent parts of the problem. In
classical compilers, the two ends communicate via a file of instructions
written in an intermediate language (IL).
Typically, a classical scanner is a single procedure, operating as a co-
procedure with the parser. It "tokenizes" the source file, reading it
character by character, recognizing language elements, translating them
into tokens, and passing them along to the parser. You can think of the
parser as an abstract machine, executing "op codes," which are the
tokens. Similarly, the parser generates op codes of a second abstract
machine, which mechanizes the IL. Typically, the IL file is written to
disk by the parser, and read back again by the code generator.
Our organization is quite different. We have no lexical scanner, in the
classical sense; our unit Scanner, though it has a similar name, is not
a single procedure or co-procedure, but merely a set of separate
subroutines which are called by the parser as needed.
Similarly, the classical code generator, the back end, is a translator
in its own right, reading an IL "source" file, and emitting an object
file. Our code generator doesn't work that way. In our compiler, there
IS no intermediate language; every construct in the source language
syntax is converted into assembly language as it is recognized by the
parser. Like Scanner, the unit CodeGen consists of individual
procedures which are called by the parser as needed.
This "code 'em as you find 'em" philosophy may not produce the world's
most efficient code -- for example, we haven't provided (yet!) a
convenient place for an optimizer to work its magic -- but it sure does
simplify the compiler, doesn't it?
And that observation prompts me to reflect, once again, on how we have
managed to reduce a compiler's functions to such comparatively simple
terms. I've waxed eloquent on this subject in past installments, so I
won't belabor the point too much here. However, because of the time
that's elapsed since those last soliloquies, I hope you'll grant me just
a little time to remind myself, as well as you, how we got here. We got
here by applying several principles that writers of commercial compilers
seldom have the luxury of using. These are:
o The KISS philosophy -- Never do things the hard way without a
reason
o Lazy coding -- Never put off until tomorrow what you can put
of forever (with credits to P.J. Plauger)
o Skepticism -- Stubborn refusal to do something just because
that's the way it's always been done.
o Acceptance of inefficient code
o Rejection of arbitrary constraints
As I've reviewed the history of compiler construction, I've learned that
virtually every production compiler in history has suffered from pre-
imposed conditions that strongly influenced its design. The original
FORTRAN compiler of John Backus, et al, had to compete with assembly
language, and therefore was constrained to produce extremely efficient
code. The IBM compilers for the minicomputers of the 70's had to run in
the very small RAM memories then available -- as small as 4k. The early
Ada compiler had to compile itself. Per Brinch Hansen decreed that his
Pascal compiler developed for the IBM PC must execute in a 64k machine.
Compilers developed in Computer Science courses had to compile the
widest variety of languages, and therefore required LALR parsers.
In each of these cases, these preconceived constraints literally
dominated the design of the compiler.
A good example is Brinch Hansen's compiler, described in his excellent
book, "Brinch Hansen on Pascal Compilers" (highly recommended). Though
his compiler is one of the most clear and un-obscure compiler
implementations I've seen, that one decision, to compile large files in
a small RAM, totally drives the design, and he ends up with not just
one, but many intermediate files, together with the drivers to write and
read them.
In time, the architectures resulting from such decisions have found
their way into computer science lore as articles of faith. In this one
man's opinion, it's time that they were re-examined critically. The
conditions, environments, and requirements that led to classical
architectures are not the same as the ones we have today. There's no
reason to believe the solutions should be the same, either.
In this tutorial, we've followed the leads of such pioneers in the world
of small compilers for Pcs as Leor Zolman, Ron Cain, and James Hendrix,
who didn't know enough compiler theory to know that they "couldn't do it
that way." We have resolutely refused to accept arbitrary constraints,
but rather have done whatever was easy. As a result, we have evolved an
architecture that, while quite different from the classical one, gets
the job done in very simple and straightforward fashion.
I'll end this philosophizing with an observation re the notion of an
intermediate language. While I've noted before that we don't have one
in our compiler, that's not exactly true; we _DO_ have one, or at least
are evolving one, in the sense that we are defining code generation
functions for the parser to call. In essence, every call to a code
generation procedure can be thought of as an instruction in an
intermediate language. Should we ever find it necessary to formalize an
intermediate language, this is the way we would do it: emit codes from
the parser, each representing a call to one of the code generator
procedures, and then process each code by calling those procedures in a
separate pass, implemented in a back end. Frankly, I don't see that
we'll ever find a need for this approach, but there is the connection,
if you choose to follow it, between the classical and the current
approaches.
FLESHING OUT THE PARSER
Though I promised you, somewhere along about Installment 14, that we'd
never again write every single function from scratch, I ended up
starting to do just that in Installment 15. One reason: that long
hiatus between the two installments made a review seem eminently
justified ... even imperative, both for you and for me. More
importantly, the decision to collect the procedures into modules
(units), forced us to look at each one yet again, whether we wanted to
or not. And, finally and frankly, I've had some new ideas in the last
four years that warranted a fresh look at some old friends. When I
first began this series, I was frankly amazed, and pleased, to learn
just how simple parsing routines can be made. But this last time
around, I've surprised myself yet again, and been able to make them just
that last little bit simpler, yet.
Still, because of this total rewrite of the parsing modules, I was only
able to include so much in the last installment. Because of this, our
hero, the parser, when last seen, was a shadow of his former self,
consisting of only enough code to parse and process a factor consisting
of either a variable or a constant. The main effort of this current
installment will be to help flesh out the parser to its former glory.
In the process, I hope you'll bear with me if we sometimes cover ground
we've long since been over and dealt with.
First, let's take care of a problem that we've addressed before: Our
current version of procedure Factor, as we left it in Installment 15,
can't handle negative arguments. To fix that, we'll introduce the
procedure SignedFactor:
{--------------------------------------------------------------}
{ Parse and Translate a Factor with Optional Sign }
procedure SignedFactor;
var Sign: char;
begin
Sign := Look;
if IsAddop(Look) then
GetChar;
Factor;
if Sign = '-' then Negate;
end;
{--------------------------------------------------------------}
Note that this procedure calls a new code generation routine, Negate:
{--------------------------------------------------------------}
{ Negate Primary }
procedure Negate;
begin
EmitLn('NEG D0');
end;
{--------------------------------------------------------------}
(Here, and elsewhere in this series, I'm only going to show you the new
routines. I'm counting on you to put them into the proper unit, which
you should normally have no trouble identifying. Don't forget to add
the procedure's prototype to the interface section of the unit.)
In the main program, simply change the procedure called from Factor to
SignedFactor, and give the code a test. Isn't it neat how the Turbo
linker and make facility handle all the details?
Yes, I know, the code isn't very efficient. If we input a number, -3,
the generated code is:
MOVE #3,D0
NEG D0
which is really, really dumb. We can do better, of course, by simply
pre-appending a minus sign to the string passed to LoadConstant, but it
adds a few lines of code to SignedFactor, and I'm applying the KISS
philosophy very aggressively here. What's more, to tell the truth, I
think I'm subconsciously enjoying generating "really, really dumb" code,
so I can have the pleasure of watching it get dramatically better when
we get into optimization methods.
Most of you have never heard of John Spray, so allow me to introduce him
to you here. John's from New Zealand, and used to teach computer
science at one of its universities. John wrote a compiler for the
Motorola 6809, based on a delightful, Pascal-like language of his own
design called "Whimsical." He later ported the compiler to the 68000,
and for awhile it was the only compiler I had for my homebrewed 68000
system.
For the record, one of my standard tests for any new compiler is to see
how the compiler deals with a null program like:
program main;
begin
end.
My test is to measure the time required to compile and link, and the
size of the object file generated. The undisputed _LOSER_ in the test
is the DEC C compiler for the VAX, which took 60 seconds to compile, on
a VAX 11/780, and generated a 50k object file. John's compiler is the
undisputed, once, future, and forever king in the code size department.
Given the null program, Whimsical generates precisely two bytes of code,
implementing the one instruction,
RET
By setting a compiler option to generate an include file rather than a
standalone program, John can even cut this size, from two bytes to zero!
Sort of hard to beat a null object file, wouldn't you say?
Needless to say, I consider John to be something of an expert on code
optimization, and I like what he has to say: "The best way to optimize
is not to have to optimize at all, but to produce good code in the first
place." Words to live by. When we get started on optimization, we'll
follow John's advice, and our first step will not be to add a peephole
optimizer or other after-the-fact device, but to improve the quality of
the code emitted before optimization. So make a note of SignedFactor as
a good first candidate for attention, and for now we'll leave it be.
TERMS AND EXPRESSIONS
I'm sure you know what's coming next: We must, yet again, create the
rest of the procedures that implement the recursive-descent parsing of
an expression. We all know that the hierarchy of procedures for
arithmetic expressions is:
expression
term
factor
However, for now let's continue to do things one step at a time,
and consider only expressions with additive terms in them. The
code to implement expressions, including a possibly signed first
term, is shown next:
{--------------------------------------------------------------}
{ Parse and Translate an Expression }
procedure Expression;
begin
SignedFactor;
while IsAddop(Look) do
case Look of
'+': Add;
'-': Subtract;
end;
end;
{--------------------------------------------------------------}
This procedure calls two other procedures to process the
operations:
{--------------------------------------------------------------}
{ Parse and Translate an Addition Operation }
procedure Add;
begin
Match('+');
Push;
Factor;
PopAdd;
end;
{--------------------------------------------------------------}
{ Parse and Translate a Subtraction Operation }
procedure Subtract;
begin
Match('-');
Push;
Factor;
PopSub;
end;
{--------------------------------------------------------------}
The three procedures Push, PopAdd, and PopSub are new code generation
routines. As the name implies, procedure Push generates code to push
the primary register (D0, in our 68000 implementation) to the stack.
PopAdd and PopSub pop the top of the stack again, and add it to, or
subtract it from, the primary register. The code is shown next:
{--------------------------------------------------------------}
{ Push Primary to Stack }
procedure Push;
begin
EmitLn('MOVE D0,-(SP)');
end;
{--------------------------------------------------------------}
{ Add TOS to Primary }
procedure PopAdd;
begin
EmitLn('ADD (SP)+,D0');
end;
{--------------------------------------------------------------}
{ Subtract TOS from Primary }
procedure PopSub;
begin
EmitLn('SUB (SP)+,D0');
Negate;
end;
{--------------------------------------------------------------}
Add these routines to Parser and CodeGen, and change the main program to
call Expression. Voila!
The next step, of course, is to add the capability for dealing with
multiplicative terms. To that end, we'll add a procedure Term, and code
generation procedures PopMul and PopDiv. These code generation
procedures are shown next:
{--------------------------------------------------------------}
{ Multiply TOS by Primary }
procedure PopMul;
begin
EmitLn('MULS (SP)+,D0');
end;
{--------------------------------------------------------------}
{ Divide Primary by TOS }
procedure PopDiv;
begin
EmitLn('MOVE (SP)+,D7');
EmitLn('EXT.L D7');
EmitLn('DIVS D0,D7');
EmitLn('MOVE D7,D0');
end;
{--------------------------------------------------------------}
I admit, the division routine is a little busy, but there's no help for
it. Unfortunately, while the 68000 CPU allows a division using the top
of stack (TOS), it wants the arguments in the wrong order, just as it
does for subtraction. So our only recourse is to pop the stack to a
scratch register (D7), perform the division there, and then move the
result back to our primary register, D0. Note the use of signed multiply
and divide operations. This follows an implied, but unstated,
assumption, that all our variables will be signed 16-bit integers. This
decision will come back to haunt us later, when we start looking at
multiple data types, type conversions, etc.
Our procedure Term is virtually a clone of Expression, and looks like
this:
{--------------------------------------------------------------}
{ Parse and Translate a Term }
procedure Term;
begin
Factor;
while IsMulop(Look) do
case Look of
'*': Multiply;
'/': Divide;
end;
end;
{--------------------------------------------------------------}
Our next step is to change some names. SignedFactor now becomes
SignedTerm, and the calls to Factor in Expression, Add, Subtract and
SignedTerm get changed to call Term:
{--------------------------------------------------------------}
{ Parse and Translate a Term with Optional Leading Sign }
procedure SignedTerm;
var Sign: char;
begin
Sign := Look;
if IsAddop(Look) then
GetChar;
Term;
if Sign = '-' then Negate;
end;
{--------------------------------------------------------------}
...
{--------------------------------------------------------------}
{ Parse and Translate an Expression }
procedure Expression;
begin
SignedTerm;
while IsAddop(Look) do
case Look of
'+': Add;
'-': Subtract;
end;
end;
{--------------------------------------------------------------}
If memory serves me correctly, we once had BOTH a procedure SignedFactor
and a procedure SignedTerm. I had reasons for doing that at the time ...
they had to do with the handling of Boolean algebra and, in particular,
the Boolean "not" function. But certainly, for arithmetic operations,
that duplication isn't necessary. In an expression like:
-x*y
it's very apparent that the sign goes with the whole TERM, x*y, and not
just the factor x, and that's the way Expression is coded.
Test this new code by executing Main. It still calls Expression, so you
should now be able to deal with expressions containing any of the four
arithmetic operators.
Our last bit of business, as far as expressions goes, is to modify
procedure Factor to allow for parenthetical expressions. By using a
recursive call to Expression, we can reduce the needed code to virtually
nothing. Five lines added to Factor do the job:
{--------------------------------------------------------------}
{ Parse and Translate a Factor }
procedure Factor;
begin
if Look ='(' then begin
Match('(');
Expression;
Match(')');
end
else if IsDigit(Look) then
LoadConstant(GetNumber)
else if IsAlpha(Look)then
LoadVariable(GetName)
else
Error('Unrecognized character ' + Look);
end;
{--------------------------------------------------------------}
At this point, your "compiler" should be able to handle any legal
expression you can throw at it. Better yet, it should reject all
illegal ones!
ASSIGNMENTS
As long as we're this close, we might as well create the code to deal
with an assignment statement. This code needs only to remember the name
of the target variable where we are to store the result of an
expression, call Expression, then store the number. The procedure is
shown next:
{--------------------------------------------------------------}
{ Parse and Translate an Assignment Statement }
procedure Assignment;
var Name: string;
begin
Name := GetName;
Match('=');
Expression;
StoreVariable(Name);
end;
{--------------------------------------------------------------}
The assignment calls for yet another code generation routine:
{--------------------------------------------------------------}
{ Store the Primary Register to a Variable }
procedure StoreVariable(Name: string);
begin
EmitLn('LEA ' + Name + '(PC),A0');
EmitLn('MOVE D0,(A0)');
end;
{--------------------------------------------------------------}
Now, change the call in Main to call Assignment, and you should see a
full assignment statement being processed correctly. Pretty neat, eh?
And painless, too.
In the past, we've always tried to show BNF relations to define the
syntax we're developing. I haven't done that here, and it's high time I
did. Here's the BNF:
<factor> ::= <variable> | <constant> | '(' <expression> ')'
<signed_term> ::= [<addop>] <term>
<term> ::= <factor> (<mulop> <factor>)*
<expression> ::= <signed_term> (<addop> <term>)*
<assignment> ::= <variable> '=' <expression>
BOOLEANS
The next step, as we've learned several times before, is to add Boolean
algebra. In the past, this step has at least doubled the amount of code
we've had to write. As I've gone over this step in my mind, I've found
myself diverging more and more from what we did in previous
installments. To refresh your memory, I noted that Pascal treats the
Boolean operators pretty much identically to the way it treats
arithmetic ones. A Boolean "and" has the same precedence level as
multiplication, and the "or" as addition. C, on the other hand, sets
them at different precedence levels, and all told has a whopping 17
levels. In our earlier work, I chose something in between, with seven
levels. As a result, we ended up with things called Boolean
expressions, paralleling in most details the arithmetic expressions, but
at a different precedence level. All of this, as it turned out, came
about because I didn't like having to put parentheses around the Boolean
expressions in statements like:
IF (c >= 'A') and (c <= 'Z') then ...
In retrospect, that seems a pretty petty reason to add many layers of
complexity to the parser. Perhaps more to the point, I'm not sure I was
even able to avoid the parens.
For kicks, let's start anew, taking a more Pascal-ish approach, and just
treat the Boolean operators at the same precedence level as the
arithmetic ones. We'll see where it leads us. If it seems to be down
the garden path, we can always backtrack to the earlier approach.
For starters, we'll add the "addition-level" operators to Expression.
That's easily done; first, modify the function IsAddop in unit Scanner
to include two extra operators: '|' for "or," and '~' for "exclusive
or":
{--------------------------------------------------------------}
function IsAddop(c: char): boolean;
begin
IsAddop := c in ['+','-', '|', '~'];
end;
{--------------------------------------------------------------}
Next, we must include the parsing of the operators in procedure
Expression:
{--------------------------------------------------------------}
procedure Expression;
begin
SignedTerm;
while IsAddop(Look) do
case Look of
'+': Add;
'-': Subtract;
'|': _Or;
'~': _Xor;
end;
{--------------------------------------------------------------}
end;
(The underscores are needed, of course, because "or" and "xor" are
reserved words in Turbo Pascal.)
Next, the procedures _Or and _Xor:
{--------------------------------------------------------------}
{ Parse and Translate a Subtraction Operation }
procedure _Or;
begin
Match('|');
Push;
Term;
PopOr;
end;
{--------------------------------------------------------------}
{ Parse and Translate a Subtraction Operation }
procedure _Xor;
begin
Match('~');
Push;
Term;
PopXor;
end;
{--------------------------------------------------------------}
And, finally, the new code generator procedures:
{--------------------------------------------------------------}
{ Or TOS with Primary }
procedure PopOr;
begin
EmitLn('OR (SP)+,D0');
end;
{--------------------------------------------------------------}
{ Exclusive-Or TOS with Primary }
procedure PopXor;
begin
EmitLn('EOR (SP)+,D0');
end;
{--------------------------------------------------------------}
Now, let's test the translator (you might want to change the call
in Main back to a call to Expression, just to avoid having to type
"x=" for an assignment every time).
So far, so good. The parser nicely handles expressions of the
form:
x|y~z
Unfortunately, it also does nothing to protect us from mixing
Boolean and arithmetic algebra. It will merrily generate code
for:
(a+b)*(c~d)
We've talked about this a bit, in the past. In general the rules
for what operations are legal or not cannot be enforced by the
parser itself, because they are not part of the syntax of the
language, but rather its semantics. A compiler that doesn't allow
mixed-mode expressions of this sort must recognize that c and d
are Boolean variables, rather than numeric ones, and balk at
multiplying them in the next step. But this "policing" can't be
done by the parser; it must be handled somewhere between the
parser and the code generator. We aren't in a position to enforce
such rules yet, because we haven't got either a way of declaring
types, or a symbol table to store the types in. So, for what
we've got to work with at the moment, the parser is doing
precisely what it's supposed to do.
Anyway, are we sure that we DON'T want to allow mixed-type
operations? We made the decision some time ago (or, at least, I
did) to adopt the value 0000 as a Boolean "false," and -1, or
FFFFh, as a Boolean "true." The nice part about this choice is
that bitwise operations work exactly the same way as logical ones.
In other words, when we do an operation on one bit of a logical
variable, we do it on all of them. This means that we don't need
to distinguish between logical and bitwise operations, as is done
in C with the operators & and &&, and | and ||. Reducing the
number of operators by half certainly doesn't seem all bad.
From the point of view of the data in storage, of course, the
computer and compiler couldn't care less whether the number FFFFh
represents the logical TRUE, or the numeric -1. Should we? I
sort of think not. I can think of many examples (though they
might be frowned upon as "tricky" code) where the ability to mix
the types might come in handy. Example, the Dirac delta function,
which could be coded in one simple line:
-(x=0)
or the absolute value function (DEFINITELY tricky code!):
x*(1+2*(x<0))
Please note, I'm not advocating coding like this as a way of life.
I'd almost certainly write these functions in more readable form,
using IFs, just to keep from confusing later maintainers. Still,
a moral question arises: Do we have the right to ENFORCE our
ideas of good coding practice on the programmer, but writing the
language so he can't do anything else? That's what Nicklaus Wirth
did, in many places in Pascal, and Pascal has been criticized for
it -- for not being as "forgiving" as C.
An interesting parallel presents itself in the example of the
Motorola 68000 design. Though Motorola brags loudly about the
orthogonality of their instruction set, the fact is that it's far
from orthogonal. For example, you can read a variable from its
address:
MOVE X,D0 (where X is the name of a variable)
but you can't write in the same way. To write, you must load an
address register with the address of X. The same is true for PC-
relative addressing:
MOVE X(PC),DO (legal)
MOVE D0,X(PC) (illegal)
When you begin asking how such non-orthogonal behavior came about,
you find that someone in Motorola had some theories about how
software should be written. Specifically, in this case, they
decided that self-modifying code, which you can implement using
PC-relative writes, is a Bad Thing. Therefore, they designed the
processor to prohibit it. Unfortunately, in the process they also
prohibited _ALL_ writes of the forms shown above, however benign.
Note that this was not something done by default. Extra design
work had to be done, and extra gates added, to destroy the natural
orthogonality of the instruction set.
One of the lessons I've learned from life: If you have two
choices, and can't decide which one to take, sometimes the best
thing to do is nothing. Why add extra gates to a processor to
enforce some stranger's idea of good programming practice? Leave
the instructions in, and let the programmers debate what good
programming practice is. Similarly, why should we add extra code
to our parser, to test for and prevent conditions that the user
might prefer to do, anyway? I'd rather leave the compiler simple,
and let the software experts debate whether the practices should
be used or not.
All of which serves as rationalization for my decision as to how
to prevent mixed-type arithmetic: I won't. For a language
intended for systems programming, the fewer rules, the better. If
you don't agree, and want to test for such conditions, we can do
it once we have a symbol table.
BOOLEAN "AND"
With that bit of philosophy out of the way, we can press on to the
"and" operator, which goes into procedure Term. By now, you can
probably do this without me, but here's the code, anyway:
In Scanner,
{--------------------------------------------------------------}
function IsMulop(c: char): boolean;
begin
IsMulop := c in ['*','/', '&'];
end;
{--------------------------------------------------------------}
In Parser,
{--------------------------------------------------------------}
procedure Term;
begin
Factor;
while IsMulop(Look) do
case Look of
'*': Multiply;
'/': Divide;
'&': _And;
end;
end;
{--------------------------------------------------------------}
{ Parse and Translate a Boolean And Operation }
procedure _And;
begin
Match('&');
Push;
Factor;
PopAnd;
end;
{--------------------------------------------------------------}
and in CodeGen,
{--------------------------------------------------------------}
{ And Primary with TOS }
procedure PopAnd;
begin
EmitLn('AND (SP)+,D0');
end;
{--------------------------------------------------------------}
Your parser should now be able to process almost any sort of logical
expression, and (should you be so inclined), mixed-mode expressions as
well.
Why not "all sorts of logical expressions"? Because, so far, we haven't
dealt with the logical "not" operator, and this is where it gets tricky.
The logical "not" operator seems, at first glance, to be identical in
its behavior to the unary minus, so my first thought was to let the
exclusive or operator, '~', double as the unary "not." That didn't
work. In my first attempt, procedure SignedTerm simply ate my '~',
because the character passed the test for an addop, but SignedTerm
ignores all addops except '-'. It would have been easy enough to add
another line to SignedTerm, but that would still not solve the problem,
because note that Expression only accepts a signed term for the _FIRST_
argument.
Mathematically, an expression like:
-a * -b
makes little or no sense, and the parser should flag it as an error.
But the same expression, using a logical "not," makes perfect sense:
not a and not b
In the case of these unary operators, choosing to make them act the same
way seems an artificial force fit, sacrificing reasonable behavior on
the altar of implementational ease. While I'm all for keeping the
implementation as simple as possible, I don't think we should do so at
the expense of reasonableness. Patching like this would be missing the
main point, which is that the logical "not" is simply NOT the same kind
of animal as the unary minus. Consider the exclusive or, which is most
naturally written as:
a~b ::= (a and not b) or (not a and b)
If we allow the "not" to modify the whole term, the last term in
parentheses would be interpreted as:
not(a and b)
which is not the same thing at all. So it's clear that the logical
"not" must be thought of as connected to the FACTOR, not the term.
The idea of overloading the '~' operator also makes no sense from a
mathematical point of view. The implication of the unary minus is that
it's equivalent to a subtraction from zero:
-x <=> 0-x
In fact, in one of my more simple-minded versions of Expression, I
reacted to a leading addop by simply preloading a zero, then processing
the operator as though it were a binary operator. But a "not" is not
equivalent to an exclusive or with zero ... that would just give back
the original number. Instead, it's an exclusive or with FFFFh, or -1.
In short, the seeming parallel between the unary "not" and the unary
minus falls apart under closer scrutiny. "not" modifies the factor, not
the term, and it is not related to either the unary minus nor the
exclusive or. Therefore, it deserves a symbol to call its own. What
better symbol than the obvious one, also used by C, the '!' character?
Using the rules about the way we think the "not" should behave, we
should be able to code the exclusive or (assuming we'd ever need to), in
the very natural form:
a & !b | !a & b
Note that no parentheses are required -- the precedence levels we've
chosen automatically take care of things.
If you're keeping score on the precedence levels, this definition puts
the '!' at the top of the heap. The levels become:
1. !
2. - (unary)
3. *, /, &
4. +, -, |, ~
Looking at this list, it's certainly not hard to see why we had trouble
using '~' as the "not" symbol!
So how do we mechanize the rules? In the same way as we did with
SignedTerm, but at the factor level. We'll define a procedure
NotFactor:
{--------------------------------------------------------------}
{ Parse and Translate a Factor with Optional "Not" }
procedure NotFactor;
begin
if Look ='!' then begin
Match('!');
Factor;
Notit;
end
else
Factor;
end;
{--------------------------------------------------------------}
and call it from all the places where we formerly called Factor, i.e.,
from Term, Multiply, Divide, and _And. Note the new code generation
procedure:
{--------------------------------------------------------------}
{ Bitwise Not Primary }
procedure NotIt;
begin
EmitLn('EOR #-1,D0');
end;
{--------------------------------------------------------------}
Try this now, with a few simple cases. In fact, try that exclusive or
example,
a&!b|!a&b
You should get the code (without the comments, of course):
MOVE A(PC),DO ; load a
MOVE D0,-(SP) ; push it
MOVE B(PC),DO ; load b
EOR #-1,D0 ; not it
AND (SP)+,D0 ; and with a
MOVE D0,-(SP) ; push result
MOVE A(PC),DO ; load a
EOR #-1,D0 ; not it
MOVE D0,-(SP) ; push it
MOVE B(PC),DO ; load b
AND (SP)+,D0 ; and with !a
OR (SP)+,D0 ; or with first term
That's precisely what we'd like to get. So, at least for both
arithmetic and logical operators, our new precedence and new, slimmer
syntax hang together. Even the peculiar, but legal, expression with
leading addop:
~x
makes sense. SignedTerm ignores the leading '~', as it should, since
the expression is equivalent to:
0~x,
which is equal to x.
When we look at the BNF we've created, we find that our boolean algebra
now adds only one extra line:
<not_factor> ::= [!] <factor>
<factor> ::= <variable> | <constant> | '(' <expression> ')'
<signed_term> ::= [<addop>] <term>
<term> ::= <not_factor> (<mulop> <not_factor>)*
<expression> ::= <signed_term> (<addop> <term>)*
<assignment> ::= <variable> '=' <expression>
That's a big improvement over earlier efforts. Will our luck continue
to hold when we get to relational operators? We'll find out soon, but
it will have to wait for the next installment. We're at a good stopping
place, and I'm anxious to get this installment into your hands. It's
already been a year since the release of Installment 15. I blush to
admit that all of this current installment has been ready for almost as
long, with the exception of relational operators. But the information
does you no good at all, sitting on my hard disk, and by holding it back
until the relational operations were done, I've kept it out of your
hands for that long. It's time for me to let go of it and get it out
where you can get value from it. Besides, there are quite a number of
serious philosophical questions associated with the relational
operators, as well, and I'd rather save them for a separate installment
where I can do them justice.
Have fun with the new, leaner arithmetic and logical parsing, and I'll
see you soon with relationals.
*****************************************************************
* *
* COPYRIGHT NOTICE *
* *
* Copyright (C) 1995 Jack W. Crenshaw. All rights reserved. *
* *
* *
*****************************************************************
--
--/*\ Jon A. Lambert - TychoMUD Internet:jlsysinc#ix,netcom.com /*\--
--/*\ Mud Server Developer's Page <<A HREF="http://www.netcom.com/~jlsysinc">http://www.netcom.com/~jlsysinc</A>> /*\--
--/*\ "Everything that deceives may be said to enchant" - Plato /*\--
</PRE>
<!--X-Body-of-Message-End-->
<!--X-MsgBody-End-->
<!--X-Follow-Ups-->
<HR>
<!--X-Follow-Ups-End-->
<!--X-References-->
<!--X-References-End-->
<!--X-BotPNI-->
<UL>
<LI>Prev by Date:
<STRONG><A HREF="msg00684.html">Tutorial: Let's build a Compiler! - Part XIV: Types</A></STRONG>
</LI>
<LI>Next by Date:
<STRONG><A HREF="msg00686.html">Tutorial: Let's build a Compiler! - Part XV: Back to the Future</A></STRONG>
</LI>
<LI>Prev by thread:
<STRONG><A HREF="msg00686.html">Tutorial: Let's build a Compiler! - Part XV: Back to the Future</A></STRONG>
</LI>
<LI>Next by thread:
<STRONG><A HREF="msg00684.html">Tutorial: Let's build a Compiler! - Part XIV: Types</A></STRONG>
</LI>
<LI>Index(es):
<UL>
<LI><A HREF="index.html#00685"><STRONG>Date</STRONG></A></LI>
<LI><A HREF="thread.html#00685"><STRONG>Thread</STRONG></A></LI>
</UL>
</LI>
</UL>
<!--X-BotPNI-End-->
<!--X-User-Footer-->
<!--X-User-Footer-End-->
<ul><li>Thread context:
<BLOCKQUOTE><UL>
<LI><strong><A NAME="00689" HREF="msg00689.html">Random Generation Tools</A></strong>,
Ling <a href="mailto:K.L.Lo-94#student,lboro.ac.uk">K.L.Lo-94#student,lboro.ac.uk</a>, Sun 08 Mar 1998, 21:51 GMT
<LI><strong><A NAME="00688" HREF="msg00688.html">Re: MUD Ratings</A></strong>,
jlsysinc.ix.netcom.com <a href="mailto:jlsysinc.ix.netcom.com#ix,netcom.com">jlsysinc.ix.netcom.com#ix,netcom.com</a>, Fri 06 Mar 1998, 22:43 GMT
<LI><strong><A NAME="00687" HREF="msg00687.html">[MUD-Dev] For Ling's FAQ</A></strong>,
Koster, Raph <a href="mailto:rkoster#origin,ea.com">rkoster#origin,ea.com</a>, Fri 06 Mar 1998, 15:14 GMT
<LI><strong><A NAME="00686" HREF="msg00686.html">Tutorial: Let's build a Compiler! - Part XV: Back to the Future</A></strong>,
Jon A. Lambert <a href="mailto:jlsysinc#ix,netcom.com">jlsysinc#ix,netcom.com</a>, Thu 05 Mar 1998, 04:24 GMT
<LI><strong><A NAME="00685" HREF="msg00685.html">Tutorial: Let's build a Compiler! - Part XVI: Unit Construction</A></strong>,
Jon A. Lambert <a href="mailto:jlsysinc#ix,netcom.com">jlsysinc#ix,netcom.com</a>, Thu 05 Mar 1998, 04:23 GMT
<LI><strong><A NAME="00684" HREF="msg00684.html">Tutorial: Let's build a Compiler! - Part XIV: Types</A></strong>,
Jon A. Lambert <a href="mailto:jlsysinc#ix,netcom.com">jlsysinc#ix,netcom.com</a>, Thu 05 Mar 1998, 04:23 GMT
<LI><strong><A NAME="00683" HREF="msg00683.html">VEIL Network Protocol</A></strong>,
Brandon Gillespie <a href="mailto:brandon#roguetrader,com">brandon#roguetrader,com</a>, Wed 04 Mar 1998, 15:32 GMT
<LI><strong><A NAME="00682" HREF="msg00682.html">Monthly FAQ Posting</A></strong>,
Ling <a href="mailto:K.L.Lo-94#student,lboro.ac.uk">K.L.Lo-94#student,lboro.ac.uk</a>, Wed 04 Mar 1998, 13:28 GMT
<LI><strong><A NAME="00681" HREF="msg00681.html">Re: [MUD-Dev] Tutorial: Let's build a Compiler!</A></strong>,
Chris Gray <a href="mailto:cg#ami-cg,GraySage.Edmonton.AB.CA">cg#ami-cg,GraySage.Edmonton.AB.CA</a>, Wed 04 Mar 1998, 01:39 GMT
</UL></BLOCKQUOTE>
</ul>
<hr>
<center>
[ <a href="../">Other Periods</a>
| <a href="../../">Other mailing lists</a>
| <a href="/search.php3">Search</a>
]
</center>
<hr>
</body>
</html>
