@c Copyright (C) 1996-2000 Free Software Foundation, Inc.
@c For copying conditions, see the file gpc.texi.
@c This file is part of the GPC Manual.
@c
@c Author: Peter Gerwinski <peter@gerwinski.de>
@c
@c Last modification: 27 Aug 2000 (file not up to date)

@node Internals
@chapter The GPC Source Reference
@cindex GPC source, internals
@cindex GPC, internals

@quotation
@i{``The Source will be with you. Always.''}
@end quotation

This chapter describes internals of GPC. It is meant for GPC
developers and those who want to become devlopers, or just want to
know more about how the compiler works. It does not contain
information needed when using GPC to compile programs.

This chapter tells you how to look up additional information about
the GNU Pascal compiler from its source code.

@c It replaces chapters like ``syntax diagrams'' you probably know from
@c the documentation of other compilers.
@c @@@@ Not really. Syntax diagrams are directed towards users, and
@c      Pascal programmers can't be expected to make sense of bison
@c      source with C statements (it's hard enough for us
@c      sometimes... ;-). OK, so we just need a tool to draw syntax
@c      diagrams from a bison input. This might be feasible,
@c      actually. However, the bison grammar might not be the
@c      optimal description of the GPC syntax from a user's point of
@c      view... --Frank

@strong{Note:} If you intend to modify GPC's source, please check
the top of each file you're going to modify. A number of files are
generated automatically by various tools. The top of these files
will tell you by which tool and from what file they were generated.
Modifying a generated file is pointless, since it will be
overwritten the next time the tool is run. Instead, modify the
original source (which will often be easier, anyway, e.g. a bison
input file vs. the generated C code). This also holds for various
documentation files.

Proprietary compilers often come with a lot of technical information
about the internals of the compiler. This is necessary because their
vendors want to avoid to distribute the source of the compiler --
which is always the most definitive source of this technical
information.

With GNU compilers, on the other hand, you are free to get the
source code, look how your compiler works internally, customize it
for your own needs, and to re-distribute it in modified or
unmodified form. You may even take money for this redistribution.
(For details, see the GNU General Public License, @ref{Copying}.)

The following subsections are your guide to the GNU Pascal source
code. If you have further questions, be welcome to ask them at the
GNU Pascal mailing list (see @ref{Support}).

All file paths mentioned in this chapter are relative to the GNU Pascal
source directory, usually a subdirectory @file{p} of the GCC source.

@menu
* Lexical analyzer::     @file{gpc-lex.c} -- How GPC reads your source.
* Language definition::  @file{parse.y} -- ``Syntax diagrams'' as ``Bison'' source.
* Tree nodes::           @file{../tree.*} -- How GPC stores your program internally.
* Parameter passing::    @file{gpc-typeck.c} -- How GPC passes parameters.
* GPI files::            @file{module.c} -- How GPC's precompiled Module/Unit interfaces work.
* AutoMake::             @file{module.c} -- How GPC automatically ``makes'' a large project.
* File Layout::          Files that make up GPC; Integrating GNU Pascal in GCC
@end menu

(Under construction.)

For more information, see chapters ``Portability'' through ``Fragments'' in
@ref{Top,the GCC documentation,,gcc,``Using and Porting GNU CC''}.


@c ========================================================================


@node Lexical analyzer
@section GPC's Lexical Analyzer
@cindex lexical analyzer, internals

The source file @file{gpc-lex.c} contains the so-called
@emph{lexical analyzer} of the GNU Pascal compiler.  (For those of
you who know @file{flex}:  This file was @emph{not} created using
@file{flex} but is maintained manually.)  This very-first stage of
the compiler (after the preprocessor which is a separate executable)
is responsible for reading what you have written and dividing it
into @emph{tokens}, the ``atoms'' of each computer language.  The
source @file{gpc-lex.c} essentially contains one large function,
@samp{yylex()}.

Here is, for example, where the real number @samp{3.14} and the subrange
@samp{3..14} are distinguished, and where Borland-style character constants
like @samp{#13} and @samp{^M} are recognized.  This is not always a trivial
task, for example look at the following type declaration:

@example
type
  X = ^Y;
  Y = packed array [^A .. ^B] of Char;
  Z = ^A .. ^Z;
@end example

If you wish to know how GPC distinguishes the pointer forward declaration
@samp{^Y} and the subrange @samp{^A..^Z}, see @file{gpc-lex.c}, function
@samp{yylex()}, @samp{case '^':} in the big @samp{switch} statement.

There are several situation where GPC's lexical analzyer becomes
context-sensitive.  One is described above, another example is the token
@samp{protected}, a reserved word in ISO-10206 Extended Pascal, but an
ordinary identifier in ISO-7185 Standard Pascal.  It appears in parameter
lists

@example
procedure foo (protected bar: Integer);
@end example

and says that the parameter @samp{bar} must not be changed inside the body of
the procedure.

OTOH, if you write a valid ISO-7185 Standard Pascal program, you can declare a
parameter @samp{protected}:

@example
procedure foo (protected, bar: Integer);
@end example

Here both standards contradict each other.  GPC solves this problem by
checking explicitly for ``protected'' in the lexical analyzer:  If a comma or
a colon follows, this is an ordinary identifier, otherwise it's a reserved
word.  Having this, GPC even understands

@example
procedure foo (protected protected: Integer);
@end example

without losing the special meaning of @samp{protected} as a reserved word.

The responsible code is in @file{gpc-lex.c} -- look out for @samp{PROTECTED}.

If you ever encouter a bug with the lexical analyzer -- now you know where to
hunt for it.


@c ========================================================================


@node Language definition
@section Language Definition: GPC's Parser
@cindex language definition, internals
@cindex parser, internals
@cindex grammar, internals
@cindex front-end, internals

The file @file{parse.y} contains the ``bison'' source code of GNU Pascal's
parser.  This stage of the compilation analyzes and checks the syntax of
your Pascal program, and it generates an intermediate, language-independent
code which is then passed to the GNU back-end.

The @emph{bison} language essentially is a machine-readable form of the
Backus-Naur Form, the symbolic notation of grammars of computer languages.
``Syntax diagrams'' are a graphical variant of the Backus-Naur Form.

For details about the ``bison'' language, see
@ref{,the Bison documentation,,bison,the Bison manual}.  A short overview
how to pick up some information you might need for programming follows.

Suppose you have forgotten how a variable is declared in Pascal.  After some
searching in @file{parse.y} you have found the following:

@example
/* variable declaration part */

variable_declaration_part:
    LEX_VAR variable_declaration_list semi
  | LEX_VAR semi
      @{ error ("missing variable declaration"); @}
  ;

variable_declaration_list:
    variable_declaration
  | variable_declaration_list semi variable_declaration
      @{ yyerrok; @}
  | error
  | variable_declaration_list error variable_declaration
      @{
        error ("missing semicolon");
        yyerrok;
      @}
  | variable_declaration_list semi error
  ;
@end example

Translated into English, this means: ``The variable declaration part
consists of the reserved word (lexical token) @samp{var} followed by
a `variable declaration list' and a semicolon.  A semicolon
immediately following @samp{var} is an error.  A `variable
declaration list' in turn consists of one or more `variable
declarations', separated by semicolons.'' (The latter explanation
requires that you understand the recursive nature of the definition
of @samp{variable_declaration_list}.)

Now we can go on and search for @samp{variable_declaration}.

@example
variable_declaration:
    id_list
      @{
        [...]
      @}
    enable_caret ':' optional_qualifier_list type_denoter
      @{
        [...]
      @}
    absolute_or_value_specification
      @{
        [...]
      @}
  ;
@end example

(The @samp{[...]} are placeholders for some C statements which aren't
important for understanding GPC's grammar.)

From this you can look up that a variable declaration in GNU Pascal consists
of an ``id list'', followed by ``enable_caret'' (whatever that means), a colon,
an ``optional qualifier list'', a ``type denoter'', and an ``absolute or value
specification''.  Some of these parts are easy to understand, the others you
can look up from @file{parse.y}.  Remember that the reserved word @samp{var}
precedes all this, and a semicolon follows all this.

Now you know the procedure how to get the exact grammar of the GNU Pascal
language from the source.

The C statements, not shown above, are in some sense the most important part
of the bison source, because they are responsible for the generation of
the intermediate code of the GNU Pascal front-end, the so-called
@emph{tree nodes}.  For instance, the C code in ``type denoter'' juggles
a while with variables of the type @samp{tree}, and finally returns (assigns
to @samp{$$}) a so-called @emph{tree list} which contains the information
about the type.  The ``variable declaration'' gets this tree list (as the
argument @samp{$6}) and passes the type information to the C function
@samp{declare_vars()} (declared in @samp{util.c}).  This function
@samp{declare_vars()} does the real work of compiling a variable
declaration.

This, the parser, is the place where it becomes Pascal.


@c ========================================================================


@node Tree nodes
@section Tree Nodes
@cindex tree nodes, internals
@cindex intermediate code, internals
@cindex front-end, internals

If you want really to understand how the GNU Pascal language front-end works
internally and perhaps want to improve the compiler, it is important that you
understand GPC's internal data structures.

The data structure used by the language front-end to hold all information
about your Pascal program are the so-called ``tree nodes''.  (Well, it needn't
be Pascal source -- tree nodes are language independent.)  The tree nodes are
kind of objects, connected to each other via pointers.  Since the GNU compiler
is written in C and was created at a time where nobody really thought about
object-orientated programming languages yet, a lot of effort has been taken to
create these ``objects'' in C.

Here is an extract from the ``object hierarchy''.  Omissions are marked
with ``@dots{}''; nodes in parentheses are ``abstract'':  They are never
instantiated and aren't really defined.  They only appear here to clarify
the structure of the tree node hierarchy.  The complete list is in
@file{../tree.def}; additional information can be found in @file{../tree.h}.

@example
(tree_node)
|
|--- error_mark  (* enables GPC to continue after first error *)
|
|--- (identifier)
|    |
|    |--- identifier_node
|    |
|    \--- op_identifier
|
|--- tree_list  (* general-purpose "container object" *)
|
|--- tree_vec
|
|--- block
|
|--- (type)  (* information about types *)
|    |
|    |--- void_type
|    |
|    |--- integer_type
|   ...
|    |
|    |--- record_type
|    |
|    |--- function_type
|    |
|    \--- lang_type  (* for language-specific extensions *)
|
|--- integer_cst  (* an integer constant *)
|
|--- real_cst
|
|--- string_cst
|
|--- complex_cst
|
|--- (declaration)
|    |
|    |--- function_decl
|   ...
|    |
|    |--- type_decl
|    |
|    \--- var_decl
|
|--- (reference)
|    |
|    |--- component_ref
|   ...
|    |
|    \--- array_ref
|
|--- constructor
|
\--- (expression)
     |
     |--- modify_expr  (* assignment *)
     |
     |--- plus_expr  (* addition *)
    ...
     |
     |--- call_expr  (* procedure/function call *)
     |
     |--- goto_expr
     |
     \--- loop_expr  (* for all loops *)
@end example

Most of these tree nodes -- struct variables in fact -- contain pointers to
other tree nodes.  A @samp{tree_list} for instance has a @samp{tree_value}
and a @samp{tree_purpose} slot which can contain arbitrary data; a third
pointer @samp{tree_chain} points to the next @samp{tree_list} node and thus
allows us to create linked lists of tree nodes.

One example:  When GPC reads the list of identifiers in a variable
declaration

@example
var
  foo, bar, baz : Integer;
@end example

@cindex magic, internals
the parser creates a chain of @samp{tree_list}s whose @samp{tree_value}s hold
@samp{identifier_node}s for the identifiers @samp{foo}, @samp{bar}, and
@samp{baz}.  The function @samp{declare_vars()} (declared in @file{util.c})
gets this tree list as a parameter, does some magic, and finally passes a
chain of @samp{var_decl} nodes to the back-end.

The @samp{var_decl} nodes in turn have a pointer @samp{tree_type} which holds
a @samp{_type} node -- an @samp{integer_type} node in the example above.
Having this, GPC can do type-checking when a variable is referenced.

For another example, let's look at the following statement:

@example
baz := foo + bar;
@end example

Here the parser creates a @samp{modify_expr} tree node.  This node has two
pointers, @samp{tree_operand[0]} which holds a representation of @samp{baz}, a
@samp{var_decl} node, and @samp{tree_operand[1]} which holds a representation
of the sum @samp{foo + bar}.  The sum in turn is represented as a
@samp{plus_expr} tree node whose @samp{tree_operand[0]} is the @samp{var_decl}
node @samp{foo}, and whose @samp{tree_operand[1]} is the @samp{var_decl} node
@samp{bar}.  Passing this (the @samp{modify_expr} node) to the back-end
results in assembler code for the assignment.

If you want to have a closer look at these tree nodes, write a line
@samp{(*$debug-tree="Foobar"*)} into your program with @samp{FooBar} being
some identifier in your program.  (Note the capitalization of the first
character in the internal representation.)  This tells GPC to output the
@samp{identifier_local_value} tree node -- the meaning of this
identifier -- to the standard error device in human-readable form.

While hacking and debugging GPC, you will also wish to have a look at these
tree nodes in other cases.  Use the @samp{debug_tree()} function to do so.
(In fact @samp{(*$debug-tree="Foobar"*)} does nothing else than to
@samp{debug_tree()} the @samp{identifier_local_value} of the @samp{Foobar}
identifier node.)


@c ========================================================================


@node Parameter passing
@section Parameter Passing
@cindex parameter passing, internals

GPC supports a lot of funny things in parameter lists:  @samp{protected} and
@samp{const} parameters, strings with specified or unspecified length,
conformant arrays, objects as @samp{var} parameters, etc.  All this requires
sophisticated type-checking; the responsible function is
@samp{convert_arguments()} in the source file @file{gpc-typeck.c}.  Every
detail can be looked up from there.

Some short notes about the most interesting cases follow.

@table @strong

@cindex conformant arrays, internals
@item Conformant arrays:
First, the array bounds are passed (an even number of parameters of an ordinal
type), then the address of the array itself.

@cindex procedural parameters, internals
@cindex functions as parameters, internals
@item Procedural parameters:
These need special care because a function passed as a parameter can be
confused with a call to the function whose result is then passed as a
parameter.  See also the functions @samp{maybe_call_function()} and
@samp{probably_call_function()} in @file{util.c}.

@item Chars:
According to ISO-10206 Extended Pascal, formal char parameters accept string
values.  GPC does the necessary conversion implicitly.  The empty string
produces a space.

@item Strings and schemata:
Value parameter strings and schemata of known size are really passed by value.
Value parameter strings and schemata of unknown size are passed by reference,
and GPC creates temporary variable to hold a copy of the string.

@item @samp{const} parameters:
If a constant value is passed to a @samp{const} parameter, GPC assigns the
value to a temporary variable whose address is passed.

Even if the parameter is passed by value??? -- Frank

@item Typeless parameters:
These are denoted by @samp{var foo} or @samp{var foo: Void} and are
compatible to C's @samp{void *} parameters; the size of such
entities is @emph{not} passed. Maybe we will change this in the
future and pass the size for @samp{var foo} parameters whereas
@samp{var foo: Void} will remain compatible to C.
(Same with @samp{const} instead of @samp{var}.)

@item @samp{CString} parameters:
GPC implicitly converts any string value such that the address of the actual
string data is passed.  However, GPC did not implicitly append a
@samp{Chr (0)} terminator, except for string constants.
It does so now. @c Frank

@end table


@c ========================================================================


@c GPI.DOC, version 1.0 for GNU Pascal 2.0 (corresponding to GCC 2.8.1)
@c 26 Jul 1996, Peter Gerwinski <peter@gerwinski.de>
@c integrated into Texinfo documentation 1 Aug 1997 PG

@node GPI files
@section GPI files -- GNU Pascal Interfaces
@cindex GPI files, internals

This section documents the mechanism how GPC transfers information
from the exporting modules and units to the program, module or
unit which imports (uses) the information.

A GPI file contains a precompiled GNU Pascal interface.
``Precompiled'' means in this context that the interface already has
been parsed (i.e.@: the front-end has done its work), but that no
assembler output has been produced yet.

The GPI file format is an implementation-dependent (but not @emph{too}
implementation-dependent ;@minus{}) file format for storing GNU Pascal
interfaces to be exported -- Extended Pascal and PXSC module
interfaces as well as interface parts of Borland Pascal Units 
compiled with GNU Pascal.

To see what information is stored in or loaded from a GPI file, run 
GPC with an additional command-line option @samp{--debug-gpi}.  Then, GPC 
will write a human-readable version of what is being stored/loaded 
to the standard error device.  (See also: @ref{Tree nodes}.)

While parsing an interface, GPC stores the names of exported objects
in tree lists -- in @file{gpc-parse.y}, the bison (yacc) source of GPC's
parser, search for @samp{handle_autoexport}.  At the end of the interface,
everything is stored in one or more GPI files.  This is called in
@file{gpc-parse.y} -- search for @samp{create_gpi_files}.  (See also:
@ref{Language definition}, for an introduction to @file{gpc-parse.y})

Everything else is done in gpc-module.c.  Here you can find the source
of @samp{create_gpi_files()} which documents the file format:  First, a header 
of 33 bytes containing the string @samp{GNU Pascal Module/Unit Interface\n}
is stored, then the name of the primary source file of the module as a
string, then the name of the exported interface as a tree node (see
below), after that all exported names in the order as they were stored
while parsing.

The names and the objects (i.e.@: constants, data types, variables and
functions) they refer to are internally represented as so-called
@emph{tree nodes} as defined in the files @file{../tree.h} and
@file{../tree.def} from the GNU compiler back-end.  (See also: @ref{Tree
nodes}.)  The names are stored as @samp{identifier_node}s, their
meanings as @samp{identifier_global_value}s of these nodes.
The main problem when storing tree nodes is that they 
form a complicated tree in memory with a lot of circular references
making it hard to decide which information must be stored and which 
mustn't.

The functions @samp{load_tree()} and @samp{store_tree} load/store
a tree node with the contents of all its contained pointers in a GPI file.

Each tree node has a @samp{tree_code} indicating what kind of information
it contains.  Each different tree node must be stored in a different
way.  See the source of @samp{load_tree()} and @samp{store_tree()} for
details.

Most tree nodes contain pointers to other tree nodes; therefore
@samp{load_tree()} and @samp{store_tree()} are recursive functions.  
The @samp{--debug-gpi} debugging informations contains the recursion 
level in parantheses, e.g.@: @samp{loaded (2):} means that the loaded 
information was requested by a pointer contained in a tree node 
requested by a pointer contained in a tree node representing an 
exported symbol.

Since this recursion can be circular (think of a record containing
a pointer to a record of the same type), we must resolve references
to tree nodes which already have been loaded.  For this reason, all
nodes are recorded in a hash buffer -- see @file{gpi-hash.c}.

There are some special tree_nodes (e.g.@: @samp{integer_type_node} or
@samp{NULL_TREE}) which are used very often.  They have been assigned
(normally invalid) unique @samp{tree_code}s, so they can be stored in
a single byte.

That's it.  Now you should be able to ``read'' GPI files using GPC's
@samp{--debug-gpi} option.  If you encounter a case where the loaded 
information differs too much from the stored information, you 
have found a bug -- congratulations!  What ``too much'' means, depends
on the object being stored in or loaded from the GPI file.  Remind
that the order things are loaded from a GPI file is the @emph{reversed}
order things are stored when considering @emph{different} recursion levels,
but the @emph{same} order when considering ths @emph{same} recursion level.


@c ========================================================================


@c AUTOMAKE.DOC, version 1.0 for GNU Pascal 2.0 (corresponding to GCC 2.7.2.1)
@c 28 Jul 1996, Peter Gerwinski <peter.gerwinski@uni-essen.de>
@c integrated into Texinfo documentation 1 Aug 1997 PG

@node AutoMake
@section GPC's AutoMake Mechanism -- How it Works
@cindex AutoMake, internals

When a program/module/unit imports (uses) an interface, GPC searches
for the GPI file (see @ref{GPI files}) derived from the name of the
interface.

Case 1:  A GPI file was found.

Each GPI file contains the name of the primary source file (normally
a @file{.pas} or @file{.p} file) of the module/unit, and the names
of all interfaces imported. GPC reads this information and invokes
itself with a command like

@example
gpc foo.pas -M -o foo.d
@end example

This means: preprocess the file, and write down the name of
the object file and those of all its source files in @file{foo.d}.
GPC reads @file{foo.d} and looks if the object file exists and if
the source was modified since the creation of the object file
and the gpi file.  If so, GPC calls itself again to compile
the primary source file.  When everything is done, the @file{.d}
file is removed.  If there was no need to recompile, all
interfaces imported by the module/unit are processed in the
same way as this one.

Case 2:  No GPI file was found.

In this case, GPC derives the name of the source file from that of
the interface by trying first @file{interface.p}, then
@file{interface.pas}. This will almost always work with Borland
Pascal units, almost never with Extended Pascal modules. With
Extended Pascal, compile the module once manually in order to
produce a GPI file.

All this is done by the function @samp{gpi_open()} which uses some
auxiliary functions such as @samp{module_must_be_recompiled()} and
@samp{compile_module()}.

Each time an object file is compiled or recognized as being
up-to-date, its name is stored in a temporary file with the same base
name as all the other temporary files used by GPC but the extension
@file{.gpc}.  When the top-level @file{gpc} is invoked 
(which calls @file{gpc1} later on), it passes the name of this 
temporary file as an additional command line parameter to
@file{gpc1}.  After compilation has been completed, the top-level 
@file{gpc} reads the temporary file and adds the new object files
to the arguments passed to the linker.

The additional command @samp{--amtmpfile} (not to be specified
by the user!) is passed to child GPC processes, so all compiles
use the same temporary file.

The source for this is merely in @file{gpc-module.c}, but there are
also some hacks in @file{gpc.c}, additional command line options in
@file{lang-options.h} and @file{gpc-decl.c}, and @file{gpc-defs.h}
contains declarations for the functions and global variables.


@c ========================================================================


@node File Layout
@section Files that make up GPC; Integrating GNU Pascal in GCC
@cindex File Layout, internals

@strong{NOTE:} This is not up to date.

@enumerate
@item
The GNU back end (gbe). Used to convert RTL into asm. Supposed to be
language independant. Files are in the GCC source directory.

@example
toplev.c
version.c
tree.c
print-tree.c
stor-layout.c
fold-const.c
function.c
stmt.c
expr.c
calls.c
expmed.c
explow.c
optabs.c
varasm.c
rtl.c
print-rtl.c
rtlanal.c
emit-rtl.c
real.c
dbxout.c
sdbout.c
dwarfout.c
xcoffout.c
integrate.c
jump.c
cse.c
loop.c
unroll.c
flow.c
stupid.c
combine.c
regclass.c
local-alloc.c
global.c
reload.c
reload1.c
caller-save.c
insn-peep.c
reorg.c
sched.c
final.c
recog.c
reg-stack.c
insn-opinit.c
insn-recog.c
insn-extract.c
insn-output.c
insn-emit.c
insn-attrtab.c
getpwd.c
convert.c
bc-emit.c
bc-optab.c
@end example

Unfortunately, some of them are not completely language independent
and need patching for GPC. This patch is in the @file{p/diffs}
subdirectory. It affects these files:

@example
convert.c
dbxout.c
expr.c
fold-const.c
function.c
optabs.c
stor-layout.c
toplev.c
@end example

@item
Pascal language implementation files hacked from GCC source. In the
ideal world, we would optimize them so much for Pascal that their
GCC roots would no longer be recognizable. ;@minus{}) These files are in
the @file{p/} subdirectory and are the following:

@example
gpcpp.c       (cccp.c)
gpc-common.c  (c-common.c)
gpc-convert.c (c-convert.c)
gpc-decl.c    (c-decl.c)
lang.c        (c-lang.c)
gpc-lex.c     (c-lex.c)
gpc-typeck.c  (c-typeck.c)
@end example

Some are even reused unmodified and are still in the GCC source
directory:

@example
c-aux-info.c
c-iterate.c
c-pragma.c
@end example

@item
GPC sources written from scratch. They are in the @file{p/}
subdirectory.

@example
gpc-defs.h
hash.h
module.c
parse.y
util.c
gpc.c
gpi-hash.c
gpi-hash.h
circle-buf.c
@end example

@end enumerate
