Menu Close

Articles

Level: Console Application

Level console is an interactive program that is encapsulate with the compiler into a single application. I have enhanced the Level main program to be able to run the console using parameters. This program is almost like an interpretor. It is based on commands that you can use to load, compiler run and debug a Level-PY program.

Install

The console application is part of Level-PY project. This require Python 3.x to be installed on your computer. Then you can use GitHub desktop application to clone the project on your computer. After cloning you will be able to run the console.

Running console

To start the console on Windows you can use a shortcut. The shortcut for starting level console available in the project and may required some modifications for your computer. You can click the shortcut Level.lnk to start.

Parameters:

For main program level.py I have added several parameters to enhance the startup configuration. All parameters are optional including the program file.

-h   --help    : display a short description of parameters
-p   --pfile   : program file to pe compiled/parse
-a   --action  : one of {run,compile,debug,parser,scanner,lexer}
-d   --driver  : one of {batch,console,service}
-o   --home    : a folder where the level programs are located
-m   --mode    : debug mode {trace,regular,silent}

To avoid writing and rewriting parameters for running console I have created a batch file to start Level console. This can be used as an example to create other batch files to start Level compiler with different configurations.

Console Commands:

When Level console is active, the following commands are available:

CommandCommand DescriptionStatus
Help display this helpdone
Exit end Level consoledone
Quit end Level consoledone
Lex execute lexerdone
Scan execute scannerdone
Parse parse a programdone
Run run a programin progress
Next execute next stepin progress
Clear remov parsed programs from memorydone
Resume  exit debug mode and continuein progress
Stop stop execution of current programin progress
Status display current statusdone
Debug change debug leveldone
Config  display configurationdone
Home change home path folderdone
Report  print a report (ast, st, status, program)done

Some commands use a parameter. For example following command will display the full syntax and description of parameters for “report” command:

lev:>help report

Console Purpose:

Level console can be used to parse, run and debug Level programs. The order of operations is important. After a program is parsed it can be run once or several times from memory. If debug mode is “manual” the program can be run “step by step”.

After a program is parsed and run several times, the “clear” command will remove parsed programs from memory. Then a program can be parsed again. Console application should not stop until you type quit or exit command even if the program contains errors.

Debugging:

Console can run “step by step”. In debug mode, the following commands are usable: {next, resume, stop}. During the program runtime the debug mode can be changed but no other program can be parsed until the current program stop running using stop or resume.

Reporting:

There are 3 reports available: AST = “Abstract Syntax Tree”, ST = “Symbol Table” and “Program”. These 3 reports can be created using “report” commands any time and will contain information only after a program is parsed.

To print a report use following commands:

lev:>report st

lev:>report ast

lev:>report program

I hope you will find Level console interesting and easy to use.

After several programming iterations Level 1 parser can actually parse a program. I have used Python 3.5 and PyCharm to create this component.

At this time parser has already 1705 lines of code and will continue to grow until all use-cases are implemented. The parser main program is at the bottom of the program. This is how Python works and many other languages require the program to have the program root at the bottom and grow like a tree toward the top.

To parse a program one must use the Level Console with command “parse <program_name>”. After the program is parsed the program is stored in memory for investigation.

To investigate parser result one can use “report” console command to print a particular report. Until now there are 3 reports available: AST = Abstract Syntax Tree, ST = Symbol Table and Program = the program rebuild from AST.

If a program is parsed and fail, the AST and ST will contain the program up to the point of failure. So developer can investigate the memory status.

Example parsing hello.lev program:

This parse was done with mode -m trace. Therefore the result of the parser is display clearly so we do not have to report the AST. This trace is almost identical to AST. First column is the line number second is the column number. I represents the indentation. When i=-1 there is no indentation, 0 means the token is at the beginning of the row and > 0 means token is indented.

Reporting back the program shows the original program from memory:

Parser Job:

The parser is building AST and ST. Also parser must do verify program syntax against possible errors. Until now the parser is able to do the following verification:

  1. Does the statement end with “;” ?
  2. Does the section terminate with “end <section>?
  3. Is the statement properly indented?
  4. Has the statement a proper structure?
  5. If is an expression then is this expression correct?
  6. If is an assignment does the variable type match to expression type?
  7. If is a procedure call does all required parameters have values?
  8. If a variable is used is this variable initialized (have assigned value)?
  9. If is a function call is this function returning the proper type for expression?
  10. If I use a record or a matrix member is this member initialized?
  11. If I assign to a record or to a matrix a literal is this literal valid?

Know How

Level1 parser is a descendant parser implemented using recursive functions. Also is a predictive parser because I’m predicting what comes next from current position.

Making ST and AST: Parser is reading the program top down token by token using the Lexer. Every token is recorded in AST (Abstract Syntax Tree) in the form of a Node object.

If a token is a new identifier (type, variable, constant or subroutine) it is recorded into ST (Symbol Table) as an object. Level use a declaration for every new identifier. Every identifier must have a base-type and a type name. An identifier can’t be used until has the type established and is assigned.

In AST I do not afford do keep all information from tokens into Nodes. This could  be a waste of memory. The token contains the entire line of the program. Therefore the parser has a buffer (a list of tokens) where I keep only several tokens just in case I need to look back for a second. Every time when I read a new token I put the token at the end of the list and I discard the first element from the list. So this is like a queue of tokens.

Parsing flow

Parser is a class that is used to parse one single program or module. I use parser to store information about the status of parsing activity. Parser is used to create a hierarchy of name-spaces and build members for ST and AST.

Parser get tokens from Lexer one by one. When a new token is read the previous token is stored in AST as a primary Node or child Node. When a token represents an identifier is also added into the ST as an Identifier object. ST (Symbol Table) is a dictionary collection of nested NameSpaces.

Backing up.

When I find a token is a new identifier I must add this to symbol table (ST). However I may not know all information about the identifier until I read several more tokens. When the information is available I must go back and fix the identifier information in Symbol Table.

This is because Level syntax specify the identifier name first and type specification next unlike Java where the type is first and the identifier name is next.

Variable initialization:

In Level, initialization of variables can be done during declaration. For example:

variable

v_ex:0 integer;

When I do the assign “:0” I have not yet read the “integer” token. So I have no idea what kind of variable is going to be. Therefore the assign type verification is postponed until the information become available. This is called “late type verification”.

For now the Level compiler do not allow a full expression to be used in declaration initialization. I allow only simple expressions: constants or variables that are already initialized or literal expressions. The late type verification will be implemented in the near future.

Level Executor:

At this time the program can’t be executed. I have started to work on implementation for the executor. I have no prior experience so I’m doing my best to understand how is done. I’m a bad student so I will probably do a lot of mistakes and learn from them.

My plan is to create an interpreter that can be executed from the console. For this I have started evaluation of a simple expression. I will try to implement a REPL application.

A read–eval–print loop (REPL), also known as an interactive top-level or language shell, is a simple, interactive computer programming environment that takes single user inputs (i.e. single expressions), evaluates them, and returns the result to the user;

Until now I have implemented a Polish notation “expression resolver”. This is based on a function that receive a list of strings. Every string represent an expression element that can be a symbol, a function or a value.

Expresion resolver can be now used in Level Console with the new command:

lev:> eval <expression>

The expression can be a infix mathematical expression. Can contain round parentheses, numbers and operators: arithmetic: (+,-,*,/,^) relations (~ = < >) and logical (and or not true false).

The evaluator convert the expression from infix notation into post-fix notation. Then it try to resolve the expression and print the result. I have manage to resolve simple infix expressions.

For this I have created several new packages: levString, levNumeric, levLogic. In these packages I have created an “operator map” and a “function map” for every data type. These are implemented as a dictionary of pointers to Python native functions or to Level built-in functions. The levExecutor include these packages and use the maps to call specific operator implementation or function implementation.

Level Library

Built-in subroutine (procedure, function) must be available in the Name-Space of level Program. This is required for the Parser to know data types of built-in subroutine and parameter specifications. Therefore the signature of the function must be registered in Level library.

I need to create a new kind of region: “declare” region will describe the built-in subroutine but will not describe the implementation. These subroutines are called “external” and will be implemented in folder level/lib

The lib/default.lev module will include several other library modules. This is the module that is automatically loaded before any Level program and is imported into program name-Space as any other module.

If someone is reading and is good at this, I will accept any help. If not subscribe to my blog and post some comments. I’m curious what you think.

Color Syntax

I have check-in Level color syntax file for Notepad++. It works very well and can be used to write examples. Color syntax is helping preventing common mistakes like misspelling a word or not finishing a string with “.

Example of Level-2 program:

You can download syntax file for Notepad++ here: syntax-color.xml

Features:

  • Support Level-4 files with extension *.lev
  • All style of comments supported;
  • Strings and text are both recognized;
  • Expressions and collections are recognized;
  • Most of the keywords are implemented.

Read Next: Level: Structured Programming