book

Build Your Own Programming Language

Name: Build Your Own Programming Language
Author: Clinton L. Jeffery
ISBN: 9781800204805

by Clinton L. Jeffery

December 2021

Intermediate to advanced

494 pages

10h 52m

English

Packt Publishing

Read now

Unlock full access

Build Your Own Programming Language
ContributorsAbout the authorAbout the reviewers
Preface
Who this book is forWhat this book coversTo get the most out of this bookDownload the example code filesCode in ActionDownload the color imagesConventions usedGet in touchShare Your Thoughts
Section 1: Programming Language Frontends
Chapter 1: Why Build Another Programming Language?
So, you want to write your own programming language…Types of programming language implementationsOrganizing a bytecode language implementationLanguages used in the examplesLanguage versus library – what's the difference?Applicability to other software engineering tasksEstablishing the requirements for your languageCase study – requirements that inspired the Unicon languageUnicon requirement #1 – preserve what people love about IconUnicon requirement #2 – support large-scale programs working on big dataUnicon requirement #3 – high-level input/output for modern applicationsUnicon requirement #4 – provide universally implementable system interfacesSummaryQuestions
Chapter 2: Programming Language Design
Determining the kinds of words and punctuation to provide in your languageSpecifying the control flowDeciding on what kinds of data to supportAtomic typesComposite typesDomain-specific typesOverall program structureCompleting the Jzero language definitionCase study – designing graphics facilities in UniconLanguage support for 2D graphicsAdding support for 3D graphicsSummaryQuestions
Chapter 3: Scanning Source Code
Technical requirementsLexemes, lexical categories, and tokensRegular expressionsRegular expression rulesRegular expression examplesUsing UFlex and JFlexHeader sectionRegular expressions sectionWriting a simple source code scannerRunning your scannerTokens and lexical attributesExpanding our example to construct tokensWriting a scanner for JzeroThe Jzero flex specificationUnicon Jzero codeJava Jzero codeRunning the Jzero scannerRegular expressions are not always enoughSummaryQuestions
Chapter 4: Parsing
Technical requirementsAnalyzing syntaxUnderstanding context-free grammarsWriting context-free grammar rulesWriting rules for programming constructsUsing iyacc and BYACC/JDeclaring symbols in the header sectionPutting together the yacc context-free grammar sectionUnderstanding yacc parsersFixing conflicts in yacc parsersSyntax error recoveryPutting together a toy exampleWriting a parser for JzeroThe Jzero lex specificationThe Jzero yacc specificationUnicon Jzero codeJava Jzero parser codeRunning the Jzero parserImproving syntax error messagesAdding detail to Unicon syntax error messagesAdding detail to Java syntax error messagesUsing Merr to generate better syntax error messagesSummaryQuestions
Chapter 5: Syntax Trees
Technical requirementsUsing GNU makeLearning about treesDefining a syntax tree typeParse trees versus syntax treesCreating leaves from terminal symbolsWrapping tokens in leavesWorking with YACC's value stackWrapping leaves for the parser's value stackDetermining which leaves you needBuilding internal nodes from production rulesAccessing tree nodes on the value stackUsing the tree node factory methodForming syntax trees for the Jzero languageDebugging and testing your syntax treeAvoiding common syntax tree bugsPrinting your tree in a text formatPrinting your tree using dotSummaryQuestions
Section 2: Syntax Tree Traversals
Chapter 6: Symbol Tables
Technical requirementsEstablishing the groundwork for symbol tablesDeclarations and scopesAssigning and dereferencing variablesChoosing the right tree traversal for the jobCreating and populating symbol tables for each scopeAdding semantic attributes to syntax treesDefining classes for symbol tables and symbol table entriesCreating symbol tablesPopulating symbol tablesSynthesizing the isConst attributeChecking for undeclared variablesIdentifying the bodies of methodsSpotting uses of variables within method bodiesFinding redeclared variablesInserting symbols into the symbol tableReporting semantic errorsHandling package and class scopes in Unicon Mangling namesInserting self for member variable referencesInserting self as the first parameter in method callsTesting and debugging symbol tablesSummaryQuestions

Chapter 7: Checking Base Types
Technical requirementsType representation in the compilerDefining a base class for representing typesSubclassing the base class for complex typesAssigning type information to declared variablesSynthesizing types from reserved wordsInheriting types into a list of variablesDetermining the type at each syntax tree nodeDetermining the type at the leavesCalculating and checking the types at internal nodesRuntime type checks and type inference in UniconSummaryQuestions
Chapter 8: Checking Types on Arrays, Method Calls, and Structure Accesses
Technical requirementsChecking operations on array typesHandling array variable declarationsChecking types during array creationChecking types during array accessesChecking method callsCalculating the parameters and return type informationChecking the types at each method call siteChecking the type at return statementsChecking structured type accessesHandling instance variable declarationsChecking types at instance creationChecking types at instance accessesSummaryQuestions
Chapter 9: Intermediate Code Generation
Technical requirementsPreparing to generate codeWhy generate intermediate code?Learning about the memory regions in the generated programIntroducing data types for intermediate codeAdding the intermediate code attributes to the treeGenerating labels and temporary variablesAn intermediate code instruction setInstructionsDeclarationsAnnotating syntax trees with labels for control flowGenerating code for expressionsGenerating code for control flowGenerating label targets for condition expressionsGenerating code for loopsGenerating intermediate code for method callsReviewing the generated intermediate codeSummary
Chapter 10: Syntax Coloring in an IDE
Downloading the example IDEs used in this chapterIntegrating a compiler into a programmer's editorAnalyzing source code from within the IDESending compiler output to the IDEAvoiding reparsing the entire file on every changeUsing lexical information to colorize tokensExtending the EditableTextList component to support colorColoring individual tokens as they are drawnHighlighting errors using parse resultsAdding Java supportSummary
Section 3: Code Generation and Runtime Systems
Chapter 11: Bytecode Interpreters
Technical requirementsUnderstanding what bytecode isComparing bytecode with intermediate codeBuilding a bytecode instruction set for JzeroDefining the Jzero bytecode file formatUnderstanding the basics of stack machine operationImplementing a bytecode interpreterLoading bytecode into memoryInitializing the interpreter stateFetching instructions and advancing the instruction pointerInstruction decodingExecuting instructionsStarting up the Jzero interpreterWriting a runtime system for JzeroRunning a Jzero programExamining iconx, the Unicon bytecode interpreterUnderstanding goal-directed bytecodeLeaving type information in at runtimeFetching, decoding, and executing instructionsCrafting the rest of the runtime systemSummaryQuestions
Chapter 12: Generating Bytecode
Technical requirementsConverting intermediate code to Jzero bytecodeAdding a class for bytecode instructionsMapping intermediate code addresses to bytecode addressesImplementing the bytecode generator methodGenerating bytecode for simple expressionsGenerating code for pointer manipulationGenerating bytecode for branches and conditional branchesGenerating code for method calls and returnsHandling labels and other pseudo-instructions in intermediate codeComparing bytecode assembler with binary formatsPrinting bytecode in assembler formatPrinting bytecode in binary formatLinking, loading, and including the runtime systemUnicon example – bytecode generation in icontSummaryQuestions
Chapter 13: Native Code Generation
Technical requirementsDeciding whether to generate native codeIntroducing the x64 instruction setAdding a class for x64 instructionsMapping memory regions to x64 register-based address modesUsing registersStarting from a null strategyAssigning registers to speed up the local regionConverting intermediate code to x64 codeMapping intermediate code addresses to x64 locationsImplementing the x64 code generator methodGenerating x64 code for simple expressionsGenerating code for pointer manipulationGenerating native code for branches and conditional branchesGenerating code for method calls and returnsHandling labels and pseudo-instructionsGenerating x64 outputWriting the x64 code in assembly language formatGoing from native assembler to an object fileLinking, loading, and including the runtime systemSummaryQuestions
Chapter 14: Implementing Operators and Built-In Functions
Implementing operatorsAsking whether operators imply hardware support and vice versaAdding String concatenation to intermediate code generationAdding String concatenation to the bytecode interpreterAdding String concatenation to the native runtime systemWriting built-in functionsAdding built-in functions to the bytecode interpreterWriting built-in functions for use with the native code implementationIntegrating built-ins with control structuresDeveloping operators and functions for UniconWriting operators in UniconDeveloping Unicon's built-in functionsSummaryQuestions
Chapter 15: Domain Control Structures
Knowing when you need a new control structureDefining what a control structure isReducing excessive redundant parametersScanning strings in Icon and UniconScanning environments and their primitive operationsEliminating excessive parameters via a control structureRendering regions in UniconRendering 3D graphics from a display listSpecifying rendering regions using built-in functionsVarying graphical levels of detail using nested rendering regionsCreating a rendering region control structureSummaryQuestions
Chapter 16: Garbage Collection
Appreciating the importance of garbage collectionCounting references to objectsAdding reference counting to JzeroGenerating code for heap allocationModifying the generated code for the assignment operatorConsidering the drawbacks and limitations of reference countingMarking live data and sweeping the restOrganizing heap memory regionsTraversing the basis to mark live dataReclaiming live memory and placing it into contiguous chunksSummaryQuestions
Chapter 17: Final Thoughts
Reflecting on what was learned from writing this bookDeciding where to go from hereStudying programming language designLearning about implementing interpreters and bytecode machinesAcquiring expertise in code optimizationMonitoring and debugging program executionsDesigning and implementing IDEs and GUI buildersExploring references for further readingStudying programming language designLearning about implementing interpreters and bytecode machinesAcquiring expertise in native code and code optimizationMonitoring and debugging program executionsDesigning and implementing IDEs and GUI buildersSummary
Section 4: Appendix
Appendix: Unicon Essentials
Running UniconUsing Unicon's declarations and data typesDeclaring different kinds of program componentsUsing atomic data typesOrganizing multiple values using structure typesEvaluating expressionsForming basic expressions using operatorsInvoking procedures, functions, and methodsIterating and selecting what and how to executeGeneratorsDebugging and environmental issuesLearning the basics of the UDB debuggerEnvironment variablesPreprocessorFunction mini-referenceSelected keywords
Assessments
Chapter 1Chapter 2Chapter 3Chapter 4Chapter 5Chapter 6Chapter 7Chapter 8Chapter 11Chapter 12Chapter 13Chapter 14Chapter 15Chapter 16
Why subscribe?
Other Books You May EnjoyPackt is searching for authors like youShare Your Thoughts

Content preview from Build Your Own Programming Language

Chapter 10: Syntax Coloring in an IDE

Creating a useful programming language requires more than just a compiler or interpreter that makes it possible to run programs—it requires an ecosystem of tools for developers. This ecosystem often includes debuggers, online help, or an integrated development environment, commonly called an IDE. An IDE can be broadly defined as any programming environment in which source code editing, compilation, linking steps (if any), and execution may all be performed within the same user interface (UI).

This chapter addresses some of the challenges of incorporating code from your programming language implementation into an IDE to provide syntax coloring and visual feedback about syntax errors. One reason that you want ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781800204805

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Build Your Own Programming Language

by Clinton L. Jeffery

Chapter 10: Syntax Coloring in an IDE

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.