JavaScript Source Text, Lexical Grammar & Parsing

Advanced

This lesson explains how ECMAScript source text becomes a syntactically recognized Script or Module. It owns lexical grammar, input elements, tokens, line terminators, comments, literals, lexical goals, parsing goals, directive prologues, and automatic semicolon insertion. It deliberately does not re-teach values, objects, functions, modules, or runtime execution semantics owned by other lessons.

Learning boundary

Use this lesson when you need to answer: “How does the language read this source?” Use the execution-model lesson for what happens after parsing, the Scripts and Modules lesson for linking/evaluation, and the regular-expressions lesson for RegExp pattern semantics.

Source text is the input to the ECMAScript grammar

An ECMAScript Script or Module begins as source text: Unicode code points that the language grammar processes. The source text is not executed character-by-character; it is first interpreted through lexical and syntactic grammar rules.

?

Lexical grammar turns characters into input elements

The lexical grammar scans source from left to right and recognizes input elements such as tokens, line terminators, comments, and white space. White space and comments generally separate or disappear before syntactic parsing, while line terminators can remain significant for parsing.

?

Tokens are not the same thing as source characters

Reserved words, identifiers, literals, and punctuators are ECMAScript tokens. A line terminator is an input element that can influence parsing, but it is not itself an ECMAScript token. This distinction matters when reasoning about automatic semicolon insertion.

?

Comments are lexical input, not program statements

Single-line and multi-line comments are handled by the lexical grammar. They do not become executable statements. A multi-line comment containing a line terminator can matter to parsing because that line terminator is preserved as a syntactic input element.

?

Identifiers have grammar rules

Identifier names are not arbitrary strings. ECMAScript defines identifier syntax, including Unicode-aware identifier characters and escape sequences. A source spelling can therefore be valid even when it is not limited to ASCII letters.

?

Literals have lexical forms

Numbers, strings, regular expression literals, templates, and other literal forms have grammar-defined source representations. The source spelling is parsed into a value or runtime structure rather than being treated as an arbitrary text fragment.

?

Regular expression literals need lexical context

The lexical grammar has multiple goal symbols because `/` can begin division or a regular expression literal depending on the surrounding syntactic context. This is one reason lexical and syntactic grammar cannot be treated as completely independent tokenizers.

?

Template literals have their own lexical components

Template literals combine literal text with embedded expressions. Their lexical components interact with the syntactic grammar, so the parser must recognize template tails and substitutions in the correct context.

?

Script and Module are distinct parse goals

ECMAScript defines two top-level syntactic goal symbols: Script and Module. Modules have module-specific syntax and semantics, including import/export declarations. This lesson explains the parse boundary; the Scripts and Modules lesson owns module linking and evaluation details.

?

Parsing produces a syntax structure before execution

A successful syntactic parse recognizes the source as a valid Script or Module and constructs a parse tree. Execution is a separate concern. This distinction prevents a syntax error from being confused with a runtime exception.

?

Automatic semicolon insertion is a parsing rule

ECMAScript can insert semicolons during parsing in specific circumstances. It does not simply insert one after every newline. The exact grammar context matters, which is why source formatting can change meaning in restricted cases.

?

Restricted productions make line breaks significant

Some productions contain a [no LineTerminator here] restriction. These include return/throw/yield behavior and postfix ++/--, among others. A line terminator in such a position changes which grammar production can apply.

?

ASI does not mean every newline ends a statement

A newline alone is not a universal statement terminator. For example, a following parenthesis, bracket, template, unary operator, or regular expression can allow the parser to continue the previous expression. Explicit semicolons can therefore be useful at statement boundaries.

?

Hashbang comments are source-level comments

ECMAScript recognizes a hashbang comment beginning with #! in the appropriate source position. It is discarded like other comments rather than becoming a JavaScript statement. Hosts can use it to identify an executable script environment.

?

Directive prologues are parsed string-literal statements

A directive prologue is a sequence of initial expression statements made from string literals. The special "use strict" directive is recognized there. The lexical/source-text lesson owns how this source construct is recognized; strict-mode consequences remain in the relevant language-semantic lessons.

?

Early syntax errors differ from runtime errors

Some invalid source is rejected before the program can execute normally. Other problems are only discovered while evaluating valid syntax. Keeping syntax errors separate from runtime exceptions helps learners diagnose the correct stage of failure.

?

Lexical grammar and syntactic grammar cooperate

The lexical grammar recognizes input elements; the syntactic grammar consumes the resulting tokens and relevant line terminators to recognize a Script or Module. Neither layer should be taught as a replacement for the other.

?

Author rules

Treat source text, lexical recognition, parsing, and execution as distinct stages. Do not assume a newline inserts a semicolon. Be especially careful around return, throw, yield, postfix ++/--, async syntax, and lines beginning with (, [, template literals, unary operators, or regular-expression literals. When debugging a failure, first decide whether the source is syntactically valid before reasoning about runtime behavior.

Specification map

ECMA-262 Clause 5 describes lexical and syntactic grammar conventions. Clause 11 defines source text and source-code-level constructs, and Clause 12 defines the lexical grammar and automatic semicolon insertion. Clauses 13–16 define the syntactic grammar for ECMAScript programs. The Scripts and Modules lesson owns module linking and evaluation; this lesson owns the source-to-parse boundary.