improve type safety of std.zig.Ast #22397

Techatrix · 2025-01-03T09:11:46Z

This PR introduces named/distinct integer types to std.zig.Ast, improving type safety and aligning it more closely with other compiler data structures like ZIR.

At it's core, the changes can be summarized by the following:

Before

// Most fields and declarations have been redacted for simplicity.
source: [:0]const u8,
tokens: TokenList.Slice,
nodes: NodeList.Slice,
extra_data: []Node.Index,

pub const TokenIndex = u32;

pub const Node = struct {
    tag: Tag,
    main_token: TokenIndex,
    data: Data,

    pub const Index = u32;

    pub const Tag = enum {...};

    pub const Data = struct {
        lhs: Index,
        rhs: Index,
    };
};

After

// Most fields and declarations have been redacted for simplicity.
source: [:0]const u8,
tokens: TokenList.Slice,
nodes: NodeList.Slice,
extra_data: []u32,

/// Index into `tokens`.
pub const TokenIndex = u32;

/// Index into `tokens`, or null.
pub const OptionalTokenIndex = enum(u32) {
    none = std.math.maxInt(u32),
    _,
};

/// Index into `extra_data`.
pub const ExtraIndex = enum(u32) { _ };

pub const Node = struct {
    tag: Tag,
    main_token: TokenIndex,
    data: Data,

    /// Index into `nodes`.
    pub const Index = enum(u32) { _ };

    /// Index into `nodes`, or null.
    pub const OptionalIndex = enum(u32) {
        none = std.math.maxInt(u32),
        _,
    };

    pub const Tag = enum {...};

    pub const Data = union {
        node: Index,
        opt_node: OptionalIndex,
        token: TokenIndex,
        node_and_node: struct { Index, Index },
        opt_node_and_opt_node: struct { OptionalIndex, OptionalIndex },
        node_and_opt_node: struct { Index, OptionalIndex },
        opt_node_and_node: struct { OptionalIndex, Index },
        node_and_extra: struct { Index, ExtraIndex },
        extra_and_node: struct { ExtraIndex, Index },
        extra_and_opt_node: struct { ExtraIndex, OptionalIndex },
        node_and_token: struct { Index, TokenIndex },
        token_and_node: struct { TokenIndex, Index },
        token_and_token: struct { TokenIndex, TokenIndex },
        opt_node_and_token: struct { OptionalIndex, TokenIndex },
        opt_token_and_node: struct { OptionalTokenIndex, Index },
        opt_token_and_opt_node: struct { OptionalTokenIndex, OptionalIndex },
        opt_token_and_opt_token: struct { OptionalTokenIndex, OptionalTokenIndex },
        @"for": struct { ExtraIndex, For },
        extra_range: SubRange,
    };
};

I initial changed TokenIndex to a named/distinct integer type as well but it made dealing with them unergonomic. This was especially noticable while porting lib/std/zig/render.zig because it frequently relied on relative token indicies.

The offset types (Node.Offset, TokenOffset, etc.) have been added for ZIR because it stores relative ast indicies as opposed to absolute indicies.

zig ast-check src/Sema.zig (ReleaseFast)

$ poop "build/stage4-fast/bin/zig ast-check src/Sema.zig" "build/stage4-ast-fast/bin/zig ast-check src/Sema.zig" -d 20000
Benchmark 1 (189 runs): build/stage4-fast/bin/zig ast-check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           106ms ± 2.12ms     103ms …  111ms          3 ( 2%)        0%
  peak_rss           77.4MB ± 90.0KB    76.9MB … 77.4MB          2 ( 1%)        0%
  cpu_cycles          346M  ± 5.94M      339M  …  359M          22 (12%)        0%
  instructions        522M  ± 9.32       522M  …  522M           0 ( 0%)        0%
  cache_references   1.44M  ± 61.6K     1.29M  … 1.60M           8 ( 4%)        0%
  cache_misses        330K  ± 25.6K      292K  …  392K           0 ( 0%)        0%
  branch_misses      4.86M  ± 1.99K     4.85M  … 4.86M           3 ( 2%)        0%
Benchmark 2 (186 runs): build/stage4-ast-fast/bin/zig ast-check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           108ms ± 2.00ms     105ms …  112ms          0 ( 0%)        💩+  1.9% ±  0.4%
  peak_rss           77.3MB ± 80.6KB    77.0MB … 77.3MB         23 (12%)          -  0.2% ±  0.0%
  cpu_cycles          353M  ± 5.65M      345M  …  366M           0 ( 0%)        💩+  2.1% ±  0.3%
  instructions        532M  ± 9.90       532M  …  532M           1 ( 1%)        💩+  1.9% ±  0.0%
  cache_references   1.41M  ± 69.9K     1.27M  … 1.58M           0 ( 0%)        ⚡-  2.0% ±  0.9%
  cache_misses        318K  ± 25.3K      290K  …  390K          17 ( 9%)        ⚡-  3.5% ±  1.6%
  branch_misses      4.82M  ± 1.82K     4.82M  … 4.83M           1 ( 1%)          -  0.7% ±  0.0%

zig fmt --check src/Sema.zig (ReleaseFast)

$ poop "build/stage4-fast/bin/zig fmt --check src/Sema.zig" "build/stage4-ast-fast/bin/zig fmt --check src/Sema.zig" -d 20000
Benchmark 1 (337 runs): build/stage4-fast/bin/zig fmt --check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time          59.3ms ± 1.03ms    58.1ms … 63.3ms          5 ( 1%)        0%
  peak_rss           68.1MB ± 69.5KB    67.9MB … 68.2MB          0 ( 0%)        0%
  cpu_cycles          179M  ± 3.30M      176M  …  192M           6 ( 2%)        0%
  instructions        360M  ± 9.59       360M  …  360M           0 ( 0%)        0%
  cache_references    626K  ± 28.3K      546K  …  668K          49 (15%)        0%
  cache_misses        263K  ± 17.7K      237K  …  333K          94 (28%)        0%
  branch_misses      1.25M  ± 1.34K     1.25M  … 1.26M          12 ( 4%)        0%
Benchmark 2 (335 runs): build/stage4-ast-fast/bin/zig fmt --check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time          59.8ms ±  957us    58.8ms … 62.9ms         36 (11%)          +  0.8% ±  0.3%
  peak_rss           68.0MB ± 65.9KB    67.8MB … 68.1MB         73 (22%)          -  0.1% ±  0.0%
  cpu_cycles          180M  ± 2.95M      178M  …  191M          64 (19%)          +  0.9% ±  0.3%
  instructions        370M  ± 9.17       370M  …  370M           0 ( 0%)        💩+  2.5% ±  0.0%
  cache_references    625K  ± 23.0K      543K  …  663K          42 (13%)          -  0.1% ±  0.6%
  cache_misses        265K  ± 17.7K      239K  …  334K          76 (23%)          +  0.6% ±  1.0%
  branch_misses      1.23M  ± 1.81K     1.22M  … 1.23M           2 ( 1%)        ⚡-  2.0% ±  0.0%

zig ast-check src/Sema.zig (Debug)

$ poop "build/stage4/bin/zig ast-check src/Sema.zig" "build/stage4-ast/bin/zig ast-check src/Sema.zig" -d 20000
Benchmark 1 (34 runs): build/stage4/bin/zig ast-check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           596ms ± 11.8ms     591ms …  662ms          1 ( 3%)        0%
  peak_rss           82.9MB ± 99.3KB    82.7MB … 83.0MB          0 ( 0%)        0%
  cpu_cycles         2.23G  ± 5.77M     2.22G  … 2.24G           2 ( 6%)        0%
  instructions       3.75G  ± 34.8K     3.75G  … 3.75G           0 ( 0%)        0%
  cache_references   10.7M  ±  195K     10.0M  … 11.1M           1 ( 3%)        0%
  cache_misses        519K  ± 29.3K      478K  …  576K           0 ( 0%)        0%
  branch_misses      5.57M  ± 12.1K     5.56M  … 5.62M           2 ( 6%)        0%
Benchmark 2 (32 runs): build/stage4-ast/bin/zig ast-check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           637ms ± 1.97ms     634ms …  644ms          1 ( 3%)        💩+  7.0% ±  0.7%
  peak_rss           85.0MB ± 83.6KB    84.8MB … 85.1MB          8 (25%)        💩+  2.6% ±  0.1%
  cpu_cycles         2.39G  ± 4.59M     2.38G  … 2.41G           1 ( 3%)        💩+  7.4% ±  0.1%
  instructions       3.89G  ± 58.7K     3.89G  … 3.89G           0 ( 0%)        💩+  3.9% ±  0.0%
  cache_references   11.7M  ±  235K     11.3M  … 12.3M           1 ( 3%)        💩+  9.4% ±  1.0%
  cache_misses        555K  ± 33.6K      511K  …  631K           0 ( 0%)        💩+  7.0% ±  3.0%
  branch_misses      5.49M  ± 11.2K     5.47M  … 5.51M           0 ( 0%)        ⚡-  1.4% ±  0.1%

zig fmt --check src/Sema.zig (Debug)

$ poop "build/stage4/bin/zig fmt --check src/Sema.zig" "build/stage4-ast/bin/zig fmt --check src/Sema.zig" -d 20000
Benchmark 1 (63 runs): build/stage4/bin/zig fmt --check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           322ms ± 2.75ms     317ms …  328ms          0 ( 0%)        0%
  peak_rss           71.7MB ± 50.8KB    71.6MB … 71.8MB         10 (16%)        0%
  cpu_cycles         1.19G  ± 9.93M     1.18G  … 1.21G           0 ( 0%)        0%
  instructions       2.21G  ± 2.17K     2.21G  … 2.21G           0 ( 0%)        0%
  cache_references   1.84M  ± 89.5K     1.64M  … 2.05M           0 ( 0%)        0%
  cache_misses        324K  ± 32.4K      266K  …  376K           0 ( 0%)        0%
  branch_misses      2.72M  ±  198K     2.54M  … 3.36M           1 ( 2%)        0%
Benchmark 2 (57 runs): build/stage4-ast/bin/zig fmt --check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           356ms ± 3.04ms     351ms …  364ms          0 ( 0%)        💩+ 10.7% ±  0.3%
  peak_rss           73.7MB ± 34.9KB    73.5MB … 73.7MB          4 ( 7%)        💩+  2.7% ±  0.0%
  cpu_cycles         1.32G  ± 11.3M     1.30G  … 1.35G           3 ( 5%)        💩+ 11.0% ±  0.3%
  instructions       2.45G  ±  590      2.45G  … 2.45G           0 ( 0%)        💩+ 10.6% ±  0.0%
  cache_references   1.93M  ±  113K     1.70M  … 2.20M           0 ( 0%)        💩+  4.6% ±  2.0%
  cache_misses        351K  ± 26.7K      289K  …  393K           0 ( 0%)        💩+  8.4% ±  3.3%
  branch_misses      2.86M  ±  210K     2.56M  … 3.45M           2 ( 4%)        💩+  5.1% ±  2.7%

mlugg · 2025-01-03T15:12:29Z

I initial changed TokenIndex to a named/distinct integer type as well but it made dealing with them unergonomic.

Did you try having an offset method?

fn offset(tok: TokenIndex, off: i32) TokenIndex {
    const new_tok = @as(i64, @intFromEnum(tok)) + off;
    return @enumFromInt(new_tok);
}

Regardless: this PR seems like a fantastic change. My main concern just from the PR description is that final performance data point -- strictly speaking, this change shouldn't impact performance at all. Do you have any idea why the last performance number is consistently so bad?

Techatrix · 2025-01-03T15:57:32Z

Did you try having an offset method?

Kind of. I had next, previous, shift and shiftBackwards methods. I don't have the performance data anymore, but before I undid the change to TokenIndex, zig fmt --check src/Sema.zig (Debug) Benchmark was at least a 30% performance regression. What would previously be simple add or subtract on a token index would require a function call and lib/std/zig/render.zig did a lot of them.

My main concern just from the PR description is that final performance data point -- strictly speaking, this change shouldn't impact performance at all. Do you have any idea why the last performance number is consistently so bad?

Here are some of the possibles reasons:

accessing the fields of nodes and tokens goes through a function call (tree.nodes.items(.data)[node] -> tree.nodeData(node))
@sizeOf(Ast.Node.Data) has increased from 8 to 16 in debug builds
accessing Ast.Node.Data.Item in a debug builds has a safety check on each field access

Removing the tokenTag and tokenStart functions by manually inlining them may help to improve performance (in debug builds).

RetroDev256 · 2025-01-03T17:53:41Z

If I remember correctly, wasn't null for optional indexes represented as zero? The PR writeup shows none = std.math.maxInt(u32), and I wonder if it could remain zero.

mlugg · 2025-01-03T18:12:50Z

maxInt(u32) is a much better value for .none than 0. With 0, you need to make sure it's not in a context where the first token/node is actually valid, or you need to reserve a dummy entry in the array. maxInt(u32) avoids this downside.

rohlem · 2025-01-03T18:44:09Z

zig fmt --check src/Sema.zig (Debug) Benchmark was at least a 30% performance regression. What would previously be simple add or subtract on a token index would require a function call [...]

accessing the fields of nodes and tokens goes through a function call (tree.nodes.items(.data)[node] -> tree.nodeData(node))

[...] manually inlining them may help to improve performance

All of these points seem like a perfectly valid use case for an inline fn / callconv(.@"inline") to me.
(Well, long-term the compiler's inlining heuristic should instead be improved, but that can be done later.)
Did that also not work / am I missing some reason the compiler inlining them w-/shouldn't be exactly equivalent?

mlugg · 2025-01-03T22:31:34Z

It might be worth seeing whether marking functions inline can improve debug performance -- if it does, I think this would be a valid use of inline.

However, I don't see a 10% regression in Debug performance for an already fairly fast subcommand as a huge deal, so this PR is fine as-is if you don't feel like looking into that.

Techatrix · 2025-01-04T12:30:34Z

Unfortunately, marking the functions as inline made the performance worse in debug builds.

zig ast-check src/Sema.zig (ReleaseFast)

poop "build/stage4-fast/bin/zig ast-check src/Sema.zig" \
     "build/stage4-ast-fast/bin/zig ast-check src/Sema.zig" \
     "build/stage4-ast-inline-fast/bin/zig ast-check src/Sema.zig" -d 30000

Benchmark 1 (288 runs): build/stage4-fast/bin/zig ast-check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           104ms ±  906us     102ms …  107ms          1 ( 0%)        0%
  peak_rss           77.6MB ± 66.0KB    77.4MB … 77.7MB         69 (24%)        0%
  cpu_cycles          340M  ± 2.74M      336M  …  351M           1 ( 0%)        0%
  instructions        523M  ± 7.68       523M  …  523M          22 ( 8%)        0%
  cache_references   1.42M  ± 31.3K     1.34M  … 1.52M           6 ( 2%)        0%
  cache_misses        348K  ± 32.0K      309K  …  402K           0 ( 0%)        0%
  branch_misses      4.83M  ± 2.67K     4.83M  … 4.84M          11 ( 4%)        0%
Benchmark 2 (283 runs): build/stage4-ast-fast/bin/zig ast-check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           106ms ±  813us     104ms …  108ms          0 ( 0%)        💩+  2.0% ±  0.1%
  peak_rss           77.6MB ± 49.4KB    77.4MB … 77.7MB         71 (25%)          -  0.1% ±  0.0%
  cpu_cycles          348M  ± 2.46M      344M  …  354M           0 ( 0%)        💩+  2.2% ±  0.1%
  instructions        533M  ± 5.64       533M  …  533M           9 ( 3%)        💩+  1.8% ±  0.0%
  cache_references   1.48M  ± 30.8K     1.41M  … 1.58M           2 ( 1%)        💩+  4.2% ±  0.4%
  cache_misses        340K  ± 29.1K      308K  …  395K           0 ( 0%)        ⚡-  2.5% ±  1.4%
  branch_misses      4.89M  ± 3.15K     4.88M  … 4.90M           1 ( 0%)        💩+  1.2% ±  0.0%
Benchmark 3 (284 runs): build/stage4-ast-inline-fast/bin/zig ast-check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           105ms ±  891us     104ms …  108ms          0 ( 0%)        💩+  1.6% ±  0.1%
  peak_rss           77.6MB ± 60.5KB    77.3MB … 77.7MB         86 (30%)          -  0.1% ±  0.0%
  cpu_cycles          346M  ± 2.54M      342M  …  353M           0 ( 0%)        💩+  1.8% ±  0.1%
  instructions        533M  ± 7.38       533M  …  533M          10 ( 4%)        💩+  1.9% ±  0.0%
  cache_references   1.55M  ± 42.0K     1.45M  … 1.70M           3 ( 1%)        💩+  8.9% ±  0.4%
  cache_misses        343K  ± 31.1K      308K  …  395K           0 ( 0%)          -  1.6% ±  1.5%
  branch_misses      4.84M  ± 2.91K     4.83M  … 4.85M           9 ( 3%)          +  0.2% ±  0.0%

zig fmt --check src/Sema.zig (ReleaseFast)

poop "build/stage4-fast/bin/zig fmt --check src/Sema.zig" \
     "build/stage4-ast-fast/bin/zig fmt --check src/Sema.zig" \
     "build/stage4-ast-inline-fast/bin/zig fmt --check src/Sema.zig" -d 30000

Benchmark 1 (509 runs): build/stage4-fast/bin/zig fmt --check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time          58.2ms ±  629us    57.5ms … 61.3ms         62 (12%)        0%
  peak_rss           68.0MB ± 64.6KB    67.8MB … 68.1MB        117 (23%)        0%
  cpu_cycles          175M  ± 2.03M      174M  …  186M          90 (18%)        0%
  instructions        353M  ± 4.94       353M  …  353M           8 ( 2%)        0%
  cache_references    617K  ± 6.73K      606K  …  644K          29 ( 6%)        0%
  cache_misses        274K  ± 21.5K      260K  …  338K          98 (19%)        0%
  branch_misses      1.31M  ± 1.61K     1.30M  … 1.32M          17 ( 3%)        0%
Benchmark 2 (499 runs): build/stage4-ast-fast/bin/zig fmt --check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time          59.5ms ±  715us    58.7ms … 63.6ms         25 ( 5%)        💩+  2.3% ±  0.1%
  peak_rss           68.2MB ± 40.2KB    68.0MB … 68.2MB         88 (18%)          +  0.2% ±  0.0%
  cpu_cycles          180M  ± 2.08M      179M  …  193M          66 (13%)        💩+  2.9% ±  0.1%
  instructions        363M  ± 5.74       363M  …  363M          17 ( 3%)        💩+  2.8% ±  0.0%
  cache_references    637K  ± 8.31K      618K  …  678K           5 ( 1%)        💩+  3.2% ±  0.2%
  cache_misses        275K  ± 22.6K      259K  …  338K          86 (17%)          +  0.7% ±  1.0%
  branch_misses      1.33M  ± 1.67K     1.32M  … 1.33M           6 ( 1%)        💩+  1.5% ±  0.0%
Benchmark 3 (500 runs): build/stage4-ast-inline-fast/bin/zig fmt --check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time          59.4ms ±  789us    58.3ms … 63.2ms          5 ( 1%)        💩+  2.1% ±  0.2%
  peak_rss           68.2MB ± 58.4KB    67.9MB … 68.2MB        102 (20%)          +  0.2% ±  0.0%
  cpu_cycles          179M  ± 2.69M      177M  …  195M           7 ( 1%)        💩+  2.4% ±  0.2%
  instructions        363M  ± 4.89       363M  …  363M           5 ( 1%)        💩+  2.8% ±  0.0%
  cache_references    631K  ± 9.62K      575K  …  671K           4 ( 1%)        💩+  2.2% ±  0.2%
  cache_misses        285K  ± 28.2K      247K  …  338K           0 ( 0%)        💩+  4.2% ±  1.1%
  branch_misses      1.31M  ± 1.56K     1.31M  … 1.32M           3 ( 1%)          +  0.2% ±  0.0%

zig ast-check src/Sema.zig (Debug)

poop "build/stage4/bin/zig ast-check src/Sema.zig" \
     "build/stage4-ast/bin/zig ast-check src/Sema.zig" \
     "build/stage4-ast-inline/bin/zig ast-check src/Sema.zig" -d 30000

Benchmark 1 (52 runs): build/stage4/bin/zig ast-check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           586ms ± 8.94ms     583ms …  648ms          1 ( 2%)        0%
  peak_rss           84.5MB ± 93.2KB    84.2MB … 84.6MB          1 ( 2%)        0%
  cpu_cycles         2.20G  ± 3.81M     2.19G  … 2.21G           0 ( 0%)        0%
  instructions       3.73G  ± 15.9      3.73G  … 3.73G           7 (13%)        0%
  cache_references   11.1M  ±  228K     10.7M  … 11.7M           0 ( 0%)        0%
  cache_misses        536K  ± 38.4K      495K  …  600K           0 ( 0%)        0%
  branch_misses      5.79M  ± 9.13K     5.78M  … 5.83M           2 ( 4%)        0%
Benchmark 2 (50 runs): build/stage4-ast/bin/zig ast-check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           603ms ± 3.12ms     601ms …  624ms          1 ( 2%)        💩+  2.9% ±  0.5%
  peak_rss           86.0MB ± 95.2KB    85.7MB … 86.1MB          1 ( 2%)        💩+  1.7% ±  0.0%
  cpu_cycles         2.27G  ± 2.94M     2.26G  … 2.27G           0 ( 0%)        💩+  3.1% ±  0.1%
  instructions       3.86G  ± 6.44      3.86G  … 3.86G           3 ( 6%)        💩+  3.6% ±  0.0%
  cache_references   10.8M  ±  258K     10.4M  … 11.5M           1 ( 2%)        ⚡-  2.1% ±  0.9%
  cache_misses        584K  ± 33.2K      522K  …  628K           0 ( 0%)        💩+  9.0% ±  2.6%
  branch_misses      5.68M  ± 7.37K     5.67M  … 5.72M           1 ( 2%)        ⚡-  2.0% ±  0.1%
Benchmark 3 (50 runs): build/stage4-ast-inline/bin/zig ast-check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           611ms ± 1.19ms     609ms …  616ms          1 ( 2%)        💩+  4.3% ±  0.4%
  peak_rss           86.5MB ± 79.9KB    86.3MB … 86.6MB          0 ( 0%)        💩+  2.3% ±  0.0%
  cpu_cycles         2.30G  ± 3.94M     2.29G  … 2.31G           1 ( 2%)        💩+  4.6% ±  0.1%
  instructions       3.88G  ± 6.90      3.88G  … 3.88G           2 ( 4%)        💩+  4.0% ±  0.0%
  cache_references   12.6M  ±  343K     12.1M  … 14.2M           1 ( 2%)        💩+ 14.0% ±  1.0%
  cache_misses        608K  ± 23.4K      521K  …  636K           3 ( 6%)        💩+ 13.4% ±  2.3%
  branch_misses      5.72M  ± 9.50K     5.71M  … 5.77M           2 ( 4%)        ⚡-  1.2% ±  0.1%

zig fmt --check src/Sema.zig (Debug)

poop "build/stage4/bin/zig fmt --check src/Sema.zig" \
     "build/stage4-ast/bin/zig fmt --check src/Sema.zig" \
     "build/stage4-ast-inline/bin/zig fmt --check src/Sema.zig" -d 30000

Benchmark 1 (93 runs): build/stage4/bin/zig fmt --check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           323ms ± 2.94ms     319ms …  332ms          0 ( 0%)        0%
  peak_rss           73.0MB ± 74.3KB    72.8MB … 73.1MB          6 ( 6%)        0%
  cpu_cycles         1.20G  ± 11.6M     1.17G  … 1.23G           0 ( 0%)        0%
  instructions       2.22G  ± 6.77      2.22G  … 2.22G           1 ( 1%)        0%
  cache_references   1.97M  ±  125K     1.72M  … 2.45M           2 ( 2%)        0%
  cache_misses        359K  ± 14.9K      300K  …  419K          13 (14%)        0%
  branch_misses      3.19M  ±  281K     2.71M  … 3.82M           0 ( 0%)        0%
Benchmark 2 (84 runs): build/stage4-ast/bin/zig fmt --check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           360ms ± 3.66ms     354ms …  372ms          4 ( 5%)        💩+ 11.2% ±  0.3%
  peak_rss           74.9MB ± 63.2KB    74.7MB … 74.9MB         18 (21%)        💩+  2.5% ±  0.0%
  cpu_cycles         1.33G  ± 13.2M     1.32G  … 1.38G           4 ( 5%)        💩+ 11.6% ±  0.3%
  instructions       2.54G  ± 11.9      2.54G  … 2.54G           7 ( 8%)        💩+ 14.5% ±  0.0%
  cache_references   1.79M  ± 92.2K     1.67M  … 2.20M           2 ( 2%)        ⚡-  9.2% ±  1.7%
  cache_misses        372K  ± 23.4K      316K  …  417K          11 (13%)        💩+  3.5% ±  1.6%
  branch_misses      3.21M  ±  386K     2.78M  … 4.44M           5 ( 6%)          +  0.5% ±  3.1%
Benchmark 3 (73 runs): build/stage4-ast-inline/bin/zig fmt --check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           415ms ± 2.64ms     411ms …  422ms          1 ( 1%)        💩+ 28.5% ±  0.3%
  peak_rss           75.1MB ± 43.5KB    75.0MB … 75.2MB         13 (18%)        💩+  2.9% ±  0.0%
  cpu_cycles         1.55G  ± 10.00M    1.54G  … 1.58G           5 ( 7%)        💩+ 29.6% ±  0.3%
  instructions       2.60G  ± 8.63      2.60G  … 2.60G           5 ( 7%)        💩+ 17.1% ±  0.0%
  cache_references   2.86M  ±  167K     2.59M  … 3.45M           3 ( 4%)        💩+ 44.9% ±  2.2%
  cache_misses        363K  ± 23.8K      317K  …  392K           0 ( 0%)          +  1.0% ±  1.7%
  branch_misses      3.74M  ±  399K     3.13M  … 4.98M           3 ( 4%)        💩+ 17.2% ±  3.2%

Patch

diff --git a/lib/std/zig/Ast.zig b/lib/std/zig/Ast.zig
index 686a2e884d..28d62b7ad2 100644
--- a/lib/std/zig/Ast.zig
+++ b/lib/std/zig/Ast.zig
@@ -75,23 +75,23 @@ pub const OptionalTokenOffset = enum(i32) {
     }
 };
 
-pub fn tokenTag(tree: *const Ast, token_index: TokenIndex) Token.Tag {
+pub inline fn tokenTag(tree: *const Ast, token_index: TokenIndex) Token.Tag {
     return tree.tokens.items(.tag)[token_index];
 }
 
-pub fn tokenStart(tree: *const Ast, token_index: TokenIndex) ByteOffset {
+pub inline fn tokenStart(tree: *const Ast, token_index: TokenIndex) ByteOffset {
     return tree.tokens.items(.start)[token_index];
 }
 
-pub fn nodeTag(tree: *const Ast, node: Node.Index) Node.Tag {
+pub inline fn nodeTag(tree: *const Ast, node: Node.Index) Node.Tag {
     return tree.nodes.items(.tag)[@intFromEnum(node)];
 }
 
-pub fn nodeMainToken(tree: *const Ast, node: Node.Index) TokenIndex {
+pub inline fn nodeMainToken(tree: *const Ast, node: Node.Index) TokenIndex {
     return tree.nodes.items(.main_token)[@intFromEnum(node)];
 }
 
-pub fn nodeData(tree: *const Ast, node: Node.Index) Node.Data {
+pub inline fn nodeData(tree: *const Ast, node: Node.Index) Node.Data {
     return tree.nodes.items(.data)[@intFromEnum(node)];
 }
 
diff --git a/lib/std/zig/Parse.zig b/lib/std/zig/Parse.zig
index abbc9b0fc1..8762cdfd8e 100644
--- a/lib/std/zig/Parse.zig
+++ b/lib/std/zig/Parse.zig
@@ -11,23 +11,23 @@ nodes: Ast.NodeList,
 extra_data: std.ArrayListUnmanaged(u32),
 scratch: std.ArrayListUnmanaged(Node.Index),
 
-fn tokenTag(p: *const Parse, token_index: TokenIndex) Token.Tag {
+inline fn tokenTag(p: *const Parse, token_index: TokenIndex) Token.Tag {
     return p.tokens.items(.tag)[token_index];
 }
 
-fn tokenStart(p: *const Parse, token_index: TokenIndex) Ast.ByteOffset {
+inline fn tokenStart(p: *const Parse, token_index: TokenIndex) Ast.ByteOffset {
     return p.tokens.items(.start)[token_index];
 }
 
-fn nodeTag(p: *const Parse, node: Node.Index) Node.Tag {
+inline fn nodeTag(p: *const Parse, node: Node.Index) Node.Tag {
     return p.nodes.items(.tag)[@intFromEnum(node)];
 }
 
-fn nodeMainToken(p: *const Parse, node: Node.Index) TokenIndex {
+inline fn nodeMainToken(p: *const Parse, node: Node.Index) TokenIndex {
     return p.nodes.items(.main_token)[@intFromEnum(node)];
 }
 
-fn nodeData(p: *const Parse, node: Node.Index) Node.Data {
+inline fn nodeData(p: *const Parse, node: Node.Index) Node.Data {
     return p.nodes.items(.data)[@intFromEnum(node)];
 }

andrewrk

Thanks for doing this.

The types aren't quite right though. To avoid 2 breakages rather than 1 let's get these right.

lib/std/zig/Ast.zig

This function checks for various possibilities that are never produced by the parser. Given that lastToken is unsafe to call on an Ast with errors, I also removed code paths that would be reachable on an Ast with errors.

fixes ziglang#22601

This commits adds the following distinct integer types to std.zig.Ast: - OptionalTokenIndex - TokenOffset - OptionalTokenOffset - Node.OptionalIndex - Node.Offset - Node.OptionalOffset The `Node.Index` type has also been converted to a distinct type while `TokenIndex` remains unchanged. `Ast.Node.Data` has also been changed to a (untagged) union to provide safety checks.

The existing comment are incomplete, outdated and sometimes incorrect.

…literal` The main_token already has the necessary information.

This makes the `.data` field the better choice over the `.main_token` for this tag.

andrewrk

Thanks, appreciate the follow-ups.

Adaptation zig 0.15 dev AST break change, see this [PR](ziglang/zig#22397)

Techatrix force-pushed the type-safe-ast branch 2 times, most recently from d842369 to 80c0304 Compare January 3, 2025 12:55

Techatrix force-pushed the type-safe-ast branch from 80c0304 to cb91fda Compare January 4, 2025 12:30

Techatrix force-pushed the type-safe-ast branch 2 times, most recently from 2b23652 to a8d3767 Compare January 14, 2025 17:58

Techatrix force-pushed the type-safe-ast branch from a8d3767 to baffeb8 Compare January 21, 2025 16:21

andrewrk requested changes Feb 3, 2025

View reviewed changes

lib/std/zig/Ast.zig Show resolved Hide resolved

lib/std/zig/Ast.zig Outdated Show resolved Hide resolved

Techatrix force-pushed the type-safe-ast branch 3 times, most recently from 0330052 to 7657993 Compare February 9, 2025 04:56

Techatrix force-pushed the type-safe-ast branch from 7657993 to bab85dd Compare February 26, 2025 19:38

Techatrix added 9 commits March 7, 2025 22:20

remove unreachable code paths from std.zig.Ast.lastToken

ad38af7

This function checks for various possibilities that are never produced by the parser. Given that lastToken is unsafe to call on an Ast with errors, I also removed code paths that would be reachable on an Ast with errors.

fix ZonGen error message when encountering an array access

c19f4c4

add a reference to ziglang#21690

e6596cb

aro_translate_c: fix ast lowering of continue node

de9c889

fixes ziglang#22601

std.zig.Ast: add blockStatements and builtinCallParams

6dcd8f4

std.zig.Ast: update doc comments of Node.Tag

ba74af2

The existing comment are incomplete, outdated and sometimes incorrect.

std.zig.Ast: don't set the data field on .error_value and `.enum_…

d84055f

…literal` The main_token already has the necessary information.

std.zig.Ast: store lbrace and rbrace token in data of .error_set_decl

4129f7f

This makes the `.data` field the better choice over the `.main_token` for this tag.

Techatrix force-pushed the type-safe-ast branch from bab85dd to 4129f7f Compare March 7, 2025 22:17

andrewrk approved these changes Mar 12, 2025

View reviewed changes

mlugg merged commit d091178 into ziglang:master Mar 12, 2025
17 of 18 checks passed

Techatrix mentioned this pull request Mar 13, 2025

Use new type-safe AST zigtools/zls#2229

Closed

sc68cal mentioned this pull request Mar 14, 2025

Zig 0.15.0 broken build jetzig-framework/jetzig#176

Closed

jinzhongjia added a commit to jinzhongjia/zig-lamp that referenced this pull request Mar 16, 2025

feat: use smpallocator to get better performance

0ab9751

Adaptation zig 0.15 dev AST break change, see this [PR](ziglang/zig#22397)

jinzhongjia added a commit to jinzhongjia/zig-lamp that referenced this pull request Mar 16, 2025

feat: use smpallocator to get better performance

e5de2a0

Adaptation zig 0.15 dev AST break change, see this [PR](ziglang/zig#22397)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

improve type safety of std.zig.Ast #22397

improve type safety of std.zig.Ast #22397

Techatrix commented Jan 3, 2025 •

edited

Loading

mlugg commented Jan 3, 2025

Techatrix commented Jan 3, 2025

RetroDev256 commented Jan 3, 2025

mlugg commented Jan 3, 2025 •

edited

Loading

rohlem commented Jan 3, 2025 •

edited

Loading

mlugg commented Jan 3, 2025

Techatrix commented Jan 4, 2025

andrewrk left a comment

andrewrk left a comment

improve type safety of std.zig.Ast #22397

improve type safety of std.zig.Ast #22397

Conversation

Techatrix commented Jan 3, 2025 • edited Loading

mlugg commented Jan 3, 2025

Techatrix commented Jan 3, 2025

RetroDev256 commented Jan 3, 2025

mlugg commented Jan 3, 2025 • edited Loading

rohlem commented Jan 3, 2025 • edited Loading

mlugg commented Jan 3, 2025

Techatrix commented Jan 4, 2025

andrewrk left a comment

Choose a reason for hiding this comment

andrewrk left a comment

Choose a reason for hiding this comment

Techatrix commented Jan 3, 2025 •

edited

Loading

mlugg commented Jan 3, 2025 •

edited

Loading

rohlem commented Jan 3, 2025 •

edited

Loading