[GlobalISel] fdiv to fmul transform #144305

davemgreen · 2025-06-16T06:52:16Z

This is a port of the SDAG DAGCombiner::combineRepeatedFPDivisors combine that looks for multiple fdiv operations with the same divisor and converts them to a single reciprocal fdiv and multiple fmuls. It is currently a fairly faithful port, with some additions to make sure that the newly created fdiv dominates all new uses. Compared to the SDAG version it also drops some logic about splat uses which assumes no vector fdivs and some logic about x/sqrt(x) which does not yet apply to GISel.

llvmbot · 2025-06-16T06:52:48Z

@llvm/pr-subscribers-backend-aarch64

@llvm/pr-subscribers-llvm-globalisel

Author: David Green (davemgreen)

Changes

This is a port of the SDAG DAGCombiner::combineRepeatedFPDivisors combine that looks like multiple fdiv operations with the same divisor and converts them to a single reciprocal fdiv and multiple fmuls. It is currently a fairly faithful port, with some additions to make sure that the newly created fdiv dominates all new uses. Compared to the SDAG version it also drops some logic about splat uses which assumes no vector fdivs and some logic about x/sqrt(x) which does not yet apply to GISel.

Full diff: https://github.com/llvm/llvm-project/pull/144305.diff

4 Files Affected:

(modified) llvm/include/llvm/CodeGen/GlobalISel/CombinerHelper.h (+4)
(modified) llvm/include/llvm/Target/GlobalISel/Combine.td (+10-1)
(modified) llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp (+73)
(modified) llvm/test/CodeGen/AArch64/fdiv-combine.ll (+42-83)

diff --git a/llvm/include/llvm/CodeGen/GlobalISel/CombinerHelper.h b/llvm/include/llvm/CodeGen/GlobalISel/CombinerHelper.h
index c15263e0b06f8..f5a8ed9edac16 100644
--- a/llvm/include/llvm/CodeGen/GlobalISel/CombinerHelper.h
+++ b/llvm/include/llvm/CodeGen/GlobalISel/CombinerHelper.h
@@ -809,6 +809,10 @@ class CombinerHelper {
 
   bool matchCombineFMinMaxNaN(MachineInstr &MI, unsigned &Info) const;
 
+  bool matchRepeatedFPDivisor(MachineInstr &MI,
+                              SmallVector<MachineInstr *> &MatchInfo) const;
+  void applyRepeatedFPDivisor(SmallVector<MachineInstr *> &MatchInfo) const;
+
   /// Transform G_ADD(x, G_SUB(y, x)) to y.
   /// Transform G_ADD(G_SUB(y, x), x) to y.
   bool matchAddSubSameReg(MachineInstr &MI, Register &Src) const;
diff --git a/llvm/include/llvm/Target/GlobalISel/Combine.td b/llvm/include/llvm/Target/GlobalISel/Combine.td
index efd88524a159e..1447b293146db 100644
--- a/llvm/include/llvm/Target/GlobalISel/Combine.td
+++ b/llvm/include/llvm/Target/GlobalISel/Combine.td
@@ -211,6 +211,7 @@ def constantfp_matchinfo : GIDefMatchData<"ConstantFP*">;
 def build_fn_matchinfo :
 GIDefMatchData<"std::function<void(MachineIRBuilder &)>">;
 def unsigned_matchinfo: GIDefMatchData<"unsigned">;
+def mi_vector_matchinfo : GIDefMatchData<"SmallVector<MachineInstr *>">;
 
 def copy_prop : GICombineRule<
   (defs root:$d),
@@ -1327,6 +1328,14 @@ def combine_minmax_nan: GICombineRule<
          [{ return Helper.matchCombineFMinMaxNaN(*${root}, ${info}); }]),
   (apply [{ Helper.replaceSingleDefInstWithOperand(*${root}, ${info}); }])>;
 
+// Combine multiple FDIVs with the same divisor into multiple FMULs by the
+// reciprocal.
+def fdiv_repeated_divison: GICombineRule<
+  (defs root:$root, mi_vector_matchinfo:$matchinfo),
+  (match (G_FDIV $dst, $src1, $src2):$root,
+         [{ return Helper.matchRepeatedFPDivisor(*${root}, ${matchinfo}); }]),
+  (apply [{ Helper.applyRepeatedFPDivisor(${matchinfo}); }])>;
+
 // Transform (add x, (sub y, x)) -> y
 // Transform (add (sub y, x), x) -> y
 def add_sub_reg_frags : GICombinePatFrag<
@@ -2051,7 +2060,7 @@ def all_combines : GICombineGroup<[integer_reassoc_combines, trivial_combines,
     constant_fold_cast_op, fabs_fneg_fold,
     intdiv_combines, mulh_combines, redundant_neg_operands,
     and_or_disjoint_mask, fma_combines, fold_binop_into_select,
-    sub_add_reg, select_to_minmax,
+    sub_add_reg, select_to_minmax, fdiv_repeated_divison,
     fsub_to_fneg, commute_constant_to_rhs, match_ands, match_ors,
     simplify_neg_minmax, combine_concat_vector,
     sext_trunc, zext_trunc, prefer_sign_combines, shuffle_combines,
diff --git a/llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp b/llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp
index b1e851183de0d..556c2665ce670 100644
--- a/llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp
+++ b/llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp
@@ -6389,6 +6389,79 @@ bool CombinerHelper::matchCombineFMinMaxNaN(MachineInstr &MI,
   return MatchNaN(1) || MatchNaN(2);
 }
 
+// Combine multiple FDIVs with the same divisor into multiple FMULs by the
+// reciprocal.
+// E.g., (a / D; b / D;) -> (recip = 1.0 / D; a * recip; b * recip)
+bool CombinerHelper::matchRepeatedFPDivisor(
+    MachineInstr &MI, SmallVector<MachineInstr *> &MatchInfo) const {
+  assert(MI.getOpcode() == TargetOpcode::G_FDIV);
+  auto *MF = MI.getMF();
+  const TargetOptions &Options = MF->getTarget().Options;
+
+  Register Dst = MI.getOperand(0).getReg();
+  Register X = MI.getOperand(1).getReg();
+  Register Y = MI.getOperand(2).getReg();
+  LLT Type = MRI.getType(Dst);
+
+  bool UnsafeMath = Options.UnsafeFPMath;
+  if (!UnsafeMath && !MI.getFlag(MachineInstr::MIFlag::FmArcp))
+    return false;
+
+  // Skip if current node is a reciprocal/fneg-reciprocal.
+  auto N0CFP = isConstantOrConstantSplatVectorFP(*MRI.getVRegDef(X), MRI);
+  if (N0CFP && (N0CFP->isExactlyValue(1.0) || N0CFP->isExactlyValue(-1.0)))
+    return false;
+
+  // Exit early if the target does not want this transform or if there can't
+  // possibly be enough uses of the divisor to make the transform worthwhile.
+  unsigned MinUses = getTargetLowering().combineRepeatedFPDivisors();
+
+  if (!MinUses)
+    return false;
+
+  // Find all FDIV users of the same divisor.
+  // Use a set because duplicates may be present in the user list.
+  // For the moment we limit all instructions to a single BB and use the first
+  // Instr as the dominating position.
+  MatchInfo.push_back(&MI);
+  for (auto &U : MRI.use_nodbg_instructions(Y)) {
+    if (&U == &MI || U.getParent() != MI.getParent())
+      continue;
+    if (U.getOpcode() == TargetOpcode::G_FDIV && U.getOperand(2).getReg() == Y) {
+      // This division is eligible for optimization only if global unsafe math
+      // is enabled or if this division allows reciprocal formation.
+      if (UnsafeMath || U.getFlag(MachineInstr::MIFlag::FmArcp)) {
+        MatchInfo.push_back(&U);
+        // Is there a better way to handle this?
+        if (dominates(U, *MatchInfo[0]))
+          std::swap(MatchInfo[0], MatchInfo.back());
+      }
+    }
+  }
+
+  // Now that we have the actual number of divisor uses, make sure it meets
+  // the minimum threshold specified by the target.
+  return MatchInfo.size() >= MinUses;
+}
+
+void CombinerHelper::applyRepeatedFPDivisor(
+    SmallVector<MachineInstr *> &MatchInfo) const {
+  // Generate the new div at the position of the first instruction, that we have
+  // ensured will dominate all other instructions.
+  Builder.setInsertPt(*MatchInfo[0]->getParent(), MatchInfo[0]);
+  LLT Ty = MRI.getType(MatchInfo[0]->getOperand(0).getReg());
+  auto Div = Builder.buildFDiv(Ty, Builder.buildFConstant(Ty, 1.0),
+                               MatchInfo[0]->getOperand(2).getReg());
+
+  // Replace all found div's with fmul instructions.
+  for (MachineInstr *MI : MatchInfo) {
+    Builder.setInsertPt(*MI->getParent(), MI);
+    Builder.buildFMul(MI->getOperand(0).getReg(), MI->getOperand(1).getReg(),
+                      Div->getOperand(0).getReg());
+    MI->eraseFromParent();
+  }
+}
+
 bool CombinerHelper::matchAddSubSameReg(MachineInstr &MI, Register &Src) const {
   assert(MI.getOpcode() == TargetOpcode::G_ADD && "Expected a G_ADD");
   Register LHS = MI.getOperand(1).getReg();
diff --git a/llvm/test/CodeGen/AArch64/fdiv-combine.ll b/llvm/test/CodeGen/AArch64/fdiv-combine.ll
index d8f7f0a306684..e566e54a2964a 100644
--- a/llvm/test/CodeGen/AArch64/fdiv-combine.ll
+++ b/llvm/test/CodeGen/AArch64/fdiv-combine.ll
@@ -12,22 +12,14 @@
 ;                =>
 ;   recip = 1.0 / D; a * recip; b * recip; c * recip;
 define void @three_fdiv_float(float %D, float %a, float %b, float %c) #0 {
-; CHECK-SD-LABEL: three_fdiv_float:
-; CHECK-SD:       // %bb.0:
-; CHECK-SD-NEXT:    fmov s4, #1.00000000
-; CHECK-SD-NEXT:    fdiv s4, s4, s0
-; CHECK-SD-NEXT:    fmul s0, s1, s4
-; CHECK-SD-NEXT:    fmul s1, s2, s4
-; CHECK-SD-NEXT:    fmul s2, s3, s4
-; CHECK-SD-NEXT:    b foo_3f
-;
-; CHECK-GI-LABEL: three_fdiv_float:
-; CHECK-GI:       // %bb.0:
-; CHECK-GI-NEXT:    fdiv s4, s1, s0
-; CHECK-GI-NEXT:    fdiv s1, s2, s0
-; CHECK-GI-NEXT:    fdiv s2, s3, s0
-; CHECK-GI-NEXT:    fmov s0, s4
-; CHECK-GI-NEXT:    b foo_3f
+; CHECK-LABEL: three_fdiv_float:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    fmov s4, #1.00000000
+; CHECK-NEXT:    fdiv s4, s4, s0
+; CHECK-NEXT:    fmul s0, s1, s4
+; CHECK-NEXT:    fmul s1, s2, s4
+; CHECK-NEXT:    fmul s2, s3, s4
+; CHECK-NEXT:    b foo_3f
   %div = fdiv float %a, %D
   %div1 = fdiv float %b, %D
   %div2 = fdiv float %c, %D
@@ -36,22 +28,14 @@ define void @three_fdiv_float(float %D, float %a, float %b, float %c) #0 {
 }
 
 define void @three_fdiv_double(double %D, double %a, double %b, double %c) #0 {
-; CHECK-SD-LABEL: three_fdiv_double:
-; CHECK-SD:       // %bb.0:
-; CHECK-SD-NEXT:    fmov d4, #1.00000000
-; CHECK-SD-NEXT:    fdiv d4, d4, d0
-; CHECK-SD-NEXT:    fmul d0, d1, d4
-; CHECK-SD-NEXT:    fmul d1, d2, d4
-; CHECK-SD-NEXT:    fmul d2, d3, d4
-; CHECK-SD-NEXT:    b foo_3d
-;
-; CHECK-GI-LABEL: three_fdiv_double:
-; CHECK-GI:       // %bb.0:
-; CHECK-GI-NEXT:    fdiv d4, d1, d0
-; CHECK-GI-NEXT:    fdiv d1, d2, d0
-; CHECK-GI-NEXT:    fdiv d2, d3, d0
-; CHECK-GI-NEXT:    fmov d0, d4
-; CHECK-GI-NEXT:    b foo_3d
+; CHECK-LABEL: three_fdiv_double:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    fmov d4, #1.00000000
+; CHECK-NEXT:    fdiv d4, d4, d0
+; CHECK-NEXT:    fmul d0, d1, d4
+; CHECK-NEXT:    fmul d1, d2, d4
+; CHECK-NEXT:    fmul d2, d3, d4
+; CHECK-NEXT:    b foo_3d
   %div = fdiv double %a, %D
   %div1 = fdiv double %b, %D
   %div2 = fdiv double %c, %D
@@ -60,22 +44,14 @@ define void @three_fdiv_double(double %D, double %a, double %b, double %c) #0 {
 }
 
 define void @three_fdiv_4xfloat(<4 x float> %D, <4 x float> %a, <4 x float> %b, <4 x float> %c) #0 {
-; CHECK-SD-LABEL: three_fdiv_4xfloat:
-; CHECK-SD:       // %bb.0:
-; CHECK-SD-NEXT:    fmov v4.4s, #1.00000000
-; CHECK-SD-NEXT:    fdiv v4.4s, v4.4s, v0.4s
-; CHECK-SD-NEXT:    fmul v0.4s, v1.4s, v4.4s
-; CHECK-SD-NEXT:    fmul v1.4s, v2.4s, v4.4s
-; CHECK-SD-NEXT:    fmul v2.4s, v3.4s, v4.4s
-; CHECK-SD-NEXT:    b foo_3_4xf
-;
-; CHECK-GI-LABEL: three_fdiv_4xfloat:
-; CHECK-GI:       // %bb.0:
-; CHECK-GI-NEXT:    fdiv v4.4s, v1.4s, v0.4s
-; CHECK-GI-NEXT:    fdiv v1.4s, v2.4s, v0.4s
-; CHECK-GI-NEXT:    fdiv v2.4s, v3.4s, v0.4s
-; CHECK-GI-NEXT:    mov v0.16b, v4.16b
-; CHECK-GI-NEXT:    b foo_3_4xf
+; CHECK-LABEL: three_fdiv_4xfloat:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    fmov v4.4s, #1.00000000
+; CHECK-NEXT:    fdiv v4.4s, v4.4s, v0.4s
+; CHECK-NEXT:    fmul v0.4s, v1.4s, v4.4s
+; CHECK-NEXT:    fmul v1.4s, v2.4s, v4.4s
+; CHECK-NEXT:    fmul v2.4s, v3.4s, v4.4s
+; CHECK-NEXT:    b foo_3_4xf
   %div = fdiv <4 x float> %a, %D
   %div1 = fdiv <4 x float> %b, %D
   %div2 = fdiv <4 x float> %c, %D
@@ -84,22 +60,14 @@ define void @three_fdiv_4xfloat(<4 x float> %D, <4 x float> %a, <4 x float> %b,
 }
 
 define void @three_fdiv_2xdouble(<2 x double> %D, <2 x double> %a, <2 x double> %b, <2 x double> %c) #0 {
-; CHECK-SD-LABEL: three_fdiv_2xdouble:
-; CHECK-SD:       // %bb.0:
-; CHECK-SD-NEXT:    fmov v4.2d, #1.00000000
-; CHECK-SD-NEXT:    fdiv v4.2d, v4.2d, v0.2d
-; CHECK-SD-NEXT:    fmul v0.2d, v1.2d, v4.2d
-; CHECK-SD-NEXT:    fmul v1.2d, v2.2d, v4.2d
-; CHECK-SD-NEXT:    fmul v2.2d, v3.2d, v4.2d
-; CHECK-SD-NEXT:    b foo_3_2xd
-;
-; CHECK-GI-LABEL: three_fdiv_2xdouble:
-; CHECK-GI:       // %bb.0:
-; CHECK-GI-NEXT:    fdiv v4.2d, v1.2d, v0.2d
-; CHECK-GI-NEXT:    fdiv v1.2d, v2.2d, v0.2d
-; CHECK-GI-NEXT:    fdiv v2.2d, v3.2d, v0.2d
-; CHECK-GI-NEXT:    mov v0.16b, v4.16b
-; CHECK-GI-NEXT:    b foo_3_2xd
+; CHECK-LABEL: three_fdiv_2xdouble:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    fmov v4.2d, #1.00000000
+; CHECK-NEXT:    fdiv v4.2d, v4.2d, v0.2d
+; CHECK-NEXT:    fmul v0.2d, v1.2d, v4.2d
+; CHECK-NEXT:    fmul v1.2d, v2.2d, v4.2d
+; CHECK-NEXT:    fmul v2.2d, v3.2d, v4.2d
+; CHECK-NEXT:    b foo_3_2xd
   %div = fdiv <2 x double> %a, %D
   %div1 = fdiv <2 x double> %b, %D
   %div2 = fdiv <2 x double> %c, %D
@@ -136,25 +104,16 @@ define void @two_fdiv_double(double %D, double %a, double %b) #0 {
 }
 
 define void @splat_three_fdiv_4xfloat(float %D, <4 x float> %a, <4 x float> %b, <4 x float> %c) #0 {
-; CHECK-SD-LABEL: splat_three_fdiv_4xfloat:
-; CHECK-SD:       // %bb.0:
-; CHECK-SD-NEXT:    // kill: def $s0 killed $s0 def $q0
-; CHECK-SD-NEXT:    fmov v4.4s, #1.00000000
-; CHECK-SD-NEXT:    dup v0.4s, v0.s[0]
-; CHECK-SD-NEXT:    fdiv v4.4s, v4.4s, v0.4s
-; CHECK-SD-NEXT:    fmul v0.4s, v1.4s, v4.4s
-; CHECK-SD-NEXT:    fmul v1.4s, v2.4s, v4.4s
-; CHECK-SD-NEXT:    fmul v2.4s, v3.4s, v4.4s
-; CHECK-SD-NEXT:    b foo_3_4xf
-;
-; CHECK-GI-LABEL: splat_three_fdiv_4xfloat:
-; CHECK-GI:       // %bb.0:
-; CHECK-GI-NEXT:    // kill: def $s0 killed $s0 def $q0
-; CHECK-GI-NEXT:    dup v4.4s, v0.s[0]
-; CHECK-GI-NEXT:    fdiv v0.4s, v1.4s, v4.4s
-; CHECK-GI-NEXT:    fdiv v1.4s, v2.4s, v4.4s
-; CHECK-GI-NEXT:    fdiv v2.4s, v3.4s, v4.4s
-; CHECK-GI-NEXT:    b foo_3_4xf
+; CHECK-LABEL: splat_three_fdiv_4xfloat:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    // kill: def $s0 killed $s0 def $q0
+; CHECK-NEXT:    fmov v4.4s, #1.00000000
+; CHECK-NEXT:    dup v0.4s, v0.s[0]
+; CHECK-NEXT:    fdiv v4.4s, v4.4s, v0.4s
+; CHECK-NEXT:    fmul v0.4s, v1.4s, v4.4s
+; CHECK-NEXT:    fmul v1.4s, v2.4s, v4.4s
+; CHECK-NEXT:    fmul v2.4s, v3.4s, v4.4s
+; CHECK-NEXT:    b foo_3_4xf
   %D.ins = insertelement <4 x float> poison, float %D, i64 0
   %splat = shufflevector <4 x float> %D.ins, <4 x float> poison, <4 x i32> zeroinitializer
   %div = fdiv <4 x float> %a, %splat

This is a port of the SDAG DAGCombiner::combineRepeatedFPDivisors combine that looks like multiple fdiv operations with the same divisor and converts them to a single reciprocal fdiv and multiple fmuls. It is currently a fairly faithful port, with some additions to make sure that the newly created fdiv dominates all new uses. Compared to the SDAG version it also drops some logic about splat uses which assumes no vector fdivs and some logic about x/sqrt(x) which does not yet apply to GISel.

jayfoad · 2025-06-16T09:54:39Z

This is a port of the SDAG DAGCombiner::combineRepeatedFPDivisors combine that looks like multiple fdiv operations

Is "looks like" a typo?

jayfoad · 2025-06-16T09:59:09Z

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

+  // Find all FDIV users of the same divisor. Use a set because duplicates may
+  // be present in the user list. For the moment we limit all instructions to a


Might be neater to iterate over use_nodbg_operands and check that the operand is the second argument of an FDIV. Then you won't need to handle duplicates.

Actually... what is the comment talking about? You're not using a set!

Oops sorry. Too much copy and pasting of comments. It doesn't seem to go wrong with the new test but I added a check for the first operand just in case.

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

jayfoad · 2025-06-16T10:00:30Z

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

+        if (dominates(U, *MatchInfo[0]))
+          std::swap(MatchInfo[0], MatchInfo.back());


I don't think you're handling the case where neither FDIV dominates the other. In that case, where would you insert the reciprocal instruction?

At the moment we need to check that they are in the same BB to prevent that case, yeah. U.getParent() != MI.getParent().
I wasn't sure how to handle this more generally. This is already technically O(n^2) as the dominates calls are currently O(n).

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

davemgreen requested review from aemerson, arsenm, c-rhodes and jyli0116 June 16, 2025 06:52

llvmbot added backend:AArch64 llvm:globalisel labels Jun 16, 2025

davemgreen force-pushed the gh-gi-fdivcombo branch from f65e3bb to 3a0b197 Compare June 16, 2025 06:52

davemgreen force-pushed the gh-gi-fdivcombo branch from 3a0b197 to f633662 Compare June 16, 2025 07:06

davemgreen requested a review from jayfoad June 16, 2025 09:57

jayfoad reviewed Jun 16, 2025

View reviewed changes

Fixups and an extra test case.

0d0182a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[GlobalISel] fdiv to fmul transform #144305

[GlobalISel] fdiv to fmul transform #144305

davemgreen commented Jun 16, 2025 •

edited

Loading

Uh oh!

llvmbot commented Jun 16, 2025 •

edited

Loading

Uh oh!

jayfoad commented Jun 16, 2025

Uh oh!

jayfoad Jun 16, 2025

Uh oh!

davemgreen Jun 17, 2025

Uh oh!

Uh oh!

jayfoad Jun 16, 2025

Uh oh!

davemgreen Jun 17, 2025

Uh oh!

Uh oh!

Uh oh!

		// Find all FDIV users of the same divisor. Use a set because duplicates may
		// be present in the user list. For the moment we limit all instructions to a

		if (dominates(U, *MatchInfo[0]))
		std::swap(MatchInfo[0], MatchInfo.back());

[GlobalISel] fdiv to fmul transform #144305

Are you sure you want to change the base?

[GlobalISel] fdiv to fmul transform #144305

Conversation

davemgreen commented Jun 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

llvmbot commented Jun 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jayfoad commented Jun 16, 2025

Uh oh!

jayfoad Jun 16, 2025

Choose a reason for hiding this comment

Uh oh!

davemgreen Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jayfoad Jun 16, 2025

Choose a reason for hiding this comment

Uh oh!

davemgreen Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

davemgreen commented Jun 16, 2025 •

edited

Loading

llvmbot commented Jun 16, 2025 •

edited

Loading