Skip to content

Commit 7d76e06

Browse files
authored
Reachability (#1372)
1 parent 2d37421 commit 7d76e06

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

42 files changed

+1812
-27
lines changed

.gitignore

+7
Original file line numberDiff line numberDiff line change
@@ -29,10 +29,17 @@ fossa.telemetry.json
2929
# Integration Tests
3030
integration-test/artifacts/
3131

32+
# macOs Finder
33+
.DS_Store
34+
3235
# One offs
3336
test/App/Fossa/VSI/DynLinked/testdata/hello_standard*
3437
test/App/Fossa/VSI/DynLinked/testdata/hello_setuid*
3538

3639
# Rust
3740

3841
target/
42+
43+
# Include targets in test
44+
!test/reachability/testdata/maven-default/target
45+
!test/reachability/testdata/maven-build-config/dist

Changelog.md

+4
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,9 @@
11
# FOSSA CLI Changelog
22

3+
## v3.9.2
4+
- Maven: Adds reachability analysis [#1372](https://github.com/fossas/fossa-cli/pull/1377)
5+
- Gradle: Adds reachability analysis [#1377](https://github.com/fossas/fossa-cli/pull/1377)
6+
37
## v3.9.1
48
- `--detect-dynamic`: Safely ignores scenarios in ldd output parsing where we run into not found error. ([#1376](https://github.com/fossas/fossa-cli/pull/1376))
59

docs/contributing/reachability.md

+134
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
# Reachability
2+
3+
### What is Reachability?
4+
Reachability Analysis is a security offering designed to enhance FOSSA's security analysis by providing context on vulnerable packages. It alleviates the constraints of traditional CVE assessments through the static analysis of application and dependency code, confirming the presence of vulnerable call paths.
5+
6+
### Limitations
7+
- Reachability currently supports all Maven and Gradle projects dynamically analyzed by fossa-cli.
8+
- The target jar of the project must exist, prior to the analysis. If the jar artifact is not present, or FOSSA CLI fails to
9+
associate this jar with project, FOSSA CLI will not perform reachability analysis.
10+
- Reachability requires that `java` is present in PATH, and `java` version must be greater than `1.8` (jdk8+).
11+
12+
For example,
13+
- if you are using maven, you should run `mvn package` to ensure jar artifact exists, prior to running `fossa analyze`
14+
- if you are using gradle, you should run `gradlew build` to ensure jar artifact exists, prior to running `fossa analyze`
15+
16+
### Maven Analysis
17+
18+
For Maven projects, FOSSA CLI performs an analysis to infer dependencies. If FOSSA CLI identifies a complete dependency graph, which may include both direct and transitive dependencies, it attempts to infer the built JAR file for reachability analysis. It looks for `./target/{artifact}-{version}.jar` from the POM directory. If the POM file provides `build.directory` or `build.finalName` attributes, they are used instead of the default target jar path. For this reason, perform `fossa analyze` after the project of interest is built (g.g. `mvn package`), and target artifact exists in the directory.
19+
20+
### Gradle Analysis
21+
22+
For Gradle projects, FOSSA CLI invokes `./gradlew -I jsonpaths.gradle jsonPaths`. Where [jsonpaths.gradle](./../../scripts/jarpaths.gradle) is gradle script, which uses `java` plugin, and `jar` task associated with gradle to infer path of the built jar file. If neither of those are present, FOSSA CLI won't be able to identify jar artifacts for analysis.
23+
24+
### How do I debug reachability from `fossa-cli`?
25+
26+
```bash
27+
fossa analyze --debug
28+
29+
cat fossa.debug.json | jq '.bundleReachabilityRaw'
30+
[
31+
{
32+
"callGraphAnalysis": {
33+
"value": [
34+
{
35+
"parsedJarContent": {
36+
"kind": "ContentRaw",
37+
"value": "some value"
38+
},
39+
"parsedJarPath": "/Users/dev/example/example-projects/maven/example-maven-project/target/example-artifact-1.1.jar"
40+
}
41+
],
42+
"kind": "JarAnalysis"
43+
},
44+
"srcUnitDependencies": [
45+
"mvn+com.fasterxml.jackson.core:jackson-databind$2.13.0",
46+
"mvn+joda-time:joda-time$2.10.14",
47+
"mvn+com.fasterxml.jackson.core:jackson-annotations$2.13.0",
48+
"mvn+com.fasterxml.jackson.core:jackson-core$2.13.0"
49+
],
50+
"srcUnitManifest": "/Users/dev/example/example-projects/maven/example-maven-project/",
51+
"srcUnitName": "/Users/dev/example/example-projects/maven/example-maven-project/",
52+
"srcUnitOriginPaths": [
53+
"pom.xml"
54+
],
55+
"srcUnitType": "maven"
56+
},
57+
{
58+
"callGraphAnalysis": {
59+
"tag": "NoCallGraphAnalysis"
60+
},
61+
"srcUnitDependencies": [],
62+
"srcUnitManifest": "manifest",
63+
"srcUnitName": "name",
64+
"srcUnitOriginPaths": [],
65+
"srcUnitType": "type"
66+
}
67+
]
68+
69+
70+
cat fossa.debug.json | jq '.bundleReachabilityEndpoint'
71+
{
72+
# content uploaded to endpoint
73+
}
74+
```
75+
76+
<!--
77+
## How do I debug reachability from endpoint?
78+
79+
```bash
80+
# get what we sent to endpoint
81+
cat fossa.debug.json | jq '.bundleReachabilityEndpoint' > rawReachabilityJob.json
82+
83+
# run job in explain mode
84+
yarn repl
85+
explainReachability('rawReachabilityJob.json')
86+
87+
# [1] I was provided 'rawReachabilityJob.json'
88+
# [2] I'm parsing file: 'rawReachabilityJob.json'
89+
# [3] I found [X] reachability units
90+
# [4] Working on [0] reachability unit
91+
# --
92+
# {
93+
# ....
94+
# }
95+
#
96+
```
97+
-->
98+
99+
## F.A.Q.
100+
101+
1. What data from my codebase is uploaded to endpoint?
102+
103+
We upload call graph of (caller, and callee) relationships, in which
104+
caller and callee are fully qualified symbol name. We do not upload source code.
105+
106+
Here is example of caller, callee relationship that is uploaded to endpoint.
107+
108+
```txt
109+
M:com.example.app.utils.ContextReader:<init>() (O)java.lang.Object:<init>()
110+
M:com.example.app.utils.ContextReader:parseWithCtx(java.net.URL) (O)java.io.File:<init>(java.lang.String)
111+
M:com.example.app.utils.ContextReader:parseWithCtx(java.net.URL) (S)com.google.common.io.Files:toString(java.io.File,java.nio.charset.Charset)
112+
M:com.example.app.utils.ContextReader:parseWithCtx(java.net.URL) (O)org.dom4j.jaxb.JAXBReader:<init>(java.lang.String)
113+
M:com.example.app.utils.ContextReader:parseWithCtx(java.net.URL) (M)org.dom4j.jaxb.JAXBReader:read(java.net.URL)
114+
M:com.example.app.App:<init>() (O)java.lang.Object:<init>()
115+
M:com.example.app.App:main(java.lang.String[]) (O)java.net.URI:<init>(java.lang.String)
116+
M:com.example.app.App:main(java.lang.String[]) (M)java.net.URI:toURL()
117+
M:com.example.app.App:main(java.lang.String[]) (S)com.example.app.App:parse(java.net.URL)
118+
M:com.example.app.App:main(java.lang.String[]) (M)java.io.PrintStream:println(java.lang.Object)
119+
M:com.example.app.App:main(java.lang.String[]) (S)com.example.app.utils.ContextReader:parseWithCtx(java.net.URL)
120+
M:com.example.app.App:main(java.lang.String[]) (M)java.io.PrintStream:println(java.lang.Object)
121+
M:com.example.app.App:parse(java.net.URL) (O)org.dom4j.io.SAXReader:<init>()
122+
M:com.example.app.App:parse(java.net.URL) (M)org.dom4j.io.SAXReader:read(java.net.URL)
123+
```
124+
125+
You can inspect the data by running:
126+
127+
```bash
128+
; fossa analyze --output --debug # --output to not communicate with endpoint
129+
; gunzip fossa.debug.json.gz # extract produced debug bundle
130+
131+
# content in .bundleReachabilityRaw is uploaded
132+
# to endpoint for reachability analysis.
133+
; cat fossa.debug.json | jq '.bundleReachabilityRaw'
134+
```

integration-test/Analysis/FixtureUtils.hs

+5-1
Original file line numberDiff line numberDiff line change
@@ -10,12 +10,14 @@ module Analysis.FixtureUtils (
1010
TestC,
1111
performDiscoveryAndAnalyses,
1212
getArtifact,
13+
testRunnerWithLogger,
14+
withResult,
1315
) where
1416

1517
import App.Fossa.Analyze.Types (AnalyzeProject (analyzeProject))
1618
import App.Fossa.Config.Analyze (ExperimentalAnalyzeConfig (ExperimentalAnalyzeConfig), GoDynamicTactic (GoModulesBasedTactic))
1719
import App.Types (OverrideDynamicAnalysisBinary)
18-
import Control.Carrier.Debug (ignoreDebug)
20+
import Control.Carrier.Debug (IgnoreDebugC, ignoreDebug)
1921
import Control.Carrier.Diagnostics (DiagnosticsC, runDiagnostics)
2022
import Control.Carrier.Finally (FinallyC, runFinally)
2123
import Control.Carrier.Lift (Lift, sendIO)
@@ -109,6 +111,7 @@ data FixtureArtifact = FixtureArtifact
109111
type TestC m =
110112
ExecIOC
111113
$ ReadFSIOC
114+
$ IgnoreDebugC
112115
$ DiagnosticsC
113116
$ LoggerC
114117
$ ReaderC OverrideDynamicAnalysisBinary
@@ -124,6 +127,7 @@ testRunnerWithLogger f env =
124127
f
125128
& runExecIOWithinEnv env
126129
& runReadFSIO
130+
& ignoreDebug
127131
& runDiagnostics
128132
& withDefaultLogger SevWarn
129133
& runReader (mempty :: OverrideDynamicAnalysisBinary)
+179
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,179 @@
1+
{-# LANGUAGE QuasiQuotes #-}
2+
{-# LANGUAGE TemplateHaskell #-}
3+
4+
module Reachability.UploadSpec (spec) where
5+
6+
import Analysis.FixtureUtils (FixtureEnvironment (..), TestC, testRunnerWithLogger, withResult)
7+
import App.Fossa.Analyze.Project (ProjectResult (..))
8+
import App.Fossa.Analyze.Types (
9+
AnalysisScanResult (..),
10+
DiscoveredProjectIdentifier (..),
11+
DiscoveredProjectScan (..),
12+
)
13+
import App.Fossa.Reachability.Jar (callGraphFromJar)
14+
import App.Fossa.Reachability.Types (
15+
CallGraphAnalysis (..),
16+
ContentRef (..),
17+
ParsedJar (..),
18+
SourceUnitReachability (..),
19+
)
20+
import App.Fossa.Reachability.Upload (
21+
analyzeForReachability,
22+
callGraphOf,
23+
)
24+
import Data.ByteString.Lazy qualified as LB
25+
import Data.Foldable (for_)
26+
import Data.String.Conversion (toText)
27+
import Data.Text (Text)
28+
import Data.Text.Encoding qualified as TL
29+
import Diag.Result (Result (..))
30+
import Graphing (empty)
31+
import Path (
32+
Abs,
33+
Dir,
34+
File,
35+
Path,
36+
Rel,
37+
mkRelDir,
38+
mkRelFile,
39+
(</>),
40+
)
41+
import Path.IO qualified as PIO
42+
import Test.Hspec (Spec, describe, it, runIO, shouldBe)
43+
import Text.RawString.QQ (r)
44+
import Types (
45+
DiscoveredProjectType (MavenProjectType),
46+
GraphBreadth (..),
47+
)
48+
49+
java8 :: FixtureEnvironment
50+
java8 = NixEnv ["jdk8"]
51+
52+
java11 :: FixtureEnvironment
53+
java11 = NixEnv ["jdk11"]
54+
55+
java17 :: FixtureEnvironment
56+
java17 = NixEnv ["jdk17"]
57+
58+
java21 :: FixtureEnvironment
59+
java21 = NixEnv ["jdk21"]
60+
61+
run :: FixtureEnvironment -> TestC IO a -> IO (Result a)
62+
run env act = testRunnerWithLogger act env
63+
64+
spec :: Spec
65+
spec = describe "Reachability" $ do
66+
describe "callGraphFromJar" $ do
67+
jarFile <- runIO sampleJarFile
68+
for_
69+
[ java8
70+
, java11
71+
, java17
72+
, java21
73+
]
74+
$ \env -> do
75+
it ("should compute in java: " <> show env) $ do
76+
let expected = Just (sampleJarParsed jarFile)
77+
res <- run env $ callGraphFromJar jarFile
78+
79+
withResult res $ \_ res' -> res' `shouldBe` expected
80+
81+
describe "callGraphOf" $ do
82+
projDir <- (</> sampleMavenProjectDir) <$> runIO PIO.getCurrentDir
83+
jarFile <- (</> sampleMavenProjectJar) <$> runIO PIO.getCurrentDir
84+
85+
it "should retrieve call graph" $ do
86+
let expected = Just (mavenCompleteScanUnit projDir jarFile)
87+
resp <- run java8 $ callGraphOf (mavenCompleteScan projDir)
88+
withResult resp $ \_ res -> res `shouldBe` expected
89+
90+
describe "analyzeForReachability" $ do
91+
projDir <- (</> sampleMavenProjectDir) <$> runIO PIO.getCurrentDir
92+
jarFile <- (</> sampleMavenProjectJar) <$> runIO PIO.getCurrentDir
93+
94+
it "should return analyzed reachability unit" $ do
95+
let expected = [mavenCompleteScanUnit projDir jarFile]
96+
let analysisResult =
97+
AnalysisScanResult
98+
[mavenCompleteScan projDir]
99+
successNothing
100+
successNothing
101+
successNothing
102+
successNothing
103+
successNothing
104+
105+
analyzed <- run java8 $ analyzeForReachability analysisResult
106+
withResult analyzed $ \_ analyzed' -> analyzed' `shouldBe` expected
107+
108+
sampleMavenProjectDir :: Path Rel Dir
109+
sampleMavenProjectDir = $(mkRelDir "test/Reachability/testdata/maven-default/")
110+
111+
sampleMavenProjectJar :: Path Rel File
112+
sampleMavenProjectJar = $(mkRelFile "test/Reachability/testdata/maven-default/target/project-1.0.0.jar")
113+
114+
mavenCompleteScan :: Path Abs Dir -> DiscoveredProjectScan
115+
mavenCompleteScan dir = mkDiscoveredProjectScan MavenProjectType dir Complete
116+
117+
mavenCompleteScanUnit :: Path Abs Dir -> Path Abs File -> SourceUnitReachability
118+
mavenCompleteScanUnit projDir jarFile =
119+
mkReachabilityUnit
120+
projDir
121+
[ ParsedJar
122+
jarFile
123+
(ContentRaw sampleJarParsedContent')
124+
]
125+
126+
mkDiscoveredProjectScan :: DiscoveredProjectType -> Path Abs Dir -> GraphBreadth -> DiscoveredProjectScan
127+
mkDiscoveredProjectScan projectType dir breadth =
128+
Scanned
129+
(DiscoveredProjectIdentifier dir projectType)
130+
( Success
131+
[]
132+
(ProjectResult projectType dir empty breadth mempty)
133+
)
134+
135+
sampleJarFile :: IO (Path Abs File)
136+
sampleJarFile = do
137+
cwd <- PIO.getCurrentDir
138+
pure (cwd </> $(mkRelFile "test/Reachability/testdata/sample.jar"))
139+
140+
successNothing :: Result (Maybe a)
141+
successNothing = Success [] Nothing
142+
143+
mkReachabilityUnit :: Path Abs Dir -> [ParsedJar] -> SourceUnitReachability
144+
mkReachabilityUnit dir jars =
145+
SourceUnitReachability
146+
"maven"
147+
(toText dir)
148+
(toText dir)
149+
[]
150+
[]
151+
(JarAnalysis jars)
152+
153+
sampleJarParsed :: Path Abs File -> ParsedJar
154+
sampleJarParsed path =
155+
ParsedJar
156+
{ parsedJarPath = path
157+
, parsedJarContent = ContentRaw sampleJarParsedContent'
158+
}
159+
160+
sampleJarParsedContent :: Text
161+
sampleJarParsedContent =
162+
[r|C:vuln.project.sample.App java.lang.Object
163+
C:vuln.project.sample.App java.net.URI
164+
C:vuln.project.sample.App java.lang.System
165+
C:vuln.project.sample.App vuln.project.sample.App
166+
C:vuln.project.sample.App java.io.PrintStream
167+
C:vuln.project.sample.App org.dom4j.io.SAXReader
168+
C:vuln.project.sample.App java.lang.Exception
169+
C:vuln.project.sample.App org.dom4j.DocumentException
170+
M:vuln.project.sample.App:<init>() (O)java.lang.Object:<init>()
171+
M:vuln.project.sample.App:main(java.lang.String[]) (O)java.net.URI:<init>(java.lang.String)
172+
M:vuln.project.sample.App:main(java.lang.String[]) (M)java.net.URI:toURL()
173+
M:vuln.project.sample.App:main(java.lang.String[]) (S)vuln.project.sample.App:parse(java.net.URL)
174+
M:vuln.project.sample.App:main(java.lang.String[]) (M)java.io.PrintStream:println(java.lang.Object)
175+
M:vuln.project.sample.App:parse(java.net.URL) (O)org.dom4j.io.SAXReader:<init>()
176+
M:vuln.project.sample.App:parse(java.net.URL) (M)org.dom4j.io.SAXReader:read(java.net.URL)|]
177+
178+
sampleJarParsedContent' :: LB.ByteString
179+
sampleJarParsedContent' = LB.fromStrict . TL.encodeUtf8 $ sampleJarParsedContent

scripts/jar-callgraph-1.0.0.jar

1.34 MB
Binary file not shown.

0 commit comments

Comments
 (0)