Skip to content

Commit e1c630a

Browse files
ggershinskydongjoon-hyun
authored andcommitted
[SPARK-42114][SQL][TESTS] Add uniform parquet encryption test case
### What changes were proposed in this pull request? Unitest of the new parquet encryption mode, available since parquet 1.12.3. This mode is triggered by setting a hadoop parameter (/df write option) `parquet.encryption.uniform.key` : master key ID for uniform encryption of all columns and footer. ### Why are the changes needed? To verify end to end Spark SQL encryption and decryption of dataframes in uniform mode. ### Does this PR introduce _any_ user-facing change? No user-facing changes. ### How was this patch tested? This patch is the unitest Closes apache#39665 from ggershinsky/uniform-unitest1. Authored-by: Gidon Gershinsky <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent ec424c5 commit e1c630a

File tree

1 file changed

+26
-0
lines changed

1 file changed

+26
-0
lines changed

sql/hive/src/test/scala/org/apache/spark/sql/hive/ParquetEncryptionSuite.scala

+26
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,32 @@ class ParquetEncryptionSuite extends QueryTest with TestHiveSingleton {
9191
}
9292
}
9393

94+
test("SPARK-42114: Test of uniform parquet encryption") {
95+
withTempDir { dir =>
96+
withSQLConf(
97+
"parquet.crypto.factory.class" ->
98+
"org.apache.parquet.crypto.keytools.PropertiesDrivenCryptoFactory",
99+
"parquet.encryption.kms.client.class" ->
100+
"org.apache.parquet.crypto.keytools.mocks.InMemoryKMS",
101+
"parquet.encryption.key.list" ->
102+
s"key1: ${key1}") {
103+
104+
val inputDF = Seq((1, 22, 333)).toDF("a", "b", "c")
105+
val parquetDir = new File(dir, "parquet").getCanonicalPath
106+
inputDF.write
107+
.option("parquet.encryption.uniform.key", "key1")
108+
.parquet(parquetDir)
109+
110+
verifyParquetEncrypted(parquetDir)
111+
112+
val parquetDF = spark.read.parquet(parquetDir)
113+
assert(parquetDF.inputFiles.nonEmpty)
114+
val readDataset = parquetDF.select("a", "b", "c")
115+
checkAnswer(readDataset, inputDF)
116+
}
117+
}
118+
}
119+
94120
/**
95121
* Verify that the directory contains an encrypted parquet in
96122
* encrypted footer mode by means of checking for all the parquet part files

0 commit comments

Comments
 (0)