[SPARK-55830][SQL] Fix JDBC predicate pushdown dropping driver properties

[yadavay-amzn]

What changes were proposed in this pull request?

When using spark.read.jdbc() with predicates, custom JDBC driver properties (like socketFactory, cloudSqlInstance) were silently dropped, causing connection failures. Without predicates, the same properties worked fine.

Root Cause

In JDBCOptions, the constructor this(url, table, parameters: Map[String, String]) did:

this(CaseInsensitiveMap(parameters ++ Map(JDBC_URL -> url, JDBC_TABLE_NAME -> table)))

When parameters is a CaseInsensitiveMap (which it is in the predicate code path), parameters ++ Map(...) calls the inherited Map.++ (because the static type is Map[String, String]). The inherited Map.++ iterates this using CaseInsensitiveMap.iterator, which returns lowercased keys from keyLowerCasedMap. This creates a plain HashMap with lowercased keys, losing the original case.

JDBC drivers expect case-sensitive property names (e.g., socketFactory, not socketfactory), so asConnectionProperties returns properties with wrong-cased keys, and Properties.getProperty("socketFactory") returns null.

Fix

Wrap parameters in CaseInsensitiveMap first, then use CaseInsensitiveMap.++ which preserves original key case via the updated() method:

this(CaseInsensitiveMap(parameters) ++ Map(JDBC_URL -> url, JDBC_TABLE_NAME -> table))

Applied to both JDBCOptions and JdbcOptionsInWrite.

How was this patch tested?

Added a unit test in JDBCSuite that simulates the predicate code path: creates JDBCOptions from CaseInsensitiveMap ++ Properties.asScala and verifies asConnectionProperties preserves original key case.

Does this PR introduce any user-facing change?

Yes. Custom JDBC driver properties (e.g., socketFactory, cloudSqlInstance) now work correctly when using spark.read.jdbc() with predicates.

Was this patch authored or co-authored using generative AI tooling?

Yes