Consistent JSON forward/backward compatibility (2.0 foundation)

CONTRIBUTING.md

@@ -75,6 +75,83 @@ git checkout -b feature/your-feature-name
    allow the reviewer to focus on incremental changes instead of having to restart the 
    review process.
 
+## Evolving wire-serialized records
+
+Records in `McpSchema` are serialized directly to the MCP JSON wire format. Follow these rules whenever you add a field to an existing record to keep the protocol forward- and backward-compatible.
+
+### Rules
+
+1. **Add new components only at the end** of the record's component list. Never reorder or rename existing components.
+2. **Annotate every component** with `@JsonProperty("fieldName")` even when the Java name already matches. This survives local renames via refactoring tools.
+3. **Use boxed types** (`Boolean`, `Integer`, `Long`, `Double`) so the field can be absent on the wire without a special sentinel.
+4. **Default to `null`**, not an empty collection or neutral value, so the `@JsonInclude(NON_NULL)` rule omits the field for clients that don't know about it yet.
+5. **Keep existing constructors as source-compatible overloads** that delegate to the new canonical constructor and pass `null` for the new component. Do not remove them in the same release that adds the field.
+6. **Do not put `@JsonCreator` on the canonical constructor** unless strictly necessary. Jackson auto-detects record canonical constructors; adding `@JsonCreator` pins deserialization to that exact parameter order forever.
+7. **Do not convert `null` to a default value in the canonical constructor.** Null carries "absent" semantics and must be preserved through the serialization round-trip.
+8. **Add three tests per new field** (put them in the relevant test class in `mcp-test`):
+   - Deserialize JSON *without* the field → succeeds, field is `null`.
+   - Serialize an instance with the field unset (`null`) → the key is absent from output.
+   - Deserialize JSON with an extra *unknown* field → succeeds.
+9. **An inner `Builder` subclass can be used.** This improves the developer experience since frequently not all fields are required. 
+
+### Example
+
+Suppose `ToolAnnotations` gains an optional `audience` field:
+
+```java
+// Before
+@JsonInclude(JsonInclude.Include.NON_NULL)
+@JsonIgnoreProperties(ignoreUnknown = true)
+public record ToolAnnotations(
+    @JsonProperty("title") String title,
+    @JsonProperty("readOnlyHint") Boolean readOnlyHint,
+    @JsonProperty("destructiveHint") Boolean destructiveHint,
+    @JsonProperty("idempotentHint") Boolean idempotentHint,
+    @JsonProperty("openWorldHint") Boolean openWorldHint) { ... }
+
+// After — new component appended at the end
+@JsonInclude(JsonInclude.Include.NON_NULL)
+@JsonIgnoreProperties(ignoreUnknown = true)
+public record ToolAnnotations(
+    @JsonProperty("title") String title,
+    @JsonProperty("readOnlyHint") Boolean readOnlyHint,
+    @JsonProperty("destructiveHint") Boolean destructiveHint,
+    @JsonProperty("idempotentHint") Boolean idempotentHint,
+    @JsonProperty("openWorldHint") Boolean openWorldHint,
+    @JsonProperty("audience") List<String> audience) {   // new — added at end
+
+    // Keep the old constructor so existing callers still compile
+    public ToolAnnotations(String title, Boolean readOnlyHint,
+            Boolean destructiveHint, Boolean idempotentHint, Boolean openWorldHint) {
+        this(title, readOnlyHint, destructiveHint, idempotentHint, openWorldHint, null);
+    }
+}
+```
+
+Tests to add:
+
+```java
+@Test
+void toolAnnotationsDeserializesWithoutAudience() throws IOException {
+    ToolAnnotations a = mapper.readValue("""
+            {"title":"My tool","readOnlyHint":true}""", ToolAnnotations.class);
+    assertThat(a.audience()).isNull();
+}
+
+@Test
+void toolAnnotationsOmitsNullAudience() throws IOException {
+    String json = mapper.writeValueAsString(new ToolAnnotations("t", null, null, null, null));
+    assertThat(json).doesNotContain("audience");
+}
+
+@Test
+void toolAnnotationsToleratesUnknownFields() throws IOException {
+    ToolAnnotations a = mapper.readValue("""
+            {"title":"t","futureField":42}""", ToolAnnotations.class);
+    assertThat(a.title()).isEqualTo("t");
+}
+```
+
 ## Code of Conduct
 
 This project follows a Code of Conduct. Please review it in

JACKSON_REFACTORING_PLAN.md

@@ -0,0 +1,103 @@
+# Jackson Forward-Compat Refactor — Execution Plan
+
+This document is the executable plan for refactoring JSON-RPC and domain-type serialization in the MCP Java SDK so that:
+
+- Domain records evolve in a backwards/forwards compatible way.
+- Sealed interfaces are removed (hard break in this release).
+- Polymorphic types deserialize correctly without hand-rolled `Map` parsing where possible.
+- JSON flows through the pipeline with the minimum number of passes.
+
+Execute the stages in order. Each stage should compile and pass the existing test suite.
+
+---
+
+## Decision log
+
+### Why `params`/`result` stay as `Object`
+
+An earlier draft of this plan changed `JSONRPCRequest.params`, `JSONRPCNotification.params`, and `JSONRPCResponse.result` from `Object` to `@JsonRawValue String`, with per-module `RawJsonDeserializer` mixins that used `JsonGenerator.copyCurrentStructure` to capture the raw JSON substring during envelope deserialization.
+
+**This was reverted.** The reason: the `RawJsonDeserializer` re-serializes the intermediate parsed tree (Map/List) back into a String, then the handler later calls `readValue(params, TargetType)` to deserialize a third time. That is three passes for what should be two. The mixin approach does not skip the intermediate Map — it just adds an extra serialization step on top.
+
+The real cost of the existing `Object params` path is:
+
+1. `readValue(jsonText, MAP_TYPE_REF)` → `HashMap` (full JSON parse)
+2. `convertValue(map, JSONRPCRequest.class)` → envelope record (in-memory structural walk, `params` is a `LinkedHashMap`)
+3. `convertValue(params, TargetType.class)` in handler → typed POJO (in-memory structural walk)
+
+Step 2 is eliminated by the `@JsonTypeInfo(DEDUCTION)` annotation added to `JSONRPCMessage` (see Stage 1), which collapses steps 1+2 into a single `readValue`. Step 3 (`convertValue`) is an in-memory walk, not a JSON parse — it is acceptable.
+
+### Why `@JsonTypeInfo` on `CompleteReference` is annotated but not yet functional
+
+`@JsonTypeInfo(DEDUCTION)` + `@JsonSubTypes` has been added to `CompleteReference`. However, during test development it was confirmed that Jackson (both version 2 and 3) does **not** discover these annotations when deserializing `CompleteRequest.ref` (a field typed as the abstract `CompleteReference` interface) from a `Map` produced by `convertValue`. The annotation is present in bytecode but is not picked up by the deserializer introspector in either Jackson version for this specific pattern (static nested interface of a final class, target of a `convertValue` from Map).
+
+The practical consequence is that `convertValue(paramsMap, CompleteRequest.class)` still fails on the `ref` field. The old `parseCompletionParams` hand-rolled Map parser has been replaced with `jsonMapper.convertValue(params, new TypeRef<CompleteRequest>() {})` — this works as long as the `ref` object in the `params` Map is deserialized correctly. **This needs investigation and a fix** (see Open issues below).
+
+---
+
+## Current state (as of last execution)
+
+### Done — all existing tests pass (274 in `mcp-core`, 30 in each Jackson module)
+
+**`McpSchema.java`**
+- `JSONRPCMessage`: `sealed` removed; `@JsonTypeInfo(DEDUCTION)` + `@JsonSubTypes` added.
+- `JSONRPCRequest`, `JSONRPCNotification`, `JSONRPCResponse`: stale `// @JsonFormat` and `// TODO: batching support` comments removed. `params`/`result` remain `Object`.
+- `deserializeJsonRpcMessage`: still uses the two-step Map approach for compatibility with non-Jackson mappers (e.g. the Gson-based mapper tested in `GsonMcpJsonMapperTests`). The `@JsonTypeInfo` annotation on `JSONRPCMessage` enables direct `mapper.readValue(json, JSONRPCMessage.class)` for callers who use a Jackson mapper directly.
+- `Request`, `Result`, `Notification`: `sealed`/`permits` removed — plain interfaces.
+- `ResourceContents`: `sealed`/`permits` removed; existing `@JsonTypeInfo(DEDUCTION)` retained.
+- `CompleteReference`: `sealed`/`permits` removed; `@JsonTypeInfo(DEDUCTION)` + `@JsonSubTypes` added. **Annotation not yet functional for `convertValue` path — see Open issues.**
+- `Content`: `sealed`/`permits` removed; `@JsonIgnore` added to default `type()` method to prevent double emission of the `type` property.
+- `LoggingLevel`: `@JsonCreator` + `static final Map BY_NAME` added (lenient deserialization, `null` for unknown values).
+- `StopReason`: `Arrays.stream` lookup replaced with `static final Map BY_VALUE`.
+- `Prompt`: constructors no longer coerce `null` arguments to `new ArrayList<>()`. `Prompt.withDefaults(...)` factory added for callers that want the empty-list behaviour.
+- `CompleteCompletion`: `@JsonInclude` changed from `ALWAYS` to `NON_ABSENT`; `@JsonIgnoreProperties(ignoreUnknown = true)` added; non-null `values` validated in canonical constructor.
+- Annotation sweep: all `public record` types inside `McpSchema` now have both `@JsonInclude(NON_ABSENT)` and `@JsonIgnoreProperties(ignoreUnknown = true)`. Records that were missing either annotation: `Sampling`, `Elicitation`, `Form`, `Url`, `CompletionCapabilities`, `LoggingCapabilities`, `PromptCapabilities`, `ResourceCapabilities`, `ToolCapabilities`, `CompleteArgument`, `CompleteContext`.
+- `JsonIgnore` import added.
+
+**`McpAsyncServer.java`**
+- `parseCompletionParams` deleted.
+- Completion handler uses `jsonMapper.convertValue(params, new TypeRef<>() {})`.
+
+**`McpStatelessAsyncServer.java`**
+- `parseCompletionParams` deleted.
+- Completion handler uses `jsonMapper.convertValue(params, new TypeRef<>() {})`.
+
+**`ServerParameters.java`**
+- `@JsonInclude` and `@JsonProperty` annotations removed; javadoc states it is not a wire type.
+
+### New tests (in `mcp-test`) — all passing ✅
+
+Four new test classes written to `mcp-test/src/test/java/io/modelcontextprotocol/spec/`:
+
+| Class | Status |
+|---|---|
+| `JsonRpcDispatchTests` | **All 5 pass** |
+| `ContentJsonTests` | **All 5 pass** |
+| `SchemaEvolutionTests` | **All 12 pass** |
+| `CompleteReferenceJsonTests` | **All 6 pass** |
+
+---
+
+## Resolved issues
+
+### 1. `CompleteReference` polymorphic dispatch
+
+**Fix:** Changed `@JsonTypeInfo` on `CompleteReference` from `DEDUCTION` to `NAME + EXISTING_PROPERTY + visible=true`. DEDUCTION failed because `PromptReference` and `ResourceReference` share the `type` field, making their field fingerprints non-disjoint. `EXISTING_PROPERTY` uses the `"type"` field value as the explicit discriminator, working correctly with both `readValue` and `convertValue`.
+
+### 2. `CompleteCompletion` null field omission
+
+**Fix:** Changed `@JsonInclude` on `CompleteCompletion` from `NON_ABSENT` to `NON_NULL`. `NON_ABSENT` does not reliably suppress plain-null `Integer`/`Boolean` record components in Jackson 2.20.
+
+### 3. `Prompt` null arguments omission
+
+**Fix:** Changed `@JsonInclude` on `Prompt` from `NON_ABSENT` to `NON_NULL`. The root cause was the same as issue 2, compounded by the stale jar in `~/.m2` masking the constructor fix. Both issues resolved together.
+
+### 4. `JSONRPCMessage` DEDUCTION removed
+
+**Fix:** Removed `@JsonTypeInfo(DEDUCTION)` and `@JsonSubTypes` from `JSONRPCMessage`. JSON-RPC message types cannot be distinguished by unique field presence alone (Request and Notification both have `method`+`params`; Request and Response both have `id`). The `deserializeJsonRpcMessage` method continues to handle dispatch correctly via the Map-based approach.
+
+---
+
+## Completed stages
+
+All planned work is done. See `CONTRIBUTING.md` (§ "Evolving wire-serialized records") and `MIGRATION-2.0.md` for the contributor recipe and migration notes.

MIGRATION-2.0.md

@@ -0,0 +1,84 @@
+# Migration Guide — 2.0
+
+This document covers breaking and behavioural changes introduced in the 2.0 release of the MCP Java SDK.
+
+---
+
+## Jackson / JSON serialization changes
+
+### Sealed interfaces removed
+
+The following interfaces were `sealed` in 1.x and are now plain interfaces in 2.0:
+
+- `McpSchema.JSONRPCMessage`
+- `McpSchema.Request`
+- `McpSchema.Result`
+- `McpSchema.Notification`
+- `McpSchema.ResourceContents`
+- `McpSchema.CompleteReference`
+- `McpSchema.Content`
+
+**Impact:** Exhaustive `switch` expressions or `switch` statements that relied on the sealed hierarchy for completeness checking must add a `default` branch. The compiler will no longer reject switches that omit one of the known subtypes.
+
+### `CompleteReference` now carries `@JsonTypeInfo`
+
+`CompleteReference` (and its implementations `PromptReference` and `ResourceReference`) is now annotated with `@JsonTypeInfo(use = NAME, include = EXISTING_PROPERTY, property = "type", visible = true)`. Jackson will automatically dispatch to the correct subtype based on the `"type"` field in the JSON without any hand-written map-walking code.
+
+**Action:** Remove any custom code that manually inspected the `"type"` field of a completion reference map and instantiated `PromptReference` / `ResourceReference` by hand. A plain `mapper.readValue(json, CompleteRequest.class)` or `mapper.convertValue(paramsMap, CompleteRequest.class)` is sufficient.
+
+### `Prompt` canonical constructor no longer coerces `null` arguments
+
+In 1.x, `new Prompt(name, description, null)` silently stored an empty list for `arguments`. In 2.0 it stores `null`.
+
+**Action:**
+
+- Code that expected `prompt.arguments()` to return an empty list when not provided will now receive `null`. Add a null-check or use the new `Prompt.withDefaults(name, description, arguments)` factory, which preserves the old behaviour by coercing `null` to `[]`.
+- On the wire, a prompt without an `arguments` field deserializes with `arguments == null` (it is not coerced to an empty list).
+
+### `CompleteCompletion` optional fields omitted when null
+
+`CompleteResult.CompleteCompletion.total` and `CompleteCompletion.hasMore` are now omitted from serialized JSON when they are `null` (previously they were always emitted). Deserializers that required these fields to be present in every response must be updated to treat their absence as `null`.
+
+### `CompleteCompletion.values` is mandatory in the Java API
+
+The compact constructor for `CompleteCompletion` asserts that `values` is not `null`. Code that constructed a completion result with a null `values` list will now fail at runtime.
+
+**Action:** Always pass a non-null list (for example `List.of()` when there are no suggestions).
+
+### `LoggingLevel` deserialization is lenient
+
+`LoggingLevel` now uses a `@JsonCreator` factory (`fromValue`) so that JSON string values deserialize in a case-insensitive way. **Unrecognized level strings deserialize to `null`** instead of causing deserialization to fail.
+
+**Impact:** `SetLevelRequest`, `LoggingMessageNotification`, and any other type that embeds `LoggingLevel` can observe a `null` level when the wire value is unknown or misspelled. Downstream code must null-check or validate before use.
+
+### `Content.type()` is ignored for Jackson serialization
+
+The `Content` interface still exposes `type()` as a convenience for Java callers, but the method is annotated with `@JsonIgnore` so Jackson does not treat it as a duplicate `"type"` property alongside `@JsonTypeInfo` on the interface.
+
+**Impact:** Custom serializers or `ObjectMapper` configuration that relied on serializing `Content` through the default `type()` accessor alone should use the concrete content records (each of which carries a real `"type"` property) or the polymorphic setup on `Content`.
+
+### `ServerParameters` no longer carries Jackson annotations
+
+`ServerParameters` (in `client/transport`) has had its `@JsonProperty` and `@JsonInclude` annotations removed. It was never a wire type and is not serialized to JSON in normal SDK usage. If your code serialized or deserialized `ServerParameters` using Jackson, switch to a plain map or a dedicated DTO.
+
+### Record annotation sweep
+
+Wire-oriented `public record` types in `McpSchema` consistently use `@JsonInclude(JsonInclude.Include.NON_ABSENT)` (or equivalent per-type configuration) and `@JsonIgnoreProperties(ignoreUnknown = true)`. Nested capability objects under `ClientCapabilities` / `ServerCapabilities` (for example `Sampling`, `Elicitation`, `CompletionCapabilities`, `LoggingCapabilities`, prompt/resource/tool capability records) also ignore unknown JSON properties. This means:
+
+- **Unknown fields** in incoming JSON are silently ignored, improving forward compatibility with newer server or client versions.
+- **Absent optional properties** are omitted from outgoing JSON where `NON_ABSENT` applies, and optional Java components deserialize as `null` when missing on the wire.
+
+### `Tool.inputSchema` is `Map<String, Object>`, not `JsonSchema`
+
+The `Tool` record now models `inputSchema` (and `outputSchema`) as arbitrary JSON Schema objects as `Map<String, Object>`, so dialect-specific keywords (`$ref`, `unevaluatedProperties`, vendor extensions, and so on) round-trip without being trimmed by a narrow `JsonSchema` record.
+
+**Impact:**
+
+- Java code that used `Tool.inputSchema()` as a `JsonSchema` must switch to `Map<String, Object>` (or copy into your own schema wrapper).
+- `Tool.Builder.inputSchema(JsonSchema)` remains as a **deprecated** helper that maps the old record into a map; prefer `inputSchema(Map)` or `inputSchema(McpJsonMapper, String)`.
+
+### Optional JSON Schema validation on `tools/call` (server)
+
+When a `JsonSchemaValidator` is available (including the default from `McpJsonDefaults.getSchemaValidator()` when you do not configure one explicitly) and `validateToolInputs` is left at its default of `true`, the server validates incoming tool arguments against `tool.inputSchema()` before invoking the tool. Failed validation produces a `CallToolResult` with `isError` set and a textual error in the content.
+
+**Action:** Ensure `inputSchema` maps are valid for your validator, tighten client arguments, or disable validation with `validateToolInputs(false)` on the server builder if you must preserve pre-2.0 behaviour.

mcp-core/src/main/java/io/modelcontextprotocol/client/transport/ServerParameters.java

@@ -1,5 +1,5 @@
 /*
- * Copyright 2024-2024 the original author or authors.
+ * Copyright 2024-2026 the original author or authors.
  */
 
 package io.modelcontextprotocol.client.transport;
@@ -11,17 +11,15 @@
 import java.util.Map;
 import java.util.stream.Collectors;
 
-import com.fasterxml.jackson.annotation.JsonInclude;
-import com.fasterxml.jackson.annotation.JsonProperty;
 import io.modelcontextprotocol.util.Assert;
 
 /**
- * Server parameters for stdio client.
+ * Server parameters for stdio client. This is not a wire type; Jackson annotations are
+ * intentionally omitted.
  *
  * @author Christian Tzolov
  * @author Dariusz Jędrzejczyk
  */
-@JsonInclude(JsonInclude.Include.NON_ABSENT)
 public class ServerParameters {
 
 	// Environment variables to inherit by default
@@ -32,13 +30,10 @@ public class ServerParameters {
 						"SYSTEMDRIVE", "SYSTEMROOT", "TEMP", "USERNAME", "USERPROFILE")
 				: Arrays.asList("HOME", "LOGNAME", "PATH", "SHELL", "TERM", "USER");
 
-	@JsonProperty("command")
 	private String command;
 
-	@JsonProperty("args")
 	private List<String> args = new ArrayList<>();
 
-	@JsonProperty("env")
 	private Map<String, String> env;
 
 	private ServerParameters(String command, List<String> args, Map<String, String> env) {