Skip to content

NullReferenceException during Auto Checkpoint when AOF is Disabled with AOF Size Limit Configured #1696

@wesleyisaacd

Description

@wesleyisaacd

Describe the bug

When AOF (Append-Only File) is disabled in Microsoft Garnet, the system still attempts to execute an auto-checkpoint task based on AofSizeLimit.

Since AOF is turned off, the internal AOF-related objects are not initialized, leading to a NullReferenceException during checkpoint evaluation.

Additionally, runtime failures occur when executing commands like KEYS, where the storage engine (Tsavorite) throws exceptions while iterating records. This indicates inconsistent or invalid internal state—likely triggered by the improper checkpoint logic.

Error Detail
Error 1:
[000.2026-04-10 17:55:41.6826] (Error) Exception received at AutoCheckpointTask
System.NullReferenceException: Object reference not set to an instance of an object.
at Garnet.server.MultiDatabaseManager.TaskCheckpointBasedOnAofSizeLimitAsync(Int64 aofSizeLimit, CancellationToken token, ILogger logger) in //libs/server/Databases/MultiDatabaseManager.cs:line 293
at Garnet.server.StoreWrapper.AutoCheckpointBasedOnAofSizeLimit(Int64 aofSizeLimit, CancellationToken token, ILogger logger) in /
/libs/server/StoreWrapper.cs:line 658
[000.2026-04-10 19:18:51.9763] (Error) Exception received at AutoCheckpointTask
System.NullReferenceException: Object reference not set to an instance of an object.
at Garnet.server.MultiDatabaseManager.TaskCheckpointBasedOnAofSizeLimitAsync(Int64 aofSizeLimit, CancellationToken token, ILogger logger) in //libs/server/Databases/MultiDatabaseManager.cs:line 293
at Garnet.server.StoreWrapper.AutoCheckpointBasedOnAofSizeLimit(Int64 aofSizeLimit, CancellationToken token, ILogger logger) in /
/libs/server/StoreWrapper.cs:line 658
[000.2026-04-13 01:00:12.8911] (Critical) [] [00319E6D] ProcessMessages threw an exception:
System.NullReferenceException: Object reference not set to an instance of an object.
at Tsavorite.core.TsavoriteKVIterator8.ProcessNonTailmostMainKvRecord(RecordInfo recordInfo, TKey key) in /_/libs/storage/Tsavorite/cs/src/core/Index/Tsavorite/TsavoriteIterator.cs:line 231 at Tsavorite.core.TsavoriteKVIterator8.PushNext[TScanFunctions](TScanFunctions& scanFunctions, Int64 numRecords, Boolean& stop) in //libs/storage/Tsavorite/cs/src/core/Index/Tsavorite/TsavoriteIterator.cs:line 184
at Tsavorite.core.TsavoriteKV4.Iterate[TInput,TOutput,TContext,TFunctions,TScanFunctions](TFunctions functions, TScanFunctions& scanFunctions, Int64 untilAddress) in /_/libs/storage/Tsavorite/cs/src/core/Index/Tsavorite/TsavoriteIterator.cs:line 39 at Garnet.server.StorageSession.DBKeys(ArgSlice pattern) in /_/libs/server/Storage/Session/Common/ArrayKeyIterationFunctions.cs:line 200 at Garnet.server.GarnetApi3.GetDbKeys(ArgSlice pattern) in /
/libs/server/API/GarnetApi.cs:line 467
at Garnet.server.RespServerSession.NetworkKEYS[TGarnetApi](TGarnetApi& storageApi) in //libs/server/Resp/ArrayCommands.cs:line 223
at Garnet.server.RespServerSession.ProcessOtherCommands[TGarnetApi](RespCommand command, TGarnetApi& storageApi) in /
/libs/server/Resp/RespServerSession.cs:line 985
at Garnet.server.RespServerSession.ProcessArrayCommands[TGarnetApi](RespCommand cmd, TGarnetApi& storageApi) in //libs/server/Resp/RespServerSession.cs:line 802
at Garnet.server.RespServerSession.ProcessMessages() in /
/libs/server/Resp/RespServerSession.cs:line 620
at Garnet.server.RespServerSession.TryConsumeMessages(Byte* reqBuffer, Int32 bytesReceived) in /_/libs/server/Resp/RespServerSession.cs:line 468
[000.2026-04-13 01:00:12.9039] (Error) [172.16.1.104:58907] [00202E7F] Unexpected response, this should never happen

Error 2:
[000.2026-04-13 02:00:03.8100] (Critical) [] [00319E6D] ProcessMessages threw an exception:
System.NullReferenceException: Object reference not set to an instance of an object.
at Tsavorite.core.TsavoriteKVIterator8.ProcessNonTailmostMainKvRecord(RecordInfo recordInfo, TKey key) in /_/libs/storage/Tsavorite/cs/src/core/Index/Tsavorite/TsavoriteIterator.cs:line 231 at Tsavorite.core.TsavoriteKVIterator8.PushNext[TScanFunctions](TScanFunctions& scanFunctions, Int64 numRecords, Boolean& stop) in //libs/storage/Tsavorite/cs/src/core/Index/Tsavorite/TsavoriteIterator.cs:line 184
at Tsavorite.core.TsavoriteKV4.Iterate[TInput,TOutput,TContext,TFunctions,TScanFunctions](TFunctions functions, TScanFunctions& scanFunctions, Int64 untilAddress) in /_/libs/storage/Tsavorite/cs/src/core/Index/Tsavorite/TsavoriteIterator.cs:line 39 at Garnet.server.StorageSession.DBKeys(ArgSlice pattern) in /_/libs/server/Storage/Session/Common/ArrayKeyIterationFunctions.cs:line 200 at Garnet.server.GarnetApi3.GetDbKeys(ArgSlice pattern) in /
/libs/server/API/GarnetApi.cs:line 467
at Garnet.server.RespServerSession.NetworkKEYS[TGarnetApi](TGarnetApi& storageApi) in //libs/server/Resp/ArrayCommands.cs:line 223
at Garnet.server.RespServerSession.ProcessOtherCommands[TGarnetApi](RespCommand command, TGarnetApi& storageApi) in /
/libs/server/Resp/RespServerSession.cs:line 985
at Garnet.server.RespServerSession.ProcessArrayCommands[TGarnetApi](RespCommand cmd, TGarnetApi& storageApi) in //libs/server/Resp/RespServerSession.cs:line 802
at Garnet.server.RespServerSession.ProcessMessages() in /
/libs/server/Resp/RespServerSession.cs:line 620
at Garnet.server.RespServerSession.TryConsumeMessages(Byte* reqBuffer, Int32 bytesReceived) in //libs/server/Resp/RespServerSession.cs:line 468
[000.2026-04-13 02:00:03.8105] (Error) [172.16.1.104:58907] [00202E7F] Unexpected response, this should never happen
[000.2026-04-13 03:00:09.9369] (Critical) [] [00319E6D] ProcessMessages threw an exception:
System.NullReferenceException: Object reference not set to an instance of an object.
at Tsavorite.core.TsavoriteKVIterator8.ProcessNonTailmostMainKvRecord(RecordInfo recordInfo, TKey key) in /_/libs/storage/Tsavorite/cs/src/core/Index/Tsavorite/TsavoriteIterator.cs:line 231 at Tsavorite.core.TsavoriteKVIterator8.PushNext[TScanFunctions](TScanFunctions& scanFunctions, Int64 numRecords, Boolean& stop) in /
/libs/storage/Tsavorite/cs/src/core/Index/Tsavorite/TsavoriteIterator.cs:line 184
at Tsavorite.core.TsavoriteKV4.Iterate[TInput,TOutput,TContext,TFunctions,TScanFunctions](TFunctions functions, TScanFunctions& scanFunctions, Int64 untilAddress) in /_/libs/storage/Tsavorite/cs/src/core/Index/Tsavorite/TsavoriteIterator.cs:line 39 at Garnet.server.StorageSession.DBKeys(ArgSlice pattern) in /_/libs/server/Storage/Session/Common/ArrayKeyIterationFunctions.cs:line 200 at Garnet.server.GarnetApi3.GetDbKeys(ArgSlice pattern) in //libs/server/API/GarnetApi.cs:line 467
at Garnet.server.RespServerSession.NetworkKEYS[TGarnetApi](TGarnetApi& storageApi) in /
/libs/server/Resp/ArrayCommands.cs:line 223
at Garnet.server.RespServerSession.ProcessOtherCommands[TGarnetApi](RespCommand command, TGarnetApi& storageApi) in //libs/server/Resp/RespServerSession.cs:line 985
at Garnet.server.RespServerSession.ProcessArrayCommands[TGarnetApi](RespCommand cmd, TGarnetApi& storageApi) in /
/libs/server/Resp/RespServerSession.cs:line 802
at Garnet.server.RespServerSession.ProcessMessages() in //libs/server/Resp/RespServerSession.cs:line 620
at Garnet.server.RespServerSession.TryConsumeMessages(Byte* reqBuffer, Int32 bytesReceived) in /
/libs/server/Resp/RespServerSession.cs:line 468
[000.2026-04-13 03:00:09.9376] (Error) [172.16.1.104:58907] [00202E7F] Unexpected response, this should never happen
[000.2026-04-13 04:00:15.8127] (Critical) [] [00319E6D] ProcessMessages threw an exception:
System.NullReferenceException: Object reference not set to an instance of an object.
at Tsavorite.core.TsavoriteKVIterator8.ProcessNonTailmostMainKvRecord(RecordInfo recordInfo, TKey key) in /_/libs/storage/Tsavorite/cs/src/core/Index/Tsavorite/TsavoriteIterator.cs:line 231 at Tsavorite.core.TsavoriteKVIterator8.PushNext[TScanFunctions](TScanFunctions& scanFunctions, Int64 numRecords, Boolean& stop) in //libs/storage/Tsavorite/cs/src/core/Index/Tsavorite/TsavoriteIterator.cs:line 184
at Tsavorite.core.TsavoriteKV4.Iterate[TInput,TOutput,TContext,TFunctions,TScanFunctions](TFunctions functions, TScanFunctions& scanFunctions, Int64 untilAddress) in /_/libs/storage/Tsavorite/cs/src/core/Index/Tsavorite/TsavoriteIterator.cs:line 39 at Garnet.server.StorageSession.DBKeys(ArgSlice pattern) in /_/libs/server/Storage/Session/Common/ArrayKeyIterationFunctions.cs:line 200 at Garnet.server.GarnetApi3.GetDbKeys(ArgSlice pattern) in /
/libs/server/API/GarnetApi.cs:line 467
at Garnet.server.RespServerSession.NetworkKEYS[TGarnetApi](TGarnetApi& storageApi) in //libs/server/Resp/ArrayCommands.cs:line 223
at Garnet.server.RespServerSession.ProcessOtherCommands[TGarnetApi](RespCommand command, TGarnetApi& storageApi) in /
/libs/server/Resp/RespServerSession.cs:line 985
at Garnet.server.RespServerSession.ProcessArrayCommands[TGarnetApi](RespCommand cmd, TGarnetApi& storageApi) in //libs/server/Resp/RespServerSession.cs:line 802
at Garnet.server.RespServerSession.ProcessMessages() in /
/libs/server/Resp/RespServerSession.cs:line 620
at Garnet.server.RespServerSession.TryConsumeMessages(Byte* reqBuffer, Int32 bytesReceived) in /_/libs/server/Resp/RespServerSession.cs:line 468
[000.2026-04-13 04:00:15.8132] (Error) [172.16.1.104:58907] [00202E7F] Unexpected response, this should never happen

Steps to reproduce the bug

Garnet configuration file is attached and Aof configuration is,

/* Enable write ahead logging (append-only file). */
"EnableAOF" : false,

/* Total AOF memory buffer used in bytes (rounds down to power of 2) - spills to disk after this limit 64m */
"AofMemorySize" : "1k",

/* Size of each AOF page in bytes(rounds down to power of 2) 4m */
"AofPageSize" : "1k",

/* AOF replication (safe tail address) refresh frequency in milliseconds. 0 = auto refresh after every enqueue. */
"AofReplicationRefreshFrequencyMs": 0,

/* Subscriber (safe tail address) refresh frequency in milliseconds (for pub-sub). 0 = auto refresh after every enqueue. */
"SubscriberRefreshFrequencyMs": 0,

/* Write ahead logging (append-only file) commit issue frequency in milliseconds. 0 = issue an immediate commit per operation, -1 = manually issue commits using COMMITAOF command */
"CommitFrequencyMs" : 0,

/* Wait for AOF to flush the commit before returning results to client. Warning: will greatly increase operation latency. */
"WaitForCommit" : false,

/* Maximum size of AOF (rounds down to power of 2) after which unsafe truncation will be applied. Left empty AOF will grow without bound unless a checkpoint is taken */
"AofSizeLimit" : "1k",

Garnet.txt

Expected behavior

Garnet should not validate or process AOF-related parameters when AOF is disabled alternatively, it should accept a minimal AOF size configuration without enforcing checks, in order to reduce memory utilization.

Screenshots

No response

Release version

v1.1.1

IDE

No response

OS version

No response

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions