Hi, We have had this crash a few times now at client site, I was unable to replicate under light load so client sent us a dump file to check against our service exe build.
Bug report: Use-after-free race in TSparkleTimer causes server crash (Sparkle.Sys.Timer / RemoteDB.Server.Wrappers)
Product: TMS Sparkle (Sparkle.Sys.Timer.pas) + TMS RemoteDB (RemoteDB.Server.Wrappers.pas, RemoteDB.Server.Module.pas)Platform: Win32 service application (Delphi ), Windows Server 2022 (10.0.20348)Severity: Process crash (access violation) in a production RemoteDB serverReported by: Tom, Development-X Limited
Summary
TSparkleTimer deletes its Windows timer-queue timer with the non-blocking form of DeleteTimerQueueTimer (CompletionEvent = 0) in all cases, including from its destructor. In-flight timer callbacks therefore continue running after the timer object has been freed. In the RemoteDB server, the instance-timeout design makes this race certain to be exercised: TRemoteDatabase.OnTimeout synchronously destroys the TRemoteDatabase, which frees the TSparkleTimer while its own callback is still executing. If any exception is raised during teardown, TSparkleTimer.DoTimer's except handler then calls HaltTimer on the freed object, reads a garbage FTimerHandle from recycled heap memory, and passes it to DeleteTimerQueueTimer — crashing the process inside ntdll.
We have a full crash dump from a production server that captures this exactly, including the freed object's memory recycled by an HTTP string (the "timer handle" read from the corpse was the bytes 48 74 74 70 = "Http").
Production symptom
Recurring service crash after long uptimes (~10 days), correlated with periods of connection churn (many sessions expiring and reconnecting, e.g. Monday morning). Event log examples (same binary, two different fault sites — classic corruption-downstream signature):
Exception 0xC0000005, faulting module remotedb.exe (RTL region)
Exception 0xC0000005, faulting module ntdll.dll, offset 0x26C6B (RtlDeleteTimer)
Dump evidence
WER full dump, WinDbg !analyze -v (abridged):
ExceptionAddress: ntdll!RtlDeleteTimer+0x5b
ExceptionCode: c0000005 (Access violation)
Attempt to read from address 70747464
CONTEXT: esi=70747448 ; "Timer handle" = bytes "Http" — freed memory recycled by a string
ntdll!RtlDeleteTimer+0x5b: mov eax, dword ptr [esi+1Ch]
STACK:
ntdll!RtlDeleteTimer+0x5b
KERNELBASE!DeleteTimerQueueTimer+0x26
<exe: TSparkleTimer.HaltTimer> ; resolved via detailed .map
<exe: TSparkleTimer.DoTimer except-block> ; return address lands immediately after
; the except-path HaltTimer call, before DoneExcept
ntdll!RtlpTpTimerCallback+0xa0 ; i.e. we are INSIDE the timer-queue callback
ntdll!TppTimerpExecuteCallback+0x98
ntdll!TppWorkerThread+0x734
The three exe frames were matched instruction-for-instruction against Sparkle.Sys.Timer.pas via a detailed map file: WinTimerProc, CreateTimer (argument push sequence for CreateTimerQueueTimer with phNewTimer = @Self.FTimerHandle, Parameter = Self), HaltTimer (DeleteTimerQueueTimer(0, [Self+10h], 0) then zeroing the field), and DoTimer (SingleShot pre-halt; try DoCallback except HaltTimer end). The faulting call is the except-path HaltTimer, executing on a freed Self.
Root cause chain
TRemoteDatabase (RemoteDB.Server.Wrappers) creates a SingleShot TSparkleTimer for instance timeout; the callback captures a raw Self pointer and invokes FOnTimeout(Id).
TCustomRemoteDBModule.CreateNewDB wires OnTimeout to RemoveDatabase(DatabaseId).
On idle expiry, the timer fires on a Windows thread-pool thread. RemoveDatabase extracts the IRemoteDatabase from FDatabases; when the interface reference releases at procedure exit, TRemoteDatabase.Destroy runs on the callback thread, inside DoCallback.
Destroy → FTimer.Free → DestroyTimer → HaltTimer → non-blocking DeleteTimerQueueTimer → the TSparkleTimer is freed while its callback frame is still live on the stack.
Control returns up into DoTimer on the freed object. If teardown raised an exception (e.g. a database disconnect error inside FConnection := nil), the except HaltTimer path executes against recycled memory → garbage handle → AV in ntdll.
A sibling variant exists without exceptions: a second thread (request worker releasing the last interface ref, or /removedb) frees the timer while a fired callback is in flight; the callback then touches freed memory in the SingleShot pre-halt or DoCallback.
Contributing issues, same files:
TSparkleTimer.Update (called from TRemoteDatabase.Touch on every request) destroys and recreates the OS timer with no synchronisation on FTimerHandle; concurrent Touch/callback can double-delete a (recyclable) handle value.
TCustomRemoteDBModule.RemoveDatabase: FDatabases.ExtractPair(Id).Value returns nil when the Id is already gone (timeout racing /removedb or the status API DELETE), and nil then flows into DoDatabaseDestroy → user OnDatabaseDestroy handler.
Fix applied locally
Sparkle.Sys.Timer.pas — TSparkleTimer (MSWINDOWS branch):
// new field
FCallbackThreadId: TThreadID;
procedure TSparkleTimer.DoTimer;
begin
FCallbackThreadId := TThread.Current.ThreadID;
// No SingleShot pre-halt: Period is Infinite so it cannot refire, and keeping the
// handle alive is what lets a concurrent Destroy block until this callback completes.
try
DoCallback; // may free Self (RemoteDB OnTimeout path does) — nothing below may touch Self
except
// swallow only; the previous HaltTimer here executed on freed memory (the crash site)
end;
end;
procedure TSparkleTimer.HaltTimer;
var
H: THandle;
begin
H := THandle(AtomicExchange(NativeInt(FTimerHandle), 0)); // race-safe, idempotent
if H <> 0 then
begin
if TThread.Current.ThreadID = FCallbackThreadId then
DeleteTimerQueueTimer(0, H, 0) // from within callback: must not block
else
DeleteTimerQueueTimer(0, H, INVALID_HANDLE_VALUE); // elsewhere: wait for in-flight callbacks
end;
end;
RemoteDB.Server.Module.pas — RemoveDatabase:
DBRef := FDatabases.ExtractPair(DatabaseId).Value;
if DBRef = nil then Exit; // already removed by a concurrent path (timeout vs /removedb race)
DB := DBRef as TRemoteDatabase;
DoDatabaseDestroy(DB);
Lock-ordering was audited: Destroy/interface finalisation never runs while FDBSection is held (finalisation occurs at procedure exit, after Leave), and Touch is always called outside FDBSection, so the blocking delete introduces no deadlock in the RemoteDB server paths.
Residual and suggested upstream redesign
The thread-ID discrimination has one theoretical gap: thread-pool threads are recycled, so a destroyer coincidentally scheduled on the thread that ran the last callback takes the non-blocking path unnecessarily. Exposure is reduced by orders of magnitude, not to zero. A structurally complete fix would avoid passing the raw Self as the timer callback context — e.g. a small reference-counted context cell owned jointly by the timer and the callback, or re-arming via ChangeTimerQueueTimer on a single long-lived handle (which would also eliminate the per-request delete/recreate churn in Update/Touch) with deletion only ever from a drained state. The raw SelfPtr capture in TRemoteDatabase's callback (and in TRemoteDBDatabase's client-side keep-alive, which uses the same class and inherits the same race) deserves the same treatment.
Happy to supply the full crash dump, matching map/PDB, and the disassembly-to-source correlation on request.