Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Application crash when locking compilationLock in RelationalCommandCache.cs #34796

Open
tub5 opened this issue Sep 30, 2024 · 1 comment
Open

Application crash when locking compilationLock in RelationalCommandCache.cs #34796

tub5 opened this issue Sep 30, 2024 · 1 comment
Assignees
@roji
Labels
customer-reported

Comments

@tub5
Copy link

tub5 commented Sep 30, 2024 •
edited
Loading

Our services are encountering a crash on production seemingly at random after queries have been run on the server. No exceptions are thrown by EF or by any of our processes.

Hosting environment: Production
Content root path: /app
Now listening on: http://localhost:5000
Application started. Press Ctrl+C to shut down.
[createdump] Gathering state for process 1 dotnet
[createdump] Crashing thread 0118 signal 11 (000b)
[createdump] Writing full dump to file xxxxxxxx
[createdump] Written xxx bytes (xxx pages) to core file
[createdump] Target process is alive
[createdump] Dump successfully written in 2687ms

Analysing the dump we found no heap corruption

(lldb) verifyheap
3,657,942 objects verified, 0 errors.
No heap corruption detected.

When analysing the stack for thread 118 we got the following:

(lldb) clrthreads
ThreadCount:      116
UnstartedThread:  0
BackgroundThread: 39
PendingThread:    0
DeadThread:       76
Hosted Runtime:   no
                                                                                                            Lock  
 DBG   ID     OSID ThreadOBJ           State GC Mode     GC Alloc Context                  Domain           Count Apt Exception
...
  68  109      118 00007F4760035CB0  1021260 Cooperative 00007F4943AA0210:00007F4943AA1130 000055AC7F94D370 -00001 Ukn (Threadpool Worker)

(lldb) setthread 68

(lldb) clrstack -a
OS Thread Id: 0x118 (68)
        Child SP               IP Call Site
00007F475F7F8CF0 00007f89b4ff9b57 [HelperMethodFrame_1OBJ: 00007f475f7f8cf0] System.Threading.Monitor.ReliableEnter(System.Object, Boolean ByRef)
00007F475F7F8E40 00007F893EFDEEEB Microsoft.EntityFrameworkCore.Query.Internal.RelationalCommandCache.GetRelationalCommandTemplate(System.Collections.Generic.IReadOnlyDictionary`2<System.String,System.Object>) [/_/src/EFCore.Relational/Query/Internal/RelationalCommandCache.cs @ 67]
    PARAMETERS:
        this (0x00007F475F7F8EF8) = 0x00007f494344c218
        parameters (0x00007F475F7F8EF0) = 0x00007f4943a9d530
    LOCALS:
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>

00007F475F7F8F80 00007F893EF8902A Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1+AsyncEnumerator+<InitializeReaderAsync>d__21[[System.__Canon, System.Private.CoreLib]].MoveNext() [/_/src/EFCore.Relational/Query/Internal/SingleQueryingEnumerable.cs @ 398]
    PARAMETERS:
        this (0x00007F475F7F8FD8) = 0x00007f475f7f91f0
    LOCALS:
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>

00007F475F7F91A0 00007F893F003F42 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1+AsyncEnumerator+<InitializeReaderAsync>d__21[[System.__Canon, System.Private.CoreLib]], Microsoft.EntityFrameworkCore.Relational]](<InitializeReaderAsync>d__21<System.__Canon> ByRef)
    PARAMETERS:
        stateMachine = <no data>
    LOCALS:
        0x00007F475F7F91C0 = 0x00007f496ccf0538
        0x00007F475F7F91B8 = 0x00007f4943aa0150
        0x00007F475F7F91B0 = 0x0000000000000000
        <no data>

00007F475F7F91F0 00007F893F003DF5 Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1+AsyncEnumerator[[System.__Canon, System.Private.CoreLib]].InitializeReaderAsync(AsyncEnumerator<System.__Canon>, System.Threading.CancellationToken)
    PARAMETERS:
        enumerator = <no data>
        cancellationToken = <no data>
    LOCALS:
        <no data>

00007F475F7F9240 00007F893EF770A3 Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy+<>c__DisplayClass30_0`2+<<ExecuteAsync>b__0>d[[System.__Canon, System.Private.CoreLib],[System.Boolean, System.Private.CoreLib]].MoveNext()
    PARAMETERS:
        this (0x00007F475F7F9250) = 0x00007f475f7f92f0
    LOCALS:
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>

00007F475F7F92A0 00007F893EFDEA12 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy+<>c__DisplayClass30_0`2+<<ExecuteAsync>b__0>d[[System.__Canon, System.Private.CoreLib],[System.Boolean, System.Private.CoreLib]], Microsoft.EntityFrameworkCore]](<<ExecuteAsync>b__0>d<System.__Canon,Boolean> ByRef)
    PARAMETERS:
        stateMachine = <no data>
    LOCALS:
        0x00007F475F7F92C0 = 0x00007f496ccf0538
        0x00007F475F7F92B8 = 0x00007f4943aa0150
        0x00007F475F7F92B0 = 0x0000000000000000
        <no data>

00007F475F7F92F0 00007F893EFDE8B3 Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy+<>c__DisplayClass30_0`2[[System.__Canon, System.Private.CoreLib],[System.Boolean, System.Private.CoreLib]].<ExecuteAsync>b__0(Microsoft.EntityFrameworkCore.DbContext, System.__Canon, System.Threading.CancellationToken)
    PARAMETERS:
        this (<CLR reg>) = 0x00007f4943aa00c8
        context = <no data>
        state = <no data>
        cancellationToken = <no data>
    LOCALS:
        <no data>

00007F475F7F9350 00007F893EF7799F Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy+<ExecuteImplementationAsync>d__31`2[[System.__Canon, System.Private.CoreLib],[System.Boolean, System.Private.CoreLib]].MoveNext()
    PARAMETERS:
        this (0x00007F475F7F9368) = 0x00007f475f7f9460
    LOCALS:
        0x00007F475F7F93BC = 0x00000000ffffffff
        0x00007F475F7F9360 = 0x00007f496fca93c0
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>

00007F475F7F9400 00007F893EFDE726 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy+<ExecuteImplementationAsync>d__31`2[[System.__Canon, System.Private.CoreLib],[System.Boolean, System.Private.CoreLib]], Microsoft.EntityFrameworkCore]](<ExecuteImplementationAsync>d__31`2<System.__Canon,Boolean> ByRef)
    PARAMETERS:
        stateMachine = <no data>
    LOCALS:
        <CLR reg> = 0x00007f496ccf0538
        0x00007F475F7F9418 = 0x00007f4943aa00a0
        0x00007F475F7F9410 = 0x0000000000000000
        <no data>

00007F475F7F9460 00007F893EFDE5C1 Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementationAsync[[System.__Canon, System.Private.CoreLib],[System.Boolean, System.Private.CoreLib]](System.Func`4<Microsoft.EntityFrameworkCore.DbContext,System.__Canon,System.Threading.CancellationToken,System.Threading.Tasks.Task`1<Microsoft.EntityFrameworkCore.Storage.ExecutionResult`1<Boolean>>>, System.Func`4<Microsoft.EntityFrameworkCore.DbContext,System.__Canon,System.Threading.CancellationToken,System.Threading.Tasks.Task`1<Microsoft.EntityFrameworkCore.Storage.ExecutionResult`1<Boolean>>>, System.__Canon, System.Threading.CancellationToken)
    PARAMETERS:
        this = <no data>
        operation = <no data>
        verifySucceeded = <no data>
        state = <no data>
        cancellationToken = <no data>
    LOCALS:
        <no data>

00007F475F7F94E0 00007F893EF78072 Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy+<ExecuteAsync>d__30`2[[System.__Canon, System.Private.CoreLib],[System.Boolean, System.Private.CoreLib]].MoveNext()
    PARAMETERS:
        this (0x00007F475F7F94F0) = 0x00007f475f7f95c0
    LOCALS:
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>

00007F475F7F9570 00007F893EFDD9D2 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy+<ExecuteAsync>d__30`2[[System.__Canon, System.Private.CoreLib],[System.Boolean, System.Private.CoreLib]], Microsoft.EntityFrameworkCore]](<ExecuteAsync>d__30`2<System.__Canon,Boolean> ByRef)
    PARAMETERS:
        stateMachine = <no data>
    LOCALS:
        0x00007F475F7F9590 = 0x00007f496ccf0538
        0x00007F475F7F9588 = 0x00007f4943aa00a0
        0x00007F475F7F9580 = 0x0000000000000000
        <no data>

00007F475F7F95C0 00007F893EFDD87B Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteAsync[[System.__Canon, System.Private.CoreLib],[System.Boolean, System.Private.CoreLib]](System.__Canon, System.Func`4<Microsoft.EntityFrameworkCore.DbContext,System.__Canon,System.Threading.CancellationToken,System.Threading.Tasks.Task`1<Boolean>>, System.Func`4<Microsoft.EntityFrameworkCore.DbContext,System.__Canon,System.Threading.CancellationToken,System.Threading.Tasks.Task`1<Microsoft.EntityFrameworkCore.Storage.ExecutionResult`1<Boolean>>>, System.Threading.CancellationToken)
    PARAMETERS:
        this = <no data>
        state = <no data>
        operation = <no data>
        verifySucceeded = <no data>
        cancellationToken = <no data>
    LOCALS:
        <no data>

00007F475F7F9630 00007F893EF7C0B8 Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1+AsyncEnumerator+<MoveNextAsync>d__20[[System.__Canon, System.Private.CoreLib]].MoveNext()
    PARAMETERS:
        this (0x00007F475F7F9648) = 0x00007f475f7f9730
    LOCALS:
        0x00007F475F7F966C = 0x00000000ffffffff
        0x00007F475F7F9640 = 0x00007f4943a9ffe0
        0x00007F475F7F9668 = 0x0000000000000000
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>

00007F475F7F96B0 00007F893EFAB296 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1+AsyncEnumerator+<MoveNextAsync>d__20[[System.__Canon, System.Private.CoreLib]], Microsoft.EntityFrameworkCore.Relational]](<MoveNextAsync>d__20<System.__Canon> ByRef)
    PARAMETERS:
        stateMachine = <no data>
    LOCALS:
        <CLR reg> = 0x00007f496ccf0538
        0x00007F475F7F96C8 = 0x0000000000000000
        0x00007F475F7F96C0 = 0x0000000000000000
        <no data>

00007F475F7F9710 00007F893EFAB143 Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1+AsyncEnumerator[[System.__Canon, System.Private.CoreLib]].MoveNextAsync()
    PARAMETERS:
        this (<CLR reg>) = 0x00007f4943a9ffe0
    LOCALS:
        <no data>

00007F475F7F9780 00007F893F06FBEE System.Runtime.CompilerServices.ConfiguredCancelableAsyncEnumerable`1+Enumerator[[System.__Canon, System.Private.CoreLib]].MoveNextAsync()
    PARAMETERS:
        this (<CLR reg>) = 0x00007f475f7f9a00
    LOCALS:
        <no data>

00007F475F7F97C0 00007F893F06F463 Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions+<ToListAsync>d__67`1[[System.__Canon, System.Private.CoreLib]].MoveNext() [/_/src/EFCore/Extensions/EntityFrameworkQueryableExtensions.cs @ 2303]
    PARAMETERS:
        this (0x00007F475F7F97D0) = 0x00007f475f7f99d0
    LOCALS:
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>

00007F475F7F98D0 00007F893F395242 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions+<ToListAsync>d__67`1[[System.__Canon, System.Private.CoreLib]], Microsoft.EntityFrameworkCore]](<ToListAsync>d__67`1<System.__Canon> ByRef)
    PARAMETERS:
        stateMachine = <no data>
    LOCALS:
        0x00007F475F7F98F0 = 0x00007f496ccf0538
        0x00007F475F7F98E8 = 0x0000000000000000
        0x00007F475F7F98E0 = 0x0000000000000000
        <no data>

<private business stack>

The crash is not isolated to one entry path or flow through our business logic, but always happens at [/_/src/EFCore.Relational/Query/Internal/RelationalCommandCache.cs @ 67]

Full core dumps can be provided over a secure channel if required.

Provider and version information

EF Core version: 8.0.6
Database provider: Pomelo.EntityFrameworkCore.MySql
Target framework: .NET 8.0
Operating system: Ubuntu 22.04.5 LTS
IDE: N/A
@roji
Copy link
Member

roji commented Oct 2, 2024

This is very unlikely to have anything to do with EF - all it's doing is acquiring a lock with the C# lock construct. I highly suspect an error or corruption somewhere else that manifests as this dump.

Are you able to provide a repro for this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@roji roji

Labels
customer-reported
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@roji @tub5