Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sentry: txn_lock_gatekeeper.go:70: ×: concurrent txn use detected. ba: Get [/NamespaceTable/30/1/104/106/"privileges"/4/1], [txn: 08871753] (1) secondary error attachment | ×: concurrent txn use d... #132743

Open
cockroach-sentry opened this issue Oct 16, 2024 · 0 comments
Labels
branch-release-24.2 Used to mark GA and release blockers, technical advisories, and bugs for 24.2 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.

Comments

@cockroach-sentry
Copy link
Collaborator

cockroach-sentry commented Oct 16, 2024

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/5995814642/?referrer=webhooks_plugin

Panic Message:

txn_lock_gatekeeper.go:70: ×: concurrent txn use detected. ba: Get [/NamespaceTable/30/1/104/106/"privileges"/4/1], [txn: 08871753]
(1) secondary error attachment
  | ×: concurrent txn use detected. ba: Get [/NamespaceTable/30/1/104/106/"privileges"/4/1], [txn: 08871753]
  | (1) attached stack trace
  |   -- stack trace:
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).execInternal.func1.1
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/internal.go:1264
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next.func1
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/internal.go:559
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/internal.go:618
  |   | github.com/cockroachdb/cockroach/pkg/sql/syntheticprivilegecache.(*Cache).readFromStorage
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/syntheticprivilegecache/cache.go:145
  |   | github.com/cockroachdb/cockroach/pkg/sql/syntheticprivilegecache.(*Cache).Get.func1
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/syntheticprivilegecache/cache.go:93
  |   | github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).doCall.func1
  |   | 	github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight/singleflight.go:389
  |   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTask
  |   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:320
  |   | github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).doCall
  |   | 	github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight/singleflight.go:388
  |   | runtime.goexit
  |   | 	src/runtime/asm_amd64.s:1695
  | Wraps: (2) ×
  | Wraps: (3) assertion failure
  | Wraps: (4)
  |   | (opaque error wrapper)
  |   | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
  |   | reportable 0:
  |   |
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnLockGatekeeper).SendLocked
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_lock_gatekeeper.go:70
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnMetricRecorder).SendLocked
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_metric_recorder.go:46
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).sendLockedWithRefreshAttempts
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:239
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).SendLocked
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:167
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnCommitter).SendLocked
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_committer.go:149
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnPipeliner).SendLocked
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_pipeliner.go:326
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSeqNumAllocator).SendLocked
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_seq_num_allocator.go:117
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnHeartbeater).SendLocked
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go:270
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:539
  |   | github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:1138
  |   | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Send
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/txn.go:1282
  |   | github.com/cockroachdb/cockroach/pkg/kv.sendAndFill
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:970
  |   | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Run
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/txn.go:800
  |   | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv.catalogQuery.query
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv/catalog_query.go:59
  |   | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv.catalogReader.GetByNames
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv/catalog_reader.go:333
  |   | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv.(*cachedCatalogReader).GetByNames
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv/catalog_reader_cached.go:398
  |   | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).getNonVirtualDescriptorID.func7
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/descriptor.go:599
  |   | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).getNonVirtualDescriptorID
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/descriptor.go:620
  |   | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.getDescriptorByName
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/descriptor.go:398
  |   | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.ByNameGetter.Table
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/getters.go:301
  |   | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.PrefixAndTable
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/helpers.go:138
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*schemaResolver).LookupObject
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/schema_resolver.go:174
  |   | github.com/cockroachdb/cockroach/pkg/sql/catalog/resolver.ResolveExisting
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/resolver/resolver.go:345
  |   | github.com/cockroachdb/cockroach/pkg/sql/catalog/resolver.ResolveExistingObject
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/resolver/resolver.go:184
  |   | github.com/cockroachdb/cockroach/pkg/sql/catalog/resolver.ResolveExistingTableObject
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/resolver/resolver.go:108
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*optCatalog).ResolveDataSource
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/opt_catalog.go:260
  |   | github.com/cockroachdb/cockroach/pkg/sql/opt.(*Metadata).CheckDependencies
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/opt/metadata.go:393
  |   | github.com/cockroachdb/cockroach/pkg/sql/opt/memo.(*Memo).IsStale
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/opt/memo/memo.go:457
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*optPlanningCtx).fetchPreparedMemoLegacy
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:775
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*optPlanningCtx).buildExecMemo
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:809
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*planner).makeOptimizerPlan
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:251
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).makeExecPlan
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2291
  | Wraps: (5) concurrent txn use detected. ba: Get [/NamespaceTable/30/1/104/106/"privileges"/4/1], [txn: 08871753]
  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *assert.withAssertionFailure (4) *errbase.opaqueWrapper (5) *errutil.leafError
Wraps: (2) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).execInternal.func1.1
  | 	github.com/cockroachdb/cockroach/pkg/sql/internal.go:1264
  | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/internal.go:559
  | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/internal.go:618
  | github.com/cockroachdb/cockroach/pkg/sql/syntheticprivilegecache.(*Cache).readFromStorage
  | 	github.com/cockroachdb/cockroach/pkg/sql/synt...
Stacktrace (expand for inline code snippets):

src/runtime/asm_amd64.s#L1694-L1696

if err := opts.Stop.RunTask(ctx, g.opName+":"+key, func(ctx context.Context) {
c.val, c.err = fn(ctx)

f(ctx)
return nil

if err := opts.Stop.RunTask(ctx, g.opName+":"+key, func(ctx context.Context) {
c.val, c.err = fn(ctx)
}); err != nil {

func(loadCtx context.Context) (_ interface{}, retErr error) {
return c.readFromStorage(loadCtx, txn, spo)
})

for {
ok, err := it.Next(ctx)
if err != nil {

r.first = nil
return handleDataObject(*first)
}

if r.errCallback != nil {
r.lastErr = r.errCallback(r.lastErr)
r.errCallback = nil

cockroach/pkg/sql/internal.go

Lines 1263 to 1265 in 217b43e

if err != nil && !errIsRetriable(err) {
err = errors.Wrapf(err, "%s", opName)
}

if err := planner.makeOptimizerPlan(ctx); err != nil {
log.VEventf(ctx, 1, "optimizer plan failed: %v", err)

execMemo, err := opc.buildExecMemo(ctx)
if err != nil {

// set to force_custom_plan.
m, err := opc.fetchPreparedMemoLegacy(ctx)
if err != nil {

// re-prepare it.
if isStale, err := prepared.BaseMemo.IsStale(ctx, p.EvalContext(), opc.catalog); err != nil {
return nil, err

// access the object.
if depsUpToDate, err := m.Metadata().CheckDependencies(ctx, evalCtx, catalog); err != nil {
return true, err

tableName := name.ToTableName()
toCheck, _, err = optCatalog.ResolveDataSource(ctx, cat.Flags{}, &tableName)
if err != nil || !dataSource.Equals(toCheck) {

}
prefix, desc, err := resolver.ResolveExistingTableObject(ctx, oc.planner, &oc.tn, lflags)
if err != nil {

un := tn.ToUnresolvedObjectName()
desc, prefix, err := ResolveExistingObject(ctx, sc, un, lookupFlags)
if err != nil || desc == nil {

) (res catalog.Descriptor, _ catalog.ResolvedObjectPrefix, err error) {
found, prefix, obj, err := ResolveExisting(ctx, un, sc, lookupFlags, sc.CurrentDatabase(), sc.CurrentSearchPath())
if err != nil {

if isVirtualSchema || curDb != "" {
if found, prefix, result, err = r.LookupObject(
ctx, lookupFlags, curDb, u.Schema(), u.Object(),

case tree.TableObject:
prefix, desc, err = descs.PrefixAndTable(ctx, g, &tn)
case tree.TypeObject:

}
tbl, err := g.Table(ctx, p.Database, p.Schema, name.Object())
return p, tbl, err

) (catalog.TableDescriptor, error) {
desc, err := getDescriptorByName(
ctx, g.KV(), g.Descriptors(), db, sc, name, g.flags, catalog.Table,

}
id, err := tc.getNonVirtualDescriptorID(ctx, txn, db, sc, name, flags)
if err != nil || id == descpb.InvalidID {

} {
isDone, id, err := fn()
if err != nil {

}
read, err := tc.cr.GetByNames(ctx, txn, []descpb.NameInfo{ni})
if err != nil {

uncachedNameInfos := nameInfos[:numUncached]
read, err := c.cr.GetByNames(ctx, txn, uncachedNameInfos)
if err != nil {

cq := catalogQuery{codec: cr.codec}
err := cq.query(ctx, txn, &mc, func(codec keys.SQLCodec, b *kv.Batch) {
for _, nameInfo := range nameInfos {

in(cq.codec, b)
if err := txn.Run(ctx, b); err != nil {
return err

cockroach/pkg/kv/txn.go

Lines 799 to 801 in 217b43e

}
return sendAndFill(ctx, txn.Send, b)
}

cockroach/pkg/kv/db.go

Lines 969 to 971 in 217b43e

ba.AdmissionHeader = b.AdmissionHeader
b.response, b.pErr = send(ctx, ba)
b.fillResults(ctx)

cockroach/pkg/kv/txn.go

Lines 1281 to 1283 in 217b43e

txn.mu.Unlock()
br, pErr := txn.db.sendUsingSender(ctx, ba, sender)

cockroach/pkg/kv/db.go

Lines 1137 to 1139 in 217b43e

br, pErr := sender.Send(ctx, ba)
if pErr != nil {

// Send the command through the txnInterceptor stack.
br, pErr := tc.interceptorStack[0].SendLocked(ctx, ba)

h.mu.ifReqs++
br, pErr := h.wrapped.SendLocked(ctx, ba)
h.mu.ifReqs--

return s.wrapped.SendLocked(ctx, ba)
}

// Send through wrapped lockedSender. Unlocks while sending then re-locks.
br, pErr := tp.wrapped.SendLocked(ctx, ba)

if !hasET {
return tc.wrapped.SendLocked(ctx, ba)
}

// Send through wrapped lockedSender. Unlocks while sending then re-locks.
br, pErr := sr.sendLockedWithRefreshAttempts(ctx, ba, sr.maxRefreshAttempts())
if pErr != nil {

}
br, pErr := sr.wrapped.SendLocked(ctx, ba)

br, pErr := m.wrapped.SendLocked(ctx, ba)
if pErr != nil {

return nil, kvpb.NewError(
errors.AssertionFailedf("concurrent txn use detected. ba: %s", ba))
}

src/runtime/asm_amd64.s in runtime.goexit at line 1695
pkg/util/syncutil/singleflight/singleflight.go in pkg/util/syncutil/singleflight.(*Group).doCall at line 388
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTask at line 320
pkg/util/syncutil/singleflight/singleflight.go in pkg/util/syncutil/singleflight.(*Group).doCall.func1 at line 389
pkg/sql/syntheticprivilegecache/cache.go in pkg/sql/syntheticprivilegecache.(*Cache).Get.func1 at line 93
pkg/sql/syntheticprivilegecache/cache.go in pkg/sql/syntheticprivilegecache.(*Cache).readFromStorage at line 145
pkg/sql/internal.go in pkg/sql.(*rowsIterator).Next at line 618
pkg/sql/internal.go in pkg/sql.(*rowsIterator).Next.func1 at line 559
pkg/sql/internal.go in pkg/sql.(*InternalExecutor).execInternal.func1.1 at line 1264
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).makeExecPlan at line 2291
pkg/sql/plan_opt.go in pkg/sql.(*planner).makeOptimizerPlan at line 251
pkg/sql/plan_opt.go in pkg/sql.(*optPlanningCtx).buildExecMemo at line 809
pkg/sql/plan_opt.go in pkg/sql.(*optPlanningCtx).fetchPreparedMemoLegacy at line 775
pkg/sql/opt/memo/memo.go in pkg/sql/opt/memo.(*Memo).IsStale at line 457
pkg/sql/opt/metadata.go in pkg/sql/opt.(*Metadata).CheckDependencies at line 393
pkg/sql/opt_catalog.go in pkg/sql.(*optCatalog).ResolveDataSource at line 260
pkg/sql/catalog/resolver/resolver.go in pkg/sql/catalog/resolver.ResolveExistingTableObject at line 108
pkg/sql/catalog/resolver/resolver.go in pkg/sql/catalog/resolver.ResolveExistingObject at line 184
pkg/sql/catalog/resolver/resolver.go in pkg/sql/catalog/resolver.ResolveExisting at line 345
pkg/sql/schema_resolver.go in pkg/sql.(*schemaResolver).LookupObject at line 174
pkg/sql/catalog/descs/helpers.go in pkg/sql/catalog/descs.PrefixAndTable at line 138
pkg/sql/catalog/descs/getters.go in pkg/sql/catalog/descs.ByNameGetter.Table at line 301
pkg/sql/catalog/descs/descriptor.go in pkg/sql/catalog/descs.getDescriptorByName at line 398
pkg/sql/catalog/descs/descriptor.go in pkg/sql/catalog/descs.(*Collection).getNonVirtualDescriptorID at line 620
pkg/sql/catalog/descs/descriptor.go in pkg/sql/catalog/descs.(*Collection).getNonVirtualDescriptorID.func7 at line 599
pkg/sql/catalog/internal/catkv/catalog_reader_cached.go in pkg/sql/catalog/internal/catkv.(*cachedCatalogReader).GetByNames at line 398
pkg/sql/catalog/internal/catkv/catalog_reader.go in pkg/sql/catalog/internal/catkv.catalogReader.GetByNames at line 333
pkg/sql/catalog/internal/catkv/catalog_query.go in pkg/sql/catalog/internal/catkv.catalogQuery.query at line 59
pkg/kv/txn.go in pkg/kv.(*Txn).Run at line 800
pkg/kv/db.go in pkg/kv.sendAndFill at line 970
pkg/kv/txn.go in pkg/kv.(*Txn).Send at line 1282
pkg/kv/db.go in pkg/kv.(*DB).sendUsingSender at line 1138
pkg/kv/kvclient/kvcoord/txn_coord_sender.go in pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send at line 539
pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go in pkg/kv/kvclient/kvcoord.(*txnHeartbeater).SendLocked at line 270
pkg/kv/kvclient/kvcoord/txn_interceptor_seq_num_allocator.go in pkg/kv/kvclient/kvcoord.(*txnSeqNumAllocator).SendLocked at line 117
pkg/kv/kvclient/kvcoord/txn_interceptor_pipeliner.go in pkg/kv/kvclient/kvcoord.(*txnPipeliner).SendLocked at line 326
pkg/kv/kvclient/kvcoord/txn_interceptor_committer.go in pkg/kv/kvclient/kvcoord.(*txnCommitter).SendLocked at line 149
pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go in pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).SendLocked at line 167
pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go in pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).sendLockedWithRefreshAttempts at line 239
pkg/kv/kvclient/kvcoord/txn_interceptor_metric_recorder.go in pkg/kv/kvclient/kvcoord.(*txnMetricRecorder).SendLocked at line 46
pkg/kv/kvclient/kvcoord/txn_lock_gatekeeper.go in pkg/kv/kvclient/kvcoord.(*txnLockGatekeeper).SendLocked at line 70

Tags

Tag Value
Command server
Environment v24.2.3
Go Version go1.22.5 X:nocoverageredesign
Platform linux amd64
Distribution CCL
Cockroach Release v24.2.3
Cockroach SHA 217b43e
# of CPUs 4
# of Goroutines 357

Jira issue: CRDB-43254

@cockroach-sentry cockroach-sentry added branch-release-24.2 Used to mark GA and release blockers, technical advisories, and bugs for 24.2 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. labels Oct 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
branch-release-24.2 Used to mark GA and release blockers, technical advisories, and bugs for 24.2 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.
Projects
None yet
Development

No branches or pull requests

1 participant