Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fatal error: concurrent map writes #397

Open
giovannibaratta opened this issue Apr 15, 2024 · 0 comments
Open

fatal error: concurrent map writes #397

giovannibaratta opened this issue Apr 15, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@giovannibaratta
Copy link

Describe the bug

While creating tanzu-mission-control_tanzu_kubernetes_cluster, Terraform crashed after ~18 mins with the following error. Even if terraform crashed, the clusters were created in TMC.

tanzu-mission-control_tanzu_kubernetes_cluster.tap_view: Still creating... [18m30s elapsed]
tanzu-mission-control_tanzu_kubernetes_cluster.tap_run: Still creating... [18m30s elapsed]
tanzu-mission-control_tanzu_kubernetes_cluster.tap_build: Still creating... [18m30s elapsed]
tanzu-mission-control_tanzu_kubernetes_cluster.tap_iterate: Still creating... [18m30s elapsed]
╷
│ Error: Plugin did not respond
│ 
│   with tanzu-mission-control_tanzu_kubernetes_cluster.tap_build,
│   on clusters-tap-build.tf line 18, in resource "tanzu-mission-control_tanzu_kubernetes_cluster" "tap_build":
│   18: resource "tanzu-mission-control_tanzu_kubernetes_cluster" "tap_build" {
│ 
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ApplyResourceChange call.
│ The plugin logs may contain more details.
╵
╷
│ Error: Plugin did not respond
│ 
│   with tanzu-mission-control_tanzu_kubernetes_cluster.tap_iterate,
│   on clusters-tap-iterate.tf line 18, in resource "tanzu-mission-control_tanzu_kubernetes_cluster" "tap_iterate":
│   18: resource "tanzu-mission-control_tanzu_kubernetes_cluster" "tap_iterate" {
│ 
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ApplyResourceChange call.
│ The plugin logs may contain more details.
╵
╷
│ Error: Plugin did not respond
│ 
│   with tanzu-mission-control_tanzu_kubernetes_cluster.tap_run,
│   on clusters-tap-run.tf line 18, in resource "tanzu-mission-control_tanzu_kubernetes_cluster" "tap_run":
│   18: resource "tanzu-mission-control_tanzu_kubernetes_cluster" "tap_run" {
│ 
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ApplyResourceChange call.
│ The plugin logs may contain more details.
╵
╷
│ Error: Plugin did not respond
│ 
│   with tanzu-mission-control_tanzu_kubernetes_cluster.tap_view,
│   on clusters-tap-view.tf line 18, in resource "tanzu-mission-control_tanzu_kubernetes_cluster" "tap_view":
│   18: resource "tanzu-mission-control_tanzu_kubernetes_cluster" "tap_view" {
│ 
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ApplyResourceChange call.
│ The plugin logs may contain more details.
╵
Releasing state lock. This may take a few moments...

Stack trace from the terraform-provider-tanzu-mission-control_v1.4.4 plugin:

fatal error: concurrent map writes

goroutine 52 [running]:
net/textproto.MIMEHeader.Set(...)
        net/textproto/header.go:22
net/http.Header.Set(...)
        net/http/header.go:40
github.com/vmware/terraform-provider-tanzu-mission-control/internal/authctx.refreshSMUserAuthCtx(0x140001b70e0)
        github.com/vmware/terraform-provider-tanzu-mission-control/internal/authctx/selfmanaged.go:236 +0x11c
github.com/vmware/terraform-provider-tanzu-mission-control/internal/authctx.glob..func1(0x140001b70e0, 0x14000b2c618?, {0x1039fd580?, 0x14000b2c618?})
        github.com/vmware/terraform-provider-tanzu-mission-control/internal/authctx/helper.go:117 +0x54
github.com/vmware/terraform-provider-tanzu-mission-control/internal/resources/tanzukubernetescluster.waitClusterReady({0x103a134a8, 0x140001b7260}, 0x140001b70e0, 0x14000abdc80)
        github.com/vmware/terraform-provider-tanzu-mission-control/internal/resources/tanzukubernetescluster/helper.go:133 +0x32c
github.com/vmware/terraform-provider-tanzu-mission-control/internal/resources/tanzukubernetescluster.readFullClusterResourceWait({0x103a134a8, 0x140001b7260}, 0x10331a63a?, 0xf?, {0x140007e97c0, 0x1, 0x0?}, 0x1)
        github.com/vmware/terraform-provider-tanzu-mission-control/internal/resources/tanzukubernetescluster/helper.go:68 +0x60
github.com/vmware/terraform-provider-tanzu-mission-control/internal/resources/tanzukubernetescluster.readResourceWait({0x103a134e0, 0x140005c64b0}, 0x10484fd40?, 0x14000a18bc0?, {0x140007e97c0, 0x1, 0x4}, 0x63?)
        github.com/vmware/terraform-provider-tanzu-mission-control/internal/resources/tanzukubernetescluster/helper.go:45 +0x15c
github.com/vmware/terraform-provider-tanzu-mission-control/internal/resources/tanzukubernetescluster.resourceTanzuKubernetesClusterRead({0x103a134e0, 0x140005c64b0}, 0x14000960e80, {0x1038fa8c0?, 0x1400059a180})
        github.com/vmware/terraform-provider-tanzu-mission-control/internal/resources/tanzukubernetescluster/resource_tanzu_kuberenetes_cluster.go:125 +0x2d8
github.com/vmware/terraform-provider-tanzu-mission-control/internal/resources/tanzukubernetescluster.resourceTanzuKubernetesClusterCreate({0x103a134e0, 0x1400041ef30}, 0x0?, {0x1038fa8c0?, 0x1400059a180?})
        github.com/vmware/terraform-provider-tanzu-mission-control/internal/resources/tanzukubernetescluster/resource_tanzu_kuberenetes_cluster.go:103 +0x5d8
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).create(0x103a134e0?, {0x103a134e0?, 0x1400041ef30?}, 0xd?, {0x1038fa8c0?, 0x1400059a180?})
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.16.0/helper/schema/resource.go:702 +0x64
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).Apply(0x1400077da40, {0x103a134e0, 0x1400041ef30}, 0x14000782b60, 0x14000693180, {0x1038fa8c0, 0x1400059a180})
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.16.0/helper/schema/resource.go:837 +0x86c
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ApplyResourceChange(0x14000780630, {0x103a13438?, 0x14000b60700?}, 0x14000180230)
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.16.0/helper/schema/grpc_provider.go:1021 +0xb70
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ApplyResourceChange(0x1400047b0e0, {0x103a134e0?, 0x1400041e900?}, 0x140009060e0)
        github.com/hashicorp/terraform-plugin-go@v0.9.0/tfprotov5/tf5server/server.go:812 +0x384
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ApplyResourceChange_Handler({0x103915180?, 0x1400047b0e0}, {0x103a134e0, 0x1400041e900}, 0x14000906070, 0x0)
        github.com/hashicorp/terraform-plugin-go@v0.9.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:385 +0x170
google.golang.org/grpc.(*Server).processUnaryRPC(0x140007681e0, {0x103a1a378, 0x14000502d00}, 0x140007dc360, 0x140006a7ef0, 0x1047daf00, 0x0)
        google.golang.org/grpc@v1.56.3/server.go:1335 +0xc64
google.golang.org/grpc.(*Server).handleStream(0x140007681e0, {0x103a1a378, 0x14000502d00}, 0x140007dc360, 0x0)
        google.golang.org/grpc@v1.56.3/server.go:1712 +0x82c
google.golang.org/grpc.(*Server).serveStreams.func1.1()
        google.golang.org/grpc@v1.56.3/server.go:947 +0xb4
created by google.golang.org/grpc.(*Server).serveStreams.func1
        google.golang.org/grpc@v1.56.3/server.go:958 +0x174

goroutine 1 [select, 18 minutes]:
github.com/hashicorp/go-plugin.Serve(0x140001b61e0)
        github.com/hashicorp/go-plugin@v1.4.3/server.go:469 +0x10a8
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.Serve({0x10330fccd, 0x8}, 0x1400004c230, {0x0, 0x0, 0x0})
        github.com/hashicorp/terraform-plugin-go@v0.9.0/tfprotov5/tf5server/server.go:311 +0x9e0
github.com/hashicorp/terraform-plugin-sdk/v2/plugin.tf5serverServe(0x140001b6180)
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.16.0/plugin/serve.go:178 +0x4d4
github.com/hashicorp/terraform-plugin-sdk/v2/plugin.Serve(0x140001b6180)
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.16.0/plugin/serve.go:118 +0x190
main.main()
        github.com/vmware/terraform-provider-tanzu-mission-control/main.go:31 +0xf4

goroutine 9 [select, 18 minutes]:
github.com/hashicorp/go-plugin.(*gRPCBrokerServer).Recv(0x0?)
        github.com/hashicorp/go-plugin@v1.4.3/grpc_broker.go:121 +0x58
github.com/hashicorp/go-plugin.(*GRPCBroker).Run(0x14000180e10)
        github.com/hashicorp/go-plugin@v1.4.3/grpc_broker.go:411 +0x40
created by github.com/hashicorp/go-plugin.(*GRPCServer).Init
        github.com/hashicorp/go-plugin@v1.4.3/grpc_server.go:85 +0x424

goroutine 10 [IO wait, 18 minutes]:
internal/poll.runtime_pollWait(0x12c728350, 0x72)
        runtime/netpoll.go:305 +0xa0
internal/poll.(*pollDesc).wait(0x140001b6660?, 0x14000225000?, 0x1)
        internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
        internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x140001b6660, {0x14000225000, 0x1000, 0x1000})
        internal/poll/fd_unix.go:167 +0x1e0
os.(*File).read(...)
        os/file_posix.go:31
os.(*File).Read(0x1400000e980, {0x14000225000?, 0x400?, 0x10371c7a0?})
        os/file.go:119 +0x5c
bufio.(*Reader).Read(0x1400008bf30, {0x14000534000, 0x400, 0x0?})
        bufio/bufio.go:237 +0x1e0
github.com/hashicorp/go-plugin.copyChan({0x103a1e178, 0x1400026a7e0}, 0x0?, {0x1039fcb60?, 0x1400000e980?})
        github.com/hashicorp/go-plugin@v1.4.3/grpc_stdio.go:181 +0x154
created by github.com/hashicorp/go-plugin.newGRPCStdioServer
        github.com/hashicorp/go-plugin@v1.4.3/grpc_stdio.go:37 +0x10c

goroutine 11 [IO wait, 18 minutes]:
internal/poll.runtime_pollWait(0x12c728170, 0x72)
        runtime/netpoll.go:305 +0xa0
internal/poll.(*pollDesc).wait(0x140001b6720?, 0x14000536000?, 0x1)
        internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
        internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x140001b6720, {0x14000536000, 0x1000, 0x1000})
        internal/poll/fd_unix.go:167 +0x1e0
os.(*File).read(...)
        os/file_posix.go:31
os.(*File).Read(0x1400000e9a0, {0x14000536000?, 0x400?, 0x10371c7a0?})
        os/file.go:119 +0x5c
bufio.(*Reader).Read(0x14000086730, {0x14000534400, 0x400, 0x0?})
        bufio/bufio.go:237 +0x1e0
github.com/hashicorp/go-plugin.copyChan({0x103a1e178, 0x1400026a7e0}, 0x0?, {0x1039fcb60?, 0x1400000e9a0?})
        github.com/hashicorp/go-plugin@v1.4.3/grpc_stdio.go:181 +0x154
created by github.com/hashicorp/go-plugin.newGRPCStdioServer
        github.com/hashicorp/go-plugin@v1.4.3/grpc_stdio.go:38 +0x198

goroutine 37 [syscall, 18 minutes]:

Error: The terraform-provider-tanzu-mission-control_v1.4.4 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

Reproduction steps

Unknown

I applied the terraform module several times in clean environments and I got this error only once
source code is available here https://github.com/giovannibaratta/vmware-tanzu-training/tree/main/terraform/stages/50-tmc

Expected behavior

Terraform should not crash

Additional context

I am using TMC self managed v1.2
MacBook M1 (Terraform v1.7.4 on darwin_arm64)
@giovannibaratta giovannibaratta added the bug Something isn't working label Apr 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@giovannibaratta