Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: SIGPROF arriving on a foreign thread before any cgo call is made will crash the process #9456

Closed
minux opened this issue Dec 27, 2014 · 4 comments
Closed

runtime: SIGPROF arriving on a foreign thread before any cgo call is made will crash the process #9456

minux opened this issue Dec 27, 2014 · 4 comments
Labels
FrozenDueToAge
Milestone
Go1.4.1

Comments

@minux
Copy link
Member

minux commented Dec 27, 2014

runtime.badsignal will try to deliver the signal received by foreign threads to sigqueue using the cgo callback mechanism, however, cgocallback will not work if the process has not made any cgo call yet.

A simple reproduction program:

package main

/*
#include <signal.h>
#include <pthread.h>
volatile int x;
static void *thread(void *p) {
    (void)p;
    while (x == 0)
        ;
    pthread_kill(pthread_self(), SIGPROF);
    return NULL;
}
__attribute__((constructor)) void test() {
    pthread_t tid;
    pthread_create(&tid, 0, thread, NULL);
}
*/
import "C"
import "sync/atomic"
import "unsafe"

func main() {
    atomic.StoreInt32((*int32)(unsafe.Pointer(&C.x)), 1)
    select {}
}

This example is contrived, but imagine the thread is created by an external library linked in, and the user is profiling Go code. See https://groups.google.com/forum/#!topic/golang-nuts/SMhWSUsfPag for a real-world example (the symptom is the same, and disabling the cgocallback line in function runtime.badsignal fixes it, but I'm still not sure about how the program could create the extra thread during benchmarking without any cgocall. The problem could also be in other part of runtime.)

Tentatively labeled 1.4.1, as there is no meaningful workaround for 1.4.
@minux minux self-assigned this Dec 27, 2014
@minux minux added this to the Go1.4.1 milestone Dec 27, 2014
@minux minux changed the title runtime: SIGPROF might arrive on a foreign thread before any cgo call is made will crash the process runtime: SIGPROF arriving on a foreign thread before any cgo call is made will crash the process Dec 27, 2014
@kortschak
Copy link
Contributor

Sorry, I was not able to construct a reproducer that does not depend on being in tests. However, this code reproduces the failure if it is used to replace the covariancematrix_test.go file in github.com/gonum/stat.

Run using go test -cpuprofile foo.prof.

(github.com/gonum/blas/cblas is installed using OpenBLAS).

@minux
Copy link
Member Author

minux commented Dec 27, 2014

Thanks. I've verified that the essence of the problem is the same as the
contrived example I created. Will fix and add a test to runtime.

OpenBLAS has a global constructor (https://github.com/xianyi/OpenBLAS/blob/develop/driver/others/memory.c#L1291)
that creates worker threads correspond to the each of processors in the system.

The worker threads are periodically waken up to check if there is work for
them (https://github.com/xianyi/OpenBLAS/blob/develop/driver/others/blas_server.c#L268),
so it's possible that during the long data preparation steps (where the Go
code has not made any cgo call yet), SIGPROF comes to one of the waken worker thread
of OpenBLAS and trigger the problem.

@minux minux closed this as completed in 5da9c8c Dec 31, 2014
minux added a commit that referenced this issue Jan 14, 2015
@minux @rsc
[release-branch.go1.4] runtime: ignore SIGPROF to foreign threads bef…
cc7bbb0 
…ore cgocallback is fully initialized

Some libraries, for example, OpenBLAS, create work threads in a global constructor.
If we're doing cpu profiling, it's possible that SIGPROF might come to some of the
worker threads before we make our first cgo call. Cgocallback used to terminate the
process when that happens, but it's better to miss a couple profiling signals than
to abort in this case.

Fixes #9456.

Change-Id: I112b8e1a6e10e6cc8ac695a4b518c0f577309b6b
Reviewed-on: https://go-review.googlesource.com/2141
Reviewed-by: Ian Lance Taylor <iant@golang.org>
(cherry picked from commit 5da9c8c)
Reviewed-on: https://go-review.googlesource.com/2789
Reviewed-by: Andrew Gerrand <adg@golang.org>
@btracey
Copy link
Contributor

btracey commented May 27, 2015

I'm seeing this periodically in Go 1.4.1. It's especially confusing because it happens seemingly randomly, and the most recent code doesn't import any cgo code, although I do have "export CGO_LDFLAGS="-L/Users/brendan/software/OpenBLAS -lopenblas" in my .bash_profile. I'd like to be more helpful, but I don't know how to make a consistent reproducer. I think I've only seen it when using "go run" if that has any relevance.

@minux
Copy link
Member Author

minux commented May 27, 2015 via email

@golang golang locked and limited conversation to collaborators Jun 25, 2016
wheatman pushed a commit to wheatman/go-akaros that referenced this issue Jun 25, 2018
@minux @wheatman
[release-branch.go1.4] runtime: ignore SIGPROF to foreign threads bef…
99b2ee6 
…ore cgocallback is fully initialized

Some libraries, for example, OpenBLAS, create work threads in a global constructor.
If we're doing cpu profiling, it's possible that SIGPROF might come to some of the
worker threads before we make our first cgo call. Cgocallback used to terminate the
process when that happens, but it's better to miss a couple profiling signals than
to abort in this case.

Fixes golang#9456.

Change-Id: I112b8e1a6e10e6cc8ac695a4b518c0f577309b6b
Reviewed-on: https://go-review.googlesource.com/2141
Reviewed-by: Ian Lance Taylor <iant@golang.org>
(cherry picked from commit 5da9c8c)
Reviewed-on: https://go-review.googlesource.com/2789
Reviewed-by: Andrew Gerrand <adg@golang.org>
wheatman pushed a commit to wheatman/go-akaros that referenced this issue Jun 26, 2018
@minux @wheatman
[release-branch.go1.4] runtime: ignore SIGPROF to foreign threads bef…
9a04ae3 
…ore cgocallback is fully initialized

Some libraries, for example, OpenBLAS, create work threads in a global constructor.
If we're doing cpu profiling, it's possible that SIGPROF might come to some of the
worker threads before we make our first cgo call. Cgocallback used to terminate the
process when that happens, but it's better to miss a couple profiling signals than
to abort in this case.

Fixes golang#9456.

Change-Id: I112b8e1a6e10e6cc8ac695a4b518c0f577309b6b
Reviewed-on: https://go-review.googlesource.com/2141
Reviewed-by: Ian Lance Taylor <iant@golang.org>
(cherry picked from commit 5da9c8c)
Reviewed-on: https://go-review.googlesource.com/2789
Reviewed-by: Andrew Gerrand <adg@golang.org>
wheatman pushed a commit to wheatman/go-akaros that referenced this issue Jul 9, 2018
@minux @wheatman
[release-branch.go1.4] runtime: ignore SIGPROF to foreign threads bef…
8c7d137 
…ore cgocallback is fully initialized

Some libraries, for example, OpenBLAS, create work threads in a global constructor.
If we're doing cpu profiling, it's possible that SIGPROF might come to some of the
worker threads before we make our first cgo call. Cgocallback used to terminate the
process when that happens, but it's better to miss a couple profiling signals than
to abort in this case.

Fixes golang#9456.

Change-Id: I112b8e1a6e10e6cc8ac695a4b518c0f577309b6b
Reviewed-on: https://go-review.googlesource.com/2141
Reviewed-by: Ian Lance Taylor <iant@golang.org>
(cherry picked from commit 5da9c8c)
Reviewed-on: https://go-review.googlesource.com/2789
Reviewed-by: Andrew Gerrand <adg@golang.org>
wheatman pushed a commit to wheatman/go-akaros that referenced this issue Jul 20, 2018
@minux @wheatman
[release-branch.go1.4] runtime: ignore SIGPROF to foreign threads bef…
44fb2fe 
…ore cgocallback is fully initialized

Some libraries, for example, OpenBLAS, create work threads in a global constructor.
If we're doing cpu profiling, it's possible that SIGPROF might come to some of the
worker threads before we make our first cgo call. Cgocallback used to terminate the
process when that happens, but it's better to miss a couple profiling signals than
to abort in this case.

Fixes golang#9456.

Change-Id: I112b8e1a6e10e6cc8ac695a4b518c0f577309b6b
Reviewed-on: https://go-review.googlesource.com/2141
Reviewed-by: Ian Lance Taylor <iant@golang.org>
(cherry picked from commit 5da9c8c)
Reviewed-on: https://go-review.googlesource.com/2789
Reviewed-by: Andrew Gerrand <adg@golang.org>
wheatman pushed a commit to wheatman/go-akaros that referenced this issue Jul 30, 2018
@minux @wheatman
[release-branch.go1.4] runtime: ignore SIGPROF to foreign threads bef…
402dfd5 
…ore cgocallback is fully initialized

Some libraries, for example, OpenBLAS, create work threads in a global constructor.
If we're doing cpu profiling, it's possible that SIGPROF might come to some of the
worker threads before we make our first cgo call. Cgocallback used to terminate the
process when that happens, but it's better to miss a couple profiling signals than
to abort in this case.

Fixes golang#9456.

Change-Id: I112b8e1a6e10e6cc8ac695a4b518c0f577309b6b
Reviewed-on: https://go-review.googlesource.com/2141
Reviewed-by: Ian Lance Taylor <iant@golang.org>
(cherry picked from commit 5da9c8c)
Reviewed-on: https://go-review.googlesource.com/2789
Reviewed-by: Andrew Gerrand <adg@golang.org>
@rsc rsc unassigned minux Jun 23, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge
Projects
None yet
Milestone
Go1.4.1
Development

No branches or pull requests
4 participants
@minux @kortschak @btracey @gopherbot