cgo argument has Go pointer to Go pointer with Go v1.6.1

[idavydov]

It seems that the rules of passing pointer between C and Go changed golang/go#12416. Now example file generates a runtime error.

$ go run example.go 
----- Go L-BFGS-B Example Program -----

----- Sphere Function -----
expected: X: [0 0 0 0 0]
          F: 0
          G: [0 0 0 0 0]
 minimum: X: [-3.552713678800501e-15 4.440892098500626e-15 -2.6645352591003757e-15 8.881784197001252e-16 -1.7763568394002505e-15]
          F: 4.338734978715565e-29
          G: [-7.105427357601002e-15 8.881784197001252e-15 -5.329070518200751e-15 1.7763568394002505e-15 -3.552713678800501e-15]
  status: Exit status: SUCCESS; Message: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL;
   stats: iters: 2; F evals: 4; G evals: 4

----- Rosenbrock Function -----
panic: runtime error: cgo argument has Go pointer to Go pointer

goroutine 1 [running]:
panic(0x5054e0, 0xc82000a510)
    /usr/lib/go-1.6/src/runtime/panic.go:464 +0x3e6
github.com/afbarnard/go-lbfgsb._cgoCheckPointer0(0x4e10a0, 0xc82000a4d0, 0x0, 0x0, 0x0, 0x2)
    github.com/afbarnard/go-lbfgsb/_obj/_cgo_gotypes.go:46 +0x4d
github.com/afbarnard/go-lbfgsb.(*Lbfgsb).Minimize(0xc82008a000, 0x7f8654337358, 0xc82000a4a0, 0xc82000a490, 0x2, 0x2, 0x0, 0x0, 0x0, 0x0, ...)
    /home/idavydov/go/src/github.com/afbarnard/go-lbfgsb/lbfgsb.go:335 +0x89a
main.main()
    /home/idavydov/go/src/github.com/afbarnard/go-lbfgsb/example/example.go:103 +0x75c
exit status 2

[afbarnard]

I have to learn the new cgo rules and build Go 1.6 before I can fix this so it might be a few days.

[idavydov]

Thanks.
So the problem now is that Go code may pass a Go pointer to C provided that the Go memory to which it points does not contain any Go pointers.. In Go 1.6 this is checked at runtime.
The only way I see this implemented for Go1.6 is to have a (global) array/map of callbackData. Then C callback wrapper would receive an identifier of callback, and call a corresponding likelihood function. This was also suggested on their github at some point.
Today it's possible to use the trick of passing pointer as an uintptr to hide it from runtime checks. But this workaround might stop working in the future, since the idea behind this restriction is to be able to move go objects in memory freely, so it might happen that by the time C callback is called, the pointer to callbackData is not valid anymore.

[idavydov]

An example of doing more or less what I described: https://github.com/golang/go/wiki/cgo#function-variables

[idavydov]

Hi, what do you think about #7?
I used the approach described in the golang wiki; this should be thread safe.
Functions registerCallback, unregisterCallback and lookupCallback are quite universal. It's possible to move them to a separate library; but I don't think it's a good idea to create an external dependency.

[idavydov]

There are two possible workaround for those who are affected by the bug.

1. You can use idavydov/go-lbfgsb. You would need to change your imports and reinstall go-lbfgsb.
2. Or you can run your software with GODEBUG=cgocheck=0 environment variable.

I hope eventually this gets fixed in the mainstream.

[hiok2000]

There are two possible workaround for those who are affected by the bug.

1. You can use idavydov/go-lbfgsb. You would need to change your imports and reinstall go-lbfgsb.
2. Or you can run your software with GODEBUG=cgocheck=0 environment variable.

I hope eventually this gets fixed in the mainstream.

I have use idavydov/go-lbfgsb.
And still got this:

----- Go L-BFGS-B Example Program -----

----- Sphere Function -----
expected: X: [0 0 0 0 0]
          F: 0
          G: [0 0 0 0 0]
 minimum: X: [-3.552713678800501e-15 4.440892098500626e-15 -2.6645352591003757e-15 8.881784197001252e-16 -1.7763568394002505e-15]
          F: 4.338734978715565e-29
          G: [-7.105427357601002e-15 8.881784197001252e-15 -5.329070518200751e-15 1.7763568394002505e-15 -3.552713678800501e-15]
  status: Exit status: SUCCESS; Message: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL;
   stats: iters: 2; F evals: 4; G evals: 4

----- Rosenbrock Function -----
panic: runtime error: cgo argument has Go pointer to Go pointer

goroutine 1 [running]:
go-lbfgsb.(*Lbfgsb).Minimize.func1(0xc000042270, 0xc000000002, 0xc00000c278, 0xc00000c2c0, 0xc00000c2d0, 0x5, 0x3ddb7cdfd9d7bdbb, 0x3ddb7cdfd9d7bdbb, 0xc00000c2b0, 0xc00000c2e0, ...)
	D:/Go/src/go-lbfgsb/lbfgsb.go:341 +0x5b
go-lbfgsb.(*Lbfgsb).Minimize(0xc00010fe88, 0x4c9bb0, 0xc000042260, 0xc00000c2b0, 0x2, 0x2, 0x0, 0x20, 0x0, 0x0, ...)
	D:/Go/src/go-lbfgsb/lbfgsb.go:341 +0x546
main.main()
	D:/Go_prj/src/go-lbfgsb/example/example.go:103 +0x5c5

So I have to use GODEBUG=cgocheck=0 environment variable. Then all seems good.
My go version: go1.16 windows/amd64

[idavydov]

Hi @hiok2000 ,

According to your error, you are using in fact afbarnard/go-lbfgsb, not idavydov/go-lbfgsb. (E.g., see the line numbers; in idavydov version there are no function calls at the line 341).

Could you try to make sure you are using idavydov/go-lbfgsb? Worst case, delete the directory manually or something similar.

If you still observer this error in idavydov/go-lbfgsb, could you open a ticket there with a reproducible example ideally?

[hiok2000]

Hi @hiok2000 ,

According to your error, you are using in fact afbarnard/go-lbfgsb, not idavydov/go-lbfgsb. (E.g., see the line numbers; in idavydov version there are no function calls at the line 341).

Could you try to make sure you are using idavydov/go-lbfgsb? Worst case, delete the directory manually or something similar.

If you still observer this error in idavydov/go-lbfgsb, could you open a ticket there with a reproducible example ideally?

I am sorry. You are right. I still use afbarnard/go-lbfgsb. I still got error in .f90 file in idavydov/go-lbfgsb. I replace .f90 and make file in idavydov/go-lbfgsb by .f30 and makefile afbarnard/go-lbfgsb. Everything is fine. I have get a good result using in my holtwinters forecast projects. Thank you both.