Replace CGO with purego

[ma6174]

purego project: https://github.com/ebitengine/purego

Benefits:

• Compilation doesn't require CGO, making cross-compilation much easier, fix Documentation on cross-compilation #279
• go-duckdb and duckdb shared libraries can be decoupled, allowing users to use their own compiled shared libraries, fix Statically Iink 'icu' extension by default #323
• go-duckdb can continue to provide pre-compiled shared libraries, embedded into the binary through go embed package, making it convenient for users. If use the embed package, don't need to put all the shared libraries of all platforms into the same binary. can use build constraints to only put the shared libraries of your own platform.

Problem

If using embed to package the shared library into the binary, it needs to be extracted to a temporary directory before loading.
Related issue: ebitengine/purego#102

Simple example

1. Download and extract the shared library: https://duckdb.org/docs/installation/?version=stable&environment=cplusplus&platform=linux&download_method=direct&architecture=x86_64
2. code example:

package main

import (
	"fmt"
	"log"

	"github.com/ebitengine/purego"
)

type (
	duckdb_database   uintptr
	duckdb_connection uintptr
	duckdb_result     uintptr
	duckdb_state      int
)

var (
	duckdb_library_version func() string
	duckdb_open            func(string, *duckdb_database) duckdb_state
	duckdb_connect         func(duckdb_database, *duckdb_connection) duckdb_state
	duckdb_query           func(duckdb_connection, string, *duckdb_result) duckdb_state
)

var query = `
COPY (

	select version() as version

) TO '/dev/stdout' (FORMAT JSON);
`

func main() {
	duckdb, err := purego.Dlopen("./libduckdb.so", purego.RTLD_NOW|purego.RTLD_GLOBAL)
	if err != nil {
		log.Panicln(err)
	}
	// duckdb_library_version
	purego.RegisterLibFunc(&duckdb_library_version, duckdb, "duckdb_library_version")
	fmt.Println("duckdb_library_version", duckdb_library_version())

	// duckdb_open
	purego.RegisterLibFunc(&duckdb_open, duckdb, "duckdb_open")
	var db duckdb_database
	stat := duckdb_open("", &db)
	fmt.Println("duckdb_open result", stat)

	// duckdb_connect
	purego.RegisterLibFunc(&duckdb_connect, duckdb, "duckdb_connect")
	var conn duckdb_connection
	stat = duckdb_connect(db, &conn)
	fmt.Println("duckdb_connect result", stat)

	// duckdb_query
	purego.RegisterLibFunc(&duckdb_query, duckdb, "duckdb_query")
	var result duckdb_result
	stat = duckdb_query(conn, query, &result)
	fmt.Println("duckdb_query result", stat)
}

3. build and run

$ GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build
$ ./purego
duckdb_library_version v1.1.3
duckdb_open result 0
duckdb_connect result 0
{"version":"v1.1.3"}
duckdb_query result 0

[taniabogatsch]

go-duckdb can continue to provide pre-compiled shared libraries, embedded into the binary through go embed package, making it convenient for users. If use the embed package, don't need to put all the shared libraries of all platforms into the same binary. can use build constraints to only put the shared libraries of your own platform.

Hi @ma6174, I've been trying to get this part of your discussion to work. Could you elaborate a bit more? How do I only pull in the necessary binaries, especially if I want to remove them from the go-duckdb repository to avoid file size limitations?

[ma6174]

just check this repositoriy: https://github.com/ma6174/duckdb-test-lib

1. The duckdb shared libraries can be compressed to reduce size, just decompress when needed. or split into multiple segments to avoid file size limitations, and combine when using.
2. The libduckdb package should also put into another repositoriy to reduce the go-duckdb repositoriy size.
3. The libduckdb version number can follows the duckdb main repository. user can easily switch to other duckdb version.
4. If user want to use custom shared libraries, not import libduckdb package, and set their own library path here https://github.com/ma6174/duckdb-test-lib/blob/develop/main.go#L16

[mlafeldt]

👋 Thanks for starting this discussion!

This repository has moved to https://github.com/duckdb/duckdb-go

If this topic is still relevant, please continue the conversation at the new location. I'm closing this now as part of the migration.

Please see #565 for more details.