File upload method

[cl-bvl]

Summary

Checklist

Delete items not relevant to your PR:

• Unit and integration tests covering the common scenarios were added
• A human-readable description of the changes was provided to include in CHANGELOG
• For significant changes, documentation in https://github.com/ClickHouse/clickhouse-docs was updated with further explanations or tutorials

[CLAassistant]

All committers have signed the CLA.

[kavirajk]

Hi @cl-bvl thanks for the PR :)

Wondering if we should make this InsertFile api more generic? Like instead of taking the path, why not some io.Reader. Rationale for this is then file can be anywhere (in-memory, disk, or s3?). I know we may need to collect additional metadata like encoding, I don't think that should be a blocker, given the flexibility of the api.

anythoughts?

[cl-bvl]

Hello.

Wondering if we should make this InsertFile api more generic? Like instead of taking the path, why not some io.Reader. Rationale for this is then file can be anywhere (in-memory, disk, or s3?). I know we may need to collect additional metadata like encoding, I don't think that should be a blocker, given the flexibility of the api.

We can do 2 methods - generic with io.Reader and wrapper for file insert. Something like this:

func (h *httpConnect) insertReader(ctx context.Context, reader io.Reader, encoding string, query string) error {
....
}

func (h *httpConnect) insertFile(ctx context.Context, filePath string, query string) error {
        f, err := os.OpenFile(filePath, os.O_RDONLY, 0644)
	if err != nil {
		return err
	}
	defer f.Close()

        if options.fileEncoding == "" {
		options.fileEncoding = encodingFromFileName(f.Name())
	}

        return insertReader(ctx, f, options.fileEncoding)
}

Is it looking good for you? If yes i will make it.

[kavirajk]

We can do 2 methods - generic with io.Reader and wrapper for file insert. Something like this:

Curious why 2 methods? If we expose two methods we need to maintain those 2 methods forever. I'm thinking just io.Reader based method and accept other metadata like encoding, compression, etc directly from the user (as mandatory arguments).

Would that work?

[cl-bvl]

Done

[cl-bvl]

Hello.
Can you please review the PR and merge it if everything ok.

[kavirajk]

@cl-bvl Thanks for being patient :)

We definitely need this feature. But I'm currently thinking about solving in a way that also fixes #1700. Often people asks support for import/export in arbitrary format that ClickHouse server supports.

I was busy with some release work. I will share my full review and thoughts in a day or two.

Thanks again for the PR.

[cl-bvl]

Hello.

#1700 — it looks like we need a addition method, for example, LoadFile(w io.Writer) .....
Which would work similarly.

And I can even prepare a PR with the necessary changes, but as a separate PR.

[kavirajk]

Update:
@cl-bvl you are right. We can take INSERT and SELECT as different PRs.

It's more of having API design that works consistently across different dimensions of Go client (e.g: Normal Execs and Batching on both Native and std interfaces supporting both TCP and HTTP)

I'm writing a small rfc. Will shared it soon.