Import / export all data

[tckb]

Prerequisites

• I have searched open and closed issues to make sure that the feature has not yet been requested.

Similar features were requested in #210 and #142 asking about importing data into Plausible.

Feature request

The Idea of this feature is that we need to enable an easy import of events into Plausible from the external vendors such as GA or other analytic solutions. To make an external import easier I propose the following --

Define Import Data format

Since, the idea is to import only analytics data, i.e, the event data it is possible to import this directly into Clickhouse. Clickhouse already suppports data import via CSV (related docs here). We can define any additional fields required in the format and enable import.

Exporter plugins/scripts

Since, we now have a "pre-defined" format of data to be imported, we can now start writing exporters for different vendors or different use-cases:

• Import from GA is possible for by downloading GA data and exporting it into plausible format and then importing it. This can achieved completely in the UI
• For any other non-supported vendor, one can run the exporter and then import it.
• The same way it is possible to import analytics data from server-side logs

Update: We now have the CSV export and the stats API too.

Update 2: You can now import your historical Google Analytics (Universal Analytics and GA4) stats into your Plausible dashboard. See more here.

Update 3: You can now import external data into Plausible using CSV files. The CSV import supports Plausible data by default but can also be used to import data from other sources if the external data is formatted according to our guidelines. See the full details: https://plausible.io/docs/csv-import

[ukutaht]

Agreed, defining an import spec would make things easier for sure. Couple thoughts:

1. A super important use-case here is moving from Plausible Cloud to self-hosted and back. We don't want to keep anyone hostage. Making it easy to go from hosted to self-hosted would give customers more ownership of their data.
2. I haven't looked into what's offered but I doubt most tools will export event-level data. I'm guessing the exports are aggregated to daily or monthly stats. This creates a problem in our schema as currently all stats are done based on event-level data. We might need special schema design for 'imported aggregated stats'.

[tckb]

super important use-case here is moving ...

this is a very legit use-case. I totally see this coming and this can addressed by keeping a consistent data format (spec) for both exporting and importing data. This keeps the implementation simple.

I'm guessing the exports are aggregated to daily or monthly stats...This creates a problem in our schema

You are right. Free Plan on GA provides only aggregated stats. Only premium plan members are able to extract full raw event data.

we might need special schema design for 'imported aggregated stats'.

I think there should be a way to import aggregated data and "spread" this data (of course this factually will not be true data).

as a first step, I believe we need to get the data format (spec) prepared.

[MaybeThisIsRu]

• A super important use-case here is moving from Plausible Cloud to self-hosted and back. We don't want to keep anyone hostage. Making it easy to go from hosted to self-hosted would give customers more ownership of their data.

Is there any active interest in this at the moment?

[vinodbollini]

Really important and urgent for us now. I do feel like we are being held hostage, because we can't do this, and we will loose our accumulated data if just switch-and-ditch.
Not sure if this is not being solved because it will hurt your cloud revenue. But is there a way to make this lucrative enough?

[dewey]

@vinodbollini The feature didn't exist when you signed up or started using it, so you are not being held hostage. You just assumed it will arrive soon, and it didn't.

[vinodbollini]

@dewey you are right, we are not entitled to it automatically. But it is important to us.

3 things:

1. We were also not expecting that we would need it so soon - nobody's problem but ours.
2. We were also not expecting to have data losses while paying for the cloud version. (Ref: Three hours of data missing from the day's report #1976)
3. This raises our hopes: https://plausible.io/data-policy#data-ownership-of-your-web-analytics

What would you have us do?

[johanrosenson]

@vinodbollini what did you end up doing? did you find a good solution for your use case?

[vinodbollini]

Did not find a solution. We just forked our data collection to a self-hosted instance for now - because at least data losses are a problem we can deal with ourselves.
PS: We still pay Plausible for a cloud-hosting plan, even while sending no data there, because we want to support them.

[ukutaht]

Not that I'm aware of

[vincentdoerig]

Hi there. In order to write a migration tool, the data schema for the various other analytics platforms have to be known to the team.

I'll start off with the schema from Simple Analytics. Since I can't directly upload CSVs here, I have uploaded it as a gist: https://gist.github.com/vincentdoerig/7b108450b46c8ad2de433f7cdd47486b

The data is cleaned up from a personal site, everything should be pretty self-explanatory.

// cc @metmarkosaric

[dewey]

Hey, adding to this I have the schema for http://get.gaug.es - they don't offer an export but I wrote a tool today (https://github.com/dewey/gaug.es-analytics-export) to export the daily aggregated data. In this case I use the same format they return it in the API but it could be adapted to support whatever "aggregated events" schema we come up with.

In this case the format is just:

date,views,people
2020-07-11,11,7
2020-07-12,21,15

[JoaaoVerona]

+1. Would be much appreciated if we could import data from Google Analytics (just like Matomo does too).

[telmiger]

+1 As mentioned in the forum I am interested in exporting features mostly (Goal Conversions in combination with campaign tracking data).

[tckb]

I haven't had a lot of time in the past months to do some work here. I will muster some time during Christmas and get something done to help this task. @ukutaht did you manage to do anything wrt to this topic?

[ukutaht]

Not yet @tckb

[Joshua-Flores]

I just moved from cloud to self hosted and it would be really cool if I could migrate my old data over.

[magicgg91]

I would really appreciate to be able to import datas from a previous Matomo installation.
it doesn't matter for me about the format, as I can transform datas if needed, but it should be great if it's a end-to-end solution.

[MaluNoPeleke]

I would also like to migrate from Matomo self-hosted to Plausible.

[Hukuma1]

Just found Plausible randomly from Reddit. Sounds amazing, but the lack of import of data would be a non-starter. Is there any server log import workflow from Apache to Plausible? Or no way at all currently?

[metmarkosaric]

there's no way to import data at this point.

[MaluNoPeleke]

If I subscribe to this discussion I should get updates about this idea, shouldn't I?

[metmarkosaric]

yes

[benui-dev]

If you are looking for more use-case information, I would really appreciate just a way to import # of visitors to each page over time.

I hope we can get this feature soon!

[felixakiragreen]

I was hoping to switch from Fathom to Plausible (because I love your Open Source philosophy & want to support 🇪🇪 businesses!) but I don't want to abandon all of my existing data or maintain 2 systems.

I'm subscribing to this issue so that when it's finished, I'll make the switch!

💚 felix

[tedivm]

Same! I have a self hosted Fathom install and I'm moving over to the hosted Plausible one, and I really would love to be able to import the data from Fathom.

[w4rner]

FYI Eric, [Simple Analytics](https://referral.simpleanalytics.com/w4rner) have that feature. You can export back to Plausible once these guys have built it ;)

…

------- Original Message -------

On Friday, July 8th, 2022 at 7:05 PM, Eric A Stalee 'notifications at github.com' ***@***.***> wrote: For the hosted version is there no way to truly export all data from a site as a (non-[limited](https://plausible.io/docs/export-stats)) CSV even though the data is just sitting somewhere in a database anyways? The API seems quite weak and limited for analytics purposes and we need to retrieve multiple metrics across multiple dimensions (which includes hundreds of pages). Self-hosting seems like it would have been the better option given how closed off data access appears to be at first glance but I don't make those decisions. — Reply to this email directly, [view it on GitHub](#211 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AHP6SEQK7GONZJFGP6XATITVTBUXHANCNFSM4V7JZ7JA). You are receiving this because you were mentioned.Message ID: ***@***.***>

[fprochazka]

I've recently started using Plausible, and I'm considering converting and dropping google analytics, but I don't want to lose the years and years of data I already have there.

Just to give you some additional motivation: I'd be willing to pay a reasonable one-time fee for the import if it were sufficiently automated and I could rely on it to work correctly.

// edit: I've finally found https://plausible.io/docs/google-analytics-import and it's disabled 😭

[metmarkosaric]

Hi @fprochazka! We had such high demand for Google Analytics imports that it negatively affected the stability of the rest of our services so we had to disable the functionality temporarily. Looks like many people want to leave Google Analytics and want to leave fast! Our developers are doing final tests on the next version of our GA import so it should make its return very soon (hopefully next week)! Stay tuned!

[d19dotca]

Looks like many people want to leave Google Analytics and want to leave fast! Our developers are doing final tests on the next version of our GA import so it should make its return very soon (hopefully next week)! Stay tuned!

What about importing from other popular analytics services such as Matomo? Is that coming anytime soon?

[metmarkosaric]

Hi @d19dotca! No short term plans for imports from other analytics services unfortunately.

[lightson]

should make its return very soon (hopefully next week)! Stay tuned!

This is very exciting @metmarkosaric, I was going to ask if there were any plans to bring import back just for self hosted as it's less likely to cause issues one our own servers due to lower concurrent usage - but sounds like that's moot if it's making a come back anyway!

[metmarkosaric]

Google Analytics import is now back!

https://plausible.io/docs/google-analytics-import

[metmarkosaric]

Hi @rutger1140! This is not yet available on self-hosted. It will be part of the next release. Our self-hosted code is not updated as frequently as the Plausible managed hosting in the cloud, which changes daily. This means that the latest features won't be immediately available.

Think of self-hosted as a long-term release. We stack all the new features and fixes that have been battle-tested on the hosted version and release a new self-hosted version twice a year or so.

We don't have any more specific timelines on when the next release will be done. As soon as it's ready, it will be posted on GitHub.

[rutger1140]

Good to hear! Nice progress 💪
I see the code in the repository is updated, can I build my own docker container using the latest master branch? Or is there another way to run 'nightlies'?

[metmarkosaric]

you could try if you want but it's not meant for self hosting so might break very often which is why we don't recommend it

[skwee357]

Hey! Thanks for working on this. Do you have any prediction when the docker will be updated to support import in self-hosted as well? Thanks!

[lightson]

@skwee357 it's in the latest release candidate, worked great for me. See #2324

[chunkydotdev]

Would love the feature of importing/exporting between self hosted and the cloud solution, is there any news on this so far? 🙏

[eduardopuente]

@metmarkosaric we would like to migrate to the cloud version. Is there a timeline you can share for this feature?

[metmarkosaric]

@eduardopuente no updates on this at the moment unfortunately. currently it's not possible to move data between self-hosted and cloud (and vice versa). we're hoping to be able to start the work on this at some point later this year but no promises. we'll update this thread when there's news

[artfulrobot]

+1 subscribe

Being able to access and move data - between different plausible hosts - is a key part of data sovereignty. And super useful.

[hanscees]

Hi its june 2023 now.

Is there a way to import apache or nginx server logging for self-hosting?

Perhaps by using a csv import?
Or by feeding loglines one-by-one into the api? (I have time )

thanks
Hans-Cees

[mhanley00]

@ruslandoga that is fantastic news, thanks for sharing these links! What is the timeline for getting this feature to production and to self-hosted?

[ruslandoga]

@mhanley00 no timeline yet, but I'll post in this thread once it's deployed :)

[mhanley00]

Beautiful, thank you so much @ruslandoga ! ❤️

[julianengel]

Just commenting to get notified once that's available :) Thank you!

[ruslandoga]

👋

v2.1.0-rc.1 includes CSV exports and imports: #4084

[kirschnerio]

@metmarkosaric Any chance, that you could use this gist to get a matomo import to plausible up and running? See: https://gist.github.com/ampaze/3a22d3c2adbc1d0bbeff409966ccd132

We are using plausible cloud version since your first days. Unfortunately we moved from Google Analytics to Matomo Self hosted before Plausible was there and all data from GA was imported to Matomo and for privacy reasons deleted from GA. Now we have all old data in Matomo and we would love to have them in our beloved place: Plausible. As mentioned before: GA import doesn't help for us. I would predict, that there are many customers like us, who would love to import old data from Matomo.

[metmarkosaric]

@kirschnerio we're about to release a CSV import functionality in the next couple of weeks if it all goes well. It's mostly meant for Plausible cloud to self hosted and vice versa but I think it should be possible to make it work for other platforms too. Stay tuned!

[kirschnerio]

@metmarkosaric This sounds great, thanks! If there would be a simple mapping tool integrated, where you could map fields, this should help for a plethora of third party analytics to plausible import scenarios.

[julianengel]

We had to migrate our Plausible CE instance to a new server because of a corruption that happened on the original one. In the process, this thread came up a lot of times. While we didn't face this exact issue, we needed to export all data from one instance and then import it into another. Took a lot of time, but I wrote everything down in case anyone faces the same issue.

Migrating Plausible Data Between Instances/Servers

Thank you @metmarkosaric for the awesome project!

[Tony-MF]

Thanks a lot! I'll test it this week. Just to be clear, both the old and new instance were v2.1.0-rc.0 or v2.0.0?

[julianengel]

Both were on v2.0.0, didn’t get around to test the rc yet :)

[metmarkosaric]

Update: The GA4 import is ready! You can see how it works here: https://plausible.io/docs/google-analytics-import

It’s now possible to import both UA and GA4 properties into the same Plausible dashboard. We’ll automatically detect time ranges to prevent double counting of periods and to avoid any gaps in the data.

The same considerations apply to the GA4 import as to the UA import: Imported data is less functional than native data. It cannot be filtered, exported and some metrics may be missing due to limitations with the Google APIs. More details in the link above.

If you encounter any issues with the import, do contact our support please: https://plausible.io/contact

We’ll now continue working to further enhance the import feature. You can follow the progress in these threads:

1. Allow import of custom events too https://feedback.plausible.io/225 (this will require you to import the same properties once again)
2. Make imported data filterable https://feedback.plausible.io/200
3. Allow export of imported data using our regular CSV export and stats API https://feedback.plausible.io/215

[MaluNoPeleke]

Does this also work with Matomo export?

[metmarkosaric]

Update: You can now import external data into Plausible using CSV files. The CSV import supports Plausible data by default but can also be used to import data from other sources if the external data is formatted according to our guidelines. See the full details: https://plausible.io/docs/csv-import

[sandros94]

Would it be possible to export all data, including the Google Analytics UA data? Since starting from tomorrow it will not be possible to reimport UA data into a new instance (in particular when migrating from CE to EE and vice versa)

[ruslandoga]

👋 @sandros94

Right now only "native" data is exported via the CSV method.

[sandros94]

Right now only "native" data is exported via the CSV method.

Yup, docs state that pretty clearly.
My current concern is how should I handle the backup and restore of such non-native data, since starting from tomorrow we are not able to restore it from the original source (making a switch from CE to EE and vice versa quite hard).

An option could be creating a separate export for each import, with meta-tags to let Plausible understand from what source they were originally from. Like Export to CSV alongside the Delete Import button.

What parts of the source code should be altered to support this?

[ruslandoga]

My current concern is how should I handle the backup and restore of such non-native data, since starting from tomorrow we are not able to restore it from the original source (making a switch from CE to EE and vice versa quite hard).

You can backup all imported_* tables in plausible_events_db database (this is the name of the database if you use the default https://github.com/plausible/community-edition setup).

What parts of the source code should be altered to support this?

This would probably involve changing CSV exporter to add a new set of files for "exported imports" and CSV importer to support "importing multiple imports".

Please don't submit any PRs yet :) First, I'll need to ask the core team if that feature is planned.

[saeedesmaili]

Are all these import and export features limited to the cloud offering and not the community edition?

[ruslandoga]

👋 @saeedesmaili

Export / import for CSV data is available in CE since v2.1.0. Currently, it should be roughly the same functionality as in the Cloud. There might have been some improvements around filtering imported data in the dashboards that haven't been released yet in CE, however.

[saeedesmaili]

Ah, my bad. I'm using the analytics v2.0 and just found out about the release of the separate community edition 2.1.0. I need to upgrade my instance, thanks.

[Lucent]

Do you ever plan to allow download of the full hosted Clickhouse database? I would pay any reasonable fee to do this and understand it would be gigabytes, but it's the only true way to migrate to local and maintain existing precision and filtering.