Extract media from various posting platforms like Twitter, Reddit, Pixiv, Youtube, Tiktok, Osnova and many others. This project written in Typescript for Node.js and it works as REST service for other services, e.g. Anime Ultra Bot or Social Picker Vue.
Start a server, make HTTP-request to it with a link to the post and receive that post's content back.
There are some configuration files:
service.json– service port, base URL of external service for viewing some and proxy for some platforms – see Platform specific/Proxiestokens.json– tokens for platforms and other filepaths. See Platform specific and types with commentspm2.production.json– config forpm2daemon- Other configuration files for particular platforms – see Platform specific
Development config files can be created and placed along production ones (e.g. tokens.dev.json or service.dev.json). You can also install all npm modules (including dev) one with npm install. npm run dev will run service in dev-environment, npm run lint will use eslint.
- Install necessary dependencies –
npm i --omit=dev - Install necessary binaries needed for some platforms
- Compile Typescript –
npm run compile - Start server –
npm run production
After launching you can access Picker with fetching it like curl http://localhost:8080/?url=__LINK_TO_ANY_POST__ (change 8080 to real port you specified in service.json).
If you're planning opening your Picker instance to the world, I'd suggest creating your own middleware service for handling user access, downloading combined videos, etc. (something like that). After creating such service set CUSTOM_IMG_VIEWER_SERVICE at service.json in this app pointing to that service (you can delete proxy search param from queries – see Platform specific/Proxies for more).
- Twitter – images, videos, GIFs and direct media from
*.twimg.com. More details - Pixiv – images, single direct media from
*.pximg.netand Ugoira-GIFs. More details - Reddit – images, videos, GIFs and galleries. More details
- Youtube – video with response in default type containing all streams (via
yt-dlp) - Tiktok – Any video in multiple formats. More details
- Instagram – images, videos, galleries and Reels. More details
- Osnova – images, videos, GIFs and galleries. More details
- Coub – looped videos with linear audio. More details
- Joyreactor – images and GIFs from multiple subdomains and direct links to media files. More details
- Tumblr – images and galleries. More details
- Kemono – images and galleries. More details
- Danbooru – images (unstable)
- Gelbooru – images (unstable)
- Konachan – images (unstable)
- Yandere – images (unstable)
- Eshuushuu – images (unstable)
- Sankaku – images (unstable)
- Zerochan – images (unstable)
- AnimePictures – images (unstable)
Some platform modules utilizes binary executable, here is the list:
ffmpeg– for media stream merging and convering. Follow instructions on the official site. After installing make sure you added its location to$PATHgallery-dl– Twitter specific. Download or install release for your platform, make it executable and ensure its location is in the$PATHyt-dlp– for picking some video-platforms. Download built release for your platform, make it executable and ensure its location is in the$PATH
Some platforms have very harsh limitation of allowed IP addresses/AS. Therefore service.json contains two fields for setting up host and port of SOCKS5 proxy to some remote server (PROXY_HOSTNAME and PROXY_PORT); you can set them to null if accessing such platforms from your IP address is not being limited. For the same reason base URL of external service hash template for proxy flag in its search query (see Open access).
Supports images, videos, GIFs (temporary unstable) and direct media from *.twimg.com. Uses gallery-dl for downloading media from Twitter/X.com.
To use this platform:
- Install this browser extension – it's open source and it supports Chrome and Firefox
- Go to x.com
- Copy cookies (click
Copybutton in extension popup) - Create file
config/twitter.cookies.txtand paste there contents from clipboard (it starts with"# Netscape HTTP Cookie File") - Check and edit
tokens.json– give a look to propertyTWITTER_COOKIE_FILE_LOCATION: set path relative to project's root, default one is in accordance to this example (or simply don't change default value).
Supports images, single direct media from *.pximg.net and Ugoira-GIFs.
Uses external service for end-user viewing of high-res images due to Referer Header issues. Uses ugoira-builder for creating mp4 video from Ugoira ZIP with PNGs (via ffmpeg).
Supports images, videos, GIFs and galleries. Because Reddit's videos are divided from audios, this app utilizes video-audio-merge for merging separated streams via ffmpeg.
Requires setup of the site's cookies in tokens.json
Any video in multiple formats. Does not support image galleries.
Uses yt-dlp for extracting metadata and then video-codec-convert to convert original highest quality HEVC video without watermark to H264 file (via ffmpeg) for compatibility. Still prioritizes HEVC one.
Supports images, videos, galleries and Reels.
To pick posts and Reels:
- Install this browser extension – it's open source and it supports Chrome and Firefox
- Go to instagram.com
- Copy cookies (click
Copybutton in extension popup) - Create file
config/instagram.cookies.txtand paste there contents from clipboard (it starts with"# Netscape HTTP Cookie File") - Check and edit
tokens.json– give a look to propertyINSTAGRAM_COOKIE_FILE_LOCATION: set path relative to project's root, default one is in accordance to this example (or simply don't change default value).
Supports images, videos, GIFs and galleries. Also extracts Twitter and Instagram blocks/links from within and handles them with parsers above.
Creates looped videos with linear audio, limits itself only with audio length and filesize if it ever exceed a reasonable size (20 MB). Uses video-audio-merge for merging separated streams (via ffmpeg).
Supports images, videos, GIFs and galleries, requires setup of the site's cookies in tokens.json.
Supports images and galleries, set OAuth keys and stuff in tokens.json.
Supports images and galleries, requires setup of the site's cookies in tokens.json.
- Telegram bot based on this service
- Frontend client built for this service
- ffmpeg docs
- yt-dlp docs
- Tumblr API
- Osnova API
- What is Pixiv's Ugoira