diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 2668242b7..073d0fb14 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -4,7 +4,7 @@ if you're reading this, you are probably interested in contributing to cobalt, w this document serves as a guide to help you make contributions that we can merge into the cobalt codebase. ## translations -currently, we are **not accepting** translations of cobalt. this is because we are making significant changes to the frontend, and the currently used localization structure is being completely reworked. if this changes, this document will be updated. +currently, we are **not accepting** translations of cobalt. we're working on changing this soon! ## adding features or support for services before putting in the effort to implement a feature, it's worth considering whether it would be appropriate to add it to cobalt. the cobalt api is built to assist people **only with downloading freely accessible content**. other functionality, such as: @@ -22,9 +22,9 @@ when contributing code to cobalt, there are a few guidelines in place to ensure ### clean commit messages internally, we use a format similar to [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) - the first part signifies which part of the code you are changing (the *scope*), and the second part explains the change. for inspiration on how to write appropriate commit titles, you can take a look at the [commit history](https://github.com/imputnet/cobalt/commits/). -the scope is not strictly defined, you can write whatever you find most fitting for the particular change. suppose you are changing a small part of a more significant part of the codebase. in that case, you can specify both the larger and smaller scopes in the commit message for clarity (e.g., if you were changing something in internal streams, the commit could be something like `stream/internal: fix object not being handled properly`). +the scope is not strictly defined, you can write whatever you find most fitting for the particular change. suppose you are changing a small part of a more significant part of the codebase. in that case, you can specify both the larger and smaller scopes in the commit message for clarity (e.g., if you were changing something in internal streams, the commit could be something like `api/stream: fix object not being handled properly`). -if you think a change deserves further explanation, we encourage you to write a short explanation in the commit message ([example](https://github.com/imputnet/cobalt/commit/d2e5b6542f71f3809ba94d56c26f382b5cb62762)), which will save both you and us time having to enquire about the change, and you explaining the reason behind it. +if you think a change deserves further explanation, we encourage you to write a short explanation in the commit message ([example](https://github.com/imputnet/cobalt/commit/31be60484de8eaf63bba8a4f508e16438aa7ba6e)), which will save both you and us time having to enquire about the change, and you explaining the reason behind it. if your contribution has uninformative commit titles, you may be asked to interactively rebase your branch and amend each commit to include a meaningful title. diff --git a/README.md b/README.md index 5bf8ae8e3..434b5943c 100644 --- a/README.md +++ b/README.md @@ -15,108 +15,43 @@ 💬 community discord server - 🐦 twitter/x + 🐦 twitter


-cobalt is a media downloader that doesn't piss you off. it's fast, friendly, and doesn't have any bullshit that modern web is filled with: ***no ads, trackers, or paywalls***. +cobalt is a media downloader that doesn't piss you off. it's friendly, efficient, and doesn't have ads, trackers, paywalls or other nonsense. -paste the link, get the file, move on. it's that simple. just how it should be. +paste the link, get the file, move on. that simple, just how it should be. -### supported services -this list is not final and keeps expanding over time. if support for a service you want is missing, create an issue (or a pull request 👀). +### cobalt monorepo +this monorepo includes source code for api, frontend, and related packages: +- [api tree](/api/) +- [web tree](/web/) +- [packages tree](/packages/) -| service | video + audio | only audio | only video | metadata | rich file names | -| :-------- | :-----------: | :--------: | :--------: | :------: | :-------------: | -| bilibili | ✅ | ✅ | ✅ | ➖ | ➖ | -| bluesky | ✅ | ✅ | ✅ | ➖ | ➖ | -| dailymotion | ✅ | ✅ | ✅ | ✅ | ✅ | -| instagram | ✅ | ✅ | ✅ | ➖ | ➖ | -| facebook | ✅ | ❌ | ✅ | ➖ | ➖ | -| loom | ✅ | ❌ | ✅ | ✅ | ➖ | -| ok.ru | ✅ | ❌ | ✅ | ✅ | ✅ | -| pinterest | ✅ | ✅ | ✅ | ➖ | ➖ | -| reddit | ✅ | ✅ | ✅ | ❌ | ❌ | -| rutube | ✅ | ✅ | ✅ | ✅ | ✅ | -| snapchat | ✅ | ✅ | ✅ | ➖ | ➖ | -| soundcloud | ➖ | ✅ | ➖ | ✅ | ✅ | -| streamable | ✅ | ✅ | ✅ | ➖ | ➖ | -| tiktok | ✅ | ✅ | ✅ | ❌ | ❌ | -| tumblr | ✅ | ✅ | ✅ | ➖ | ➖ | -| twitch clips | ✅ | ✅ | ✅ | ✅ | ✅ | -| twitter/x | ✅ | ✅ | ✅ | ➖ | ➖ | -| vimeo | ✅ | ✅ | ✅ | ✅ | ✅ | -| vine | ✅ | ✅ | ✅ | ➖ | ➖ | -| vk videos & clips | ✅ | ❌ | ✅ | ✅ | ✅ | -| youtube | ✅ | ✅ | ✅ | ✅ | ✅ | - -| emoji | meaning | -| :-----: | :---------------------- | -| ✅ | supported | -| ➖ | impossible/unreasonable | -| ❌ | not supported | - -### additional notes or features (per service) -| service | notes or features | -| :-------- | :----- | -| instagram | supports reels, photos, and videos. lets you pick what to save from multi-media posts. | -| facebook | supports public accessible videos content only. | -| pinterest | supports photos, gifs, videos and stories. | -| reddit | supports gifs and videos. | -| snapchat | supports spotlights and stories. lets you pick what to save from stories. | -| rutube | supports yappy & private links. | -| soundcloud | supports private links. | -| tiktok | supports videos with or without watermark, images from slideshow without watermark, and full (original) audios. | -| twitter/x | lets you pick what to save from multi-media posts. may not be 100% reliable due to current management. | -| vimeo | audio downloads are only available for dash. | -| youtube | supports videos, music, and shorts. 8K, 4K, HDR, VR, and high FPS videos. rich metadata & dubs. h264/av1/vp9 codecs. | +it also includes documentation in the [docs tree](/docs/): +- [cobalt api documentation](/docs/api.md) +- [how to run a cobalt instance](/docs/run-an-instance.md) +- [how to protect a cobalt instance](/docs/protect-an-instance.md) +- [how to configure a cobalt instance for youtube](/docs/configure-for-youtube.md) ### partners -cobalt is sponsored by [royalehosting.net](https://royalehosting.net/?partner=cobalt), all main instances are currently hosted on their network :) +cobalt is sponsored by [royalehosting.net](https://royalehosting.net/?partner=cobalt) and the main processing instance is hosted on their network. we really appreciate their kindness! + +### ethics +cobalt is a tool that makes downloading public content easier. it takes **zero liability**. +the end user is responsible for what they download, how they use and distribute that content. +cobalt never caches any content, it [works like a fancy proxy](/api/src/stream/). -### ethics and disclaimer -cobalt is a tool for easing content downloads from internet and takes ***zero liability***. you are responsible for what you download, how you use and distribute that content. please be mindful when using content of others and always credit original creators. fair use and credits benefit everyone. +cobalt is in no way a piracy tool and cannot be used as such. +it can only download free & publicly accessible content. +same content can be downloaded via dev tools of any modern web browser. -cobalt is ***NOT*** a piracy tool and cannot be used as such. it can only download free, publicly accessible content. such content can be easily downloaded through any browser's dev tools. pressing one button is easier, so i made a convenient, ad-less tool for such repeated actions. +### contributing +thank you for considering making a contribution to cobalt! please check the [contributing guidelines here](/CONTRIBUTING.md) before making a pull request. -### cobalt license +### licenses for relevant licensing information, see the [api](api/README.md) and [web](web/README.md) READMEs. unless specified otherwise, the remainder of this repository is licensed under [AGPL-3.0](LICENSE). - -## acknowledgements -### ffmpeg -cobalt heavily relies on ffmpeg for converting and merging media files. it's an absolutely amazing piece of software offered for anyone for free, yet doesn't receive as much credit as it should. - -you can [support ffmpeg here](https://ffmpeg.org/donations.html)! - -#### ffmpeg-static -we use [ffmpeg-static](https://github.com/eugeneware/ffmpeg-static) to get binaries for ffmpeg depending on the platform. - -you can support the developer via various methods listed on their github page! (linked above) - -### youtube.js -cobalt relies on [youtube.js](https://github.com/LuanRT/YouTube.js) for interacting with the innertube api, it wouldn't have been possible without it. - -you can support the developer via various methods listed on their github page! (linked above) - -### many others -cobalt also depends on: - -- [content-disposition-header](https://www.npmjs.com/package/content-disposition-header) to simplify the provision of `content-disposition` headers. -- [cors](https://www.npmjs.com/package/cors) to manage cross-origin resource sharing within expressjs. -- [dotenv](https://www.npmjs.com/package/dotenv) to load environment variables from the `.env` file. -- [esbuild](https://www.npmjs.com/package/esbuild) to minify the frontend files. -- [express](https://www.npmjs.com/package/express) as the backbone of cobalt servers. -- [express-rate-limit](https://www.npmjs.com/package/express-rate-limit) to rate limit api endpoints. -- [hls-parser](https://www.npmjs.com/package/hls-parser) to parse `m3u8` playlists for certain services. -- [ipaddr.js](https://www.npmjs.com/package/ipaddr.js) to parse ip addresses (for rate limiting). -- [nanoid](https://www.npmjs.com/package/nanoid) to generate unique (temporary) identifiers for each requested stream. -- [node-cache](https://www.npmjs.com/package/node-cache) to cache stream info in server ram for a limited amount of time. -- [psl](https://www.npmjs.com/package/psl) as the domain name parser. -- [set-cookie-parser](https://www.npmjs.com/package/set-cookie-parser) to parse cookies that cobalt receives from certain services. -- [undici](https://www.npmjs.com/package/undici) for making http requests. -- [url-pattern](https://www.npmjs.com/package/url-pattern) to match provided links with supported patterns. - -...and many other packages that these packages rely on. diff --git a/api/README.md b/api/README.md index 5c281246f..381046260 100644 --- a/api/README.md +++ b/api/README.md @@ -1,4 +1,67 @@ # cobalt api +this directory includes the source code for cobalt api. it's made with [express.js](https://www.npmjs.com/package/express) and love! + +## running your own instance +if you want to run your own instance for whatever purpose, [follow this guide](/docs/run-an-instance.md). +we recommend to use docker compose unless you intend to run cobalt for developing/debugging purposes. + +## accessing the api +there is currently no publicly available pre-hosted api. +we recommend [deploying your own instance](/docs/run-an-instance.md) if you wish to use the cobalt api. + +you can read [the api documentation here](/docs/api.md). + +> [!WARNING] +> the v7 public api (/api/json) will be shut down on **november 11th, 2024**. +> you can access documentation for it [here](https://github.com/imputnet/cobalt/blob/7/docs/api.md). + +## supported services +this list is not final and keeps expanding over time. if support for a service you want is missing, create an issue (or a pull request 👀). + +| service | video + audio | only audio | only video | metadata | rich file names | +| :-------- | :-----------: | :--------: | :--------: | :------: | :-------------: | +| bilibili | ✅ | ✅ | ✅ | ➖ | ➖ | +| bluesky | ✅ | ✅ | ✅ | ➖ | ➖ | +| dailymotion | ✅ | ✅ | ✅ | ✅ | ✅ | +| instagram | ✅ | ✅ | ✅ | ➖ | ➖ | +| facebook | ✅ | ❌ | ✅ | ➖ | ➖ | +| loom | ✅ | ❌ | ✅ | ✅ | ➖ | +| ok.ru | ✅ | ❌ | ✅ | ✅ | ✅ | +| pinterest | ✅ | ✅ | ✅ | ➖ | ➖ | +| reddit | ✅ | ✅ | ✅ | ❌ | ❌ | +| rutube | ✅ | ✅ | ✅ | ✅ | ✅ | +| snapchat | ✅ | ✅ | ✅ | ➖ | ➖ | +| soundcloud | ➖ | ✅ | ➖ | ✅ | ✅ | +| streamable | ✅ | ✅ | ✅ | ➖ | ➖ | +| tiktok | ✅ | ✅ | ✅ | ❌ | ❌ | +| tumblr | ✅ | ✅ | ✅ | ➖ | ➖ | +| twitch clips | ✅ | ✅ | ✅ | ✅ | ✅ | +| twitter/x | ✅ | ✅ | ✅ | ➖ | ➖ | +| vimeo | ✅ | ✅ | ✅ | ✅ | ✅ | +| vine | ✅ | ✅ | ✅ | ➖ | ➖ | +| vk videos & clips | ✅ | ❌ | ✅ | ✅ | ✅ | +| youtube | ✅ | ✅ | ✅ | ✅ | ✅ | + +| emoji | meaning | +| :-----: | :---------------------- | +| ✅ | supported | +| ➖ | impossible/unreasonable | +| ❌ | not supported | + +### additional notes or features (per service) +| service | notes or features | +| :-------- | :----- | +| instagram | supports reels, photos, and videos. lets you pick what to save from multi-media posts. | +| facebook | supports public accessible videos content only. | +| pinterest | supports photos, gifs, videos and stories. | +| reddit | supports gifs and videos. | +| snapchat | supports spotlights and stories. lets you pick what to save from stories. | +| rutube | supports yappy & private links. | +| soundcloud | supports private links. | +| tiktok | supports videos with or without watermark, images from slideshow without watermark, and full (original) audios. | +| twitter/x | lets you pick what to save from multi-media posts. may not be 100% reliable due to current management. | +| vimeo | audio downloads are only available for dash. | +| youtube | supports videos, music, and shorts. 8K, 4K, HDR, VR, and high FPS videos. rich metadata & dubs. h264/av1/vp9 codecs. | ## license cobalt api code is licensed under [AGPL-3.0](LICENSE). @@ -9,14 +72,38 @@ as long as you: - provide a link to the license and indicate if changes to the code were made, and - release the code under the **same license** -## running your own instance -if you want to run your own instance for whatever purpose, [follow this guide](/docs/run-an-instance.md). -it's *highly* recommended to use a docker compose method unless you run for developing/debugging purposes. +## acknowledgements +### ffmpeg +cobalt heavily relies on ffmpeg for converting and merging media files. it's an absolutely amazing piece of software offered for anyone for free, yet doesn't receive as much credit as it should. -## accessing the api -currently, there is no publicly accessible main api. we plan on providing a public api for -cobalt 10 in some form in the future. we recommend deploying your own instance if you wish -to use the latest api. you can access [the documentation](/docs/api.md) for it here. +you can [support ffmpeg here](https://ffmpeg.org/donations.html)! + +#### ffmpeg-static +we use [ffmpeg-static](https://github.com/eugeneware/ffmpeg-static) to get binaries for ffmpeg depending on the platform. + +you can support the developer via various methods listed on their github page! (linked above) + +### youtube.js +cobalt relies on [youtube.js](https://github.com/LuanRT/YouTube.js) for interacting with the innertube api, it wouldn't have been possible without it. + +you can support the developer via various methods listed on their github page! (linked above) + +### many others +cobalt also depends on: + +- [content-disposition-header](https://www.npmjs.com/package/content-disposition-header) to simplify the provision of `content-disposition` headers. +- [cors](https://www.npmjs.com/package/cors) to manage cross-origin resource sharing within expressjs. +- [dotenv](https://www.npmjs.com/package/dotenv) to load environment variables from the `.env` file. +- [esbuild](https://www.npmjs.com/package/esbuild) to minify the frontend files. +- [express](https://www.npmjs.com/package/express) as the backbone of cobalt servers. +- [express-rate-limit](https://www.npmjs.com/package/express-rate-limit) to rate limit api endpoints. +- [hls-parser](https://www.npmjs.com/package/hls-parser) to parse `m3u8` playlists for certain services. +- [ipaddr.js](https://www.npmjs.com/package/ipaddr.js) to parse ip addresses (for rate limiting). +- [nanoid](https://www.npmjs.com/package/nanoid) to generate unique (temporary) identifiers for each requested stream. +- [node-cache](https://www.npmjs.com/package/node-cache) to cache stream info in server ram for a limited amount of time. +- [psl](https://www.npmjs.com/package/psl) as the domain name parser. +- [set-cookie-parser](https://www.npmjs.com/package/set-cookie-parser) to parse cookies that cobalt receives from certain services. +- [undici](https://www.npmjs.com/package/undici) for making http requests. +- [url-pattern](https://www.npmjs.com/package/url-pattern) to match provided links with supported patterns. -if you are looking for the documentation for the old (7.x) api, you can find -it [here](https://github.com/imputnet/cobalt/blob/7/docs/api.md) \ No newline at end of file +...and many other packages that these packages rely on. diff --git a/api/package.json b/api/package.json index 339d383f7..948f81805 100644 --- a/api/package.json +++ b/api/package.json @@ -10,9 +10,9 @@ }, "scripts": { "start": "node src/cobalt", - "setup": "node src/util/setup", "test": "node src/util/test", - "token:youtube": "node src/util/generate-youtube-tokens" + "token:youtube": "node src/util/generate-youtube-tokens", + "token:jwt": "node src/util/generate-jwt-secret" }, "repository": { "type": "git", diff --git a/api/src/cobalt.js b/api/src/cobalt.js index c548e792b..363930ba4 100644 --- a/api/src/cobalt.js +++ b/api/src/cobalt.js @@ -2,26 +2,24 @@ import "dotenv/config"; import express from "express"; -import path from 'path'; -import { fileURLToPath } from 'url'; +import path from "path"; +import { fileURLToPath } from "url"; import { env } from "./config.js" -import { Bright, Green, Red } from "./misc/console-text.js"; +import { Red } from "./misc/console-text.js"; const app = express(); const __filename = fileURLToPath(import.meta.url); const __dirname = path.dirname(__filename).slice(0, -4); -app.disable('x-powered-by'); +app.disable("x-powered-by"); if (env.apiURL) { - const { runAPI } = await import('./core/api.js'); + const { runAPI } = await import("./core/api.js"); runAPI(express, app, __dirname) } else { console.log( - Red(`cobalt wasn't configured yet or configuration is invalid.\n`) - + Bright(`please run the setup script to fix this: `) - + Green(`npm run setup`) + Red("API_URL env variable is missing, cobalt api can't start.") ) } diff --git a/api/src/config.js b/api/src/config.js index 3a28d7ce1..025842123 100644 --- a/api/src/config.js +++ b/api/src/config.js @@ -54,6 +54,10 @@ const env = { const genericUserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"; const cobaltUserAgent = `cobalt/${version} (+https://github.com/imputnet/cobalt)`; +if (env.sessionEnabled && env.jwtSecret.length < 16) { + throw new Error("JWT_SECRET env is too short (must be at least 16 characters long)"); +} + export { env, genericUserAgent, diff --git a/api/src/core/api.js b/api/src/core/api.js index b11d689a2..b31230331 100644 --- a/api/src/core/api.js +++ b/api/src/core/api.js @@ -158,19 +158,20 @@ export const runAPI = (express, app, __dirname) => { return fail(res, "error.api.auth.jwt.missing"); } - if (!authorization.startsWith("Bearer ") || authorization.length > 256) { + if (authorization.length >= 256) { return fail(res, "error.api.auth.jwt.invalid"); } - const verifyJwt = jwt.verify( - authorization.split("Bearer ", 2)[1] - ); + const [ type, token, ...rest ] = authorization.split(" "); + if (!token || type.toLowerCase() !== 'bearer' || rest.length) { + return fail(res, "error.api.auth.jwt.invalid"); + } - if (!verifyJwt) { + if (!jwt.verify(token)) { return fail(res, "error.api.auth.jwt.invalid"); } - req.rateLimitKey = generateHmac(req.header("Authorization"), ipSalt); + req.rateLimitKey = generateHmac(token, ipSalt); } catch { return fail(res, "error.api.generic"); } diff --git a/api/src/util/generate-jwt-secret.js b/api/src/util/generate-jwt-secret.js new file mode 100644 index 000000000..83f0aa5bf --- /dev/null +++ b/api/src/util/generate-jwt-secret.js @@ -0,0 +1,13 @@ +// run with `pnpm -r token:jwt` + +const makeSecureString = (length = 64) => { + const alphabet = 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_-'; + const out = []; + + for (const byte of crypto.getRandomValues(new Uint8Array(length))) + out.push(alphabet[byte % alphabet.length]); + + return out.join(''); +} + +console.log(`JWT_SECRET: ${JSON.stringify(makeSecureString(64))}`) diff --git a/api/src/util/setup.js b/api/src/util/setup.js deleted file mode 100644 index 34b870cb6..000000000 --- a/api/src/util/setup.js +++ /dev/null @@ -1,105 +0,0 @@ -import { existsSync, unlinkSync, appendFileSync } from "fs"; -import { createInterface } from "readline"; -import { Cyan, Bright } from "../misc/console-text.js"; -import { loadJSON } from "../misc/load-from-fs.js"; -import { execSync } from "child_process"; - -const { version } = loadJSON("./package.json"); - -let envPath = './.env'; -let q = `${Cyan('?')} \x1b[1m`; -let ob = {}; -let rl = createInterface({ input: process.stdin, output: process.stdout }); - -let final = () => { - if (existsSync(envPath)) unlinkSync(envPath); - - for (let i in ob) { - appendFileSync(envPath, `${i}=${ob[i]}\n`) - } - console.log(Bright("\nAwesome! I've created a fresh .env file for you.")); - console.log(`${Bright("Now I'll run")} ${Cyan("npm install")} ${Bright("to install all dependencies. It shouldn't take long.\n\n")}`); - execSync('pnpm install', { stdio: [0, 1, 2] }); - console.log(`\n\n${Cyan("All done!\n")}`); - console.log(Bright("You can re-run this script at any time to update the configuration.")); - console.log(Bright("\nYou're now ready to start cobalt. Simply run ") + Cyan("npm start") + Bright('!\nHave fun :)')); - rl.close() -} - -console.log( - `${Cyan(`Hey, this is cobalt v.${version}!`)}\n${Bright("Let's start by creating a new ")}${Cyan(".env")}${Bright(" file. You can always change it later.")}` -) - -function setup() { - console.log(Bright("\nWhat kind of server will this instance be?\nOptions: api, web.")); - - rl.question(q, r1 => { - switch (r1.toLowerCase()) { - case 'api': - console.log(Bright("\nCool! What's the domain this API instance will be running on? (localhost)\nExample: api.cobalt.tools")); - - rl.question(q, apiURL => { - ob.API_URL = `http://localhost:9000/`; - ob.API_PORT = 9000; - if (apiURL && apiURL !== "localhost") ob.API_URL = `https://${apiURL.toLowerCase()}/`; - - console.log(Bright("\nGreat! Now, what port will it be running on? (9000)")); - - rl.question(q, apiPort => { - if (apiPort) ob.API_PORT = apiPort; - if (apiPort && (apiURL === "localhost" || !apiURL)) ob.API_URL = `http://localhost:${apiPort}/`; - - console.log(Bright("\nWhat will your instance's name be? Usually it's something like eu-nl aka region-country. (local)")); - - rl.question(q, apiName => { - ob.API_NAME = apiName.toLowerCase(); - if (!apiName || apiName === "local") ob.API_NAME = "local"; - - console.log(Bright("\nOne last thing: would you like to enable CORS? It allows other websites and extensions to use your instance's API.\ny/n (n)")); - - rl.question(q, apiCors => { - let answCors = apiCors.toLowerCase().trim(); - if (answCors !== "y" && answCors !== "yes") ob.CORS_WILDCARD = '0' - final() - }) - }) - }); - - }) - break; - case 'web': - console.log(Bright("\nAwesome! What's the domain this web app instance will be running on? (localhost)\nExample: cobalt.tools")); - - rl.question(q, webURL => { - ob.WEB_URL = `http://localhost:9001/`; - ob.WEB_PORT = 9001; - if (webURL && webURL !== "localhost") ob.WEB_URL = `https://${webURL.toLowerCase()}/`; - - console.log( - Bright("\nGreat! Now, what port will it be running on? (9001)") - ) - rl.question(q, webPort => { - if (webPort) ob.WEB_PORT = webPort; - if (webPort && (webURL === "localhost" || !webURL)) ob.WEB_URL = `http://localhost:${webPort}/`; - - console.log( - Bright("\nOne last thing: what default API domain should be used? (api.cobalt.tools)\nIf it's hosted locally, make sure to include the port:") + Cyan(" localhost:9000") - ); - - rl.question(q, apiURL => { - ob.API_URL = `https://${apiURL.toLowerCase()}/`; - if (apiURL.includes(':')) ob.API_URL = `http://${apiURL.toLowerCase()}/`; - if (!apiURL) ob.API_URL = "https://api.cobalt.tools/"; - final() - }) - }); - - }); - break; - default: - console.log(Bright("\nThis is not an option. Try again.")); - setup() - } - }) -} -setup() diff --git a/api/src/util/tests.json b/api/src/util/tests.json index 94005c0a0..402a2baf4 100644 --- a/api/src/util/tests.json +++ b/api/src/util/tests.json @@ -238,7 +238,7 @@ }, { "name": "private song", - "url": "https://soundcloud.com/4kayy/unhappy-new-year-prod4kay/s-9bKbvwLdRWG", + "url": "https://soundcloud.com/user-798052861/asdasdasdsdsd/s-9TqZ7edLJ90", "params": { "audioFormat": "mp3" }, @@ -249,7 +249,7 @@ }, { "name": "private song (wav, isAudioMuted)", - "url": "https://soundcloud.com/4kayy/unhappy-new-year-prod4kay/s-9bKbvwLdRWG", + "url": "https://soundcloud.com/user-798052861/asdasdasdsdsd/s-9TqZ7edLJ90", "params": { "downloadMode": "mute", "audioFormat": "wav" @@ -261,7 +261,7 @@ }, { "name": "private song (ogg, isAudioMuted, isAudioOnly)", - "url": "https://soundcloud.com/4kayy/unhappy-new-year-prod4kay/s-9bKbvwLdRWG", + "url": "https://soundcloud.com/user-798052861/asdasdasdsdsd/s-9TqZ7edLJ90", "params": { "downloadMode": "audio", "audioFormat": "ogg" diff --git a/docs/api.md b/docs/api.md index 39b172096..dc7e9c59c 100644 --- a/docs/api.md +++ b/docs/api.md @@ -1,8 +1,8 @@ # cobalt api documentation this document provides info about methods and acceptable variables for all cobalt api requests. -> if you are looking for the documentation for the old (7.x) api, you can find -> it [here](https://github.com/imputnet/cobalt/blob/7/docs/api.md) +> [!IMPORTANT] +> hosted api instances (such as `api.cobalt.tools`) use bot protection and are **not** intended to be used in other projects without explicit permission. if you want to access the cobalt api reliably, you should [host your own instance](/docs/run-an-instance.md) or ask an instance owner for access. ## authentication an api instance may be configured to require you to authenticate yourself. @@ -46,9 +46,10 @@ cobalt's main processing endpoint. request body type: `application/json` response body type: `application/json` -``` -⚠️ you must include Accept and Content-Type headers with every `POST /` request. +> [!IMPORTANT] +> you must include `Accept` and `Content-Type` headers with every `POST /` request. +``` Accept: application/json Content-Type: application/json ``` diff --git a/docs/configure-for-youtube.md b/docs/configure-for-youtube.md new file mode 100644 index 000000000..fe286d860 --- /dev/null +++ b/docs/configure-for-youtube.md @@ -0,0 +1,33 @@ +# how to configure a cobalt instance for youtube +if you get various errors when attempting to download videos that are: +publicly available, not region locked, and not age-restricted; +then your instance's ip address may have bad reputation. + +in this case you have to use disposable google accounts. +there's no other known workaround as of time of writing this document. + +> [!CAUTION] +> **NEVER** use your personal google account for downloading videos via any means. +> you can use any google accounts that you're willing to sacrifice, +> but be prepared to have them **permanently suspended**. +> +> we recommend that you use accounts that don't link back to your personal google account or identity, just in case. +> +> use incognito mode when signing in. +> we also recommend using vpn/proxy services (such as [mullvad](https://mullvad.net/)). + +1. if you haven't done it already, clone the cobalt repo, go to the cloned directory, and run `pnpm install` + +2. run `pnpm -C api token:youtube` + +3. follow instructions, use incognito mode in your browser when signing in. +i cannot stress this enough, but again, **DO NOT USE YOUR PERSONAL GOOGLE ACCOUNT**. + +4. once you have the oauth token, add it to `youtube_oauth` in your cookies file. +you can see an [example here](/docs/examples/cookies.example.json). +you can have several account tokens in this file, if you like. + +5. all done! enjoy freedom. + +### liability +you're responsible for any damage done to any of your google accounts or any other damages. you do this by yourself and at your own risk. diff --git a/docs/images/protect-an-instance/add.png b/docs/images/protect-an-instance/add.png new file mode 100644 index 000000000..e186a65ce Binary files /dev/null and b/docs/images/protect-an-instance/add.png differ diff --git a/docs/images/protect-an-instance/created.png b/docs/images/protect-an-instance/created.png new file mode 100644 index 000000000..546a68978 Binary files /dev/null and b/docs/images/protect-an-instance/created.png differ diff --git a/docs/images/protect-an-instance/domain.png b/docs/images/protect-an-instance/domain.png new file mode 100644 index 000000000..249a8a925 Binary files /dev/null and b/docs/images/protect-an-instance/domain.png differ diff --git a/docs/images/protect-an-instance/mode.png b/docs/images/protect-an-instance/mode.png new file mode 100644 index 000000000..242b35a5b Binary files /dev/null and b/docs/images/protect-an-instance/mode.png differ diff --git a/docs/images/protect-an-instance/name.png b/docs/images/protect-an-instance/name.png new file mode 100644 index 000000000..fd39dc954 Binary files /dev/null and b/docs/images/protect-an-instance/name.png differ diff --git a/docs/images/protect-an-instance/sidebar.png b/docs/images/protect-an-instance/sidebar.png new file mode 100644 index 000000000..8294c4a05 Binary files /dev/null and b/docs/images/protect-an-instance/sidebar.png differ diff --git a/docs/images/troubleshooting/clipboard/config.png b/docs/images/troubleshooting/clipboard/config.png deleted file mode 100644 index b0c0a0480..000000000 Binary files a/docs/images/troubleshooting/clipboard/config.png and /dev/null differ diff --git a/docs/images/troubleshooting/clipboard/risk.png b/docs/images/troubleshooting/clipboard/risk.png deleted file mode 100644 index 1948f0eb4..000000000 Binary files a/docs/images/troubleshooting/clipboard/risk.png and /dev/null differ diff --git a/docs/images/troubleshooting/clipboard/search.png b/docs/images/troubleshooting/clipboard/search.png deleted file mode 100644 index 95684ff42..000000000 Binary files a/docs/images/troubleshooting/clipboard/search.png and /dev/null differ diff --git a/docs/images/troubleshooting/clipboard/toggle.png b/docs/images/troubleshooting/clipboard/toggle.png deleted file mode 100644 index 32060dc7f..000000000 Binary files a/docs/images/troubleshooting/clipboard/toggle.png and /dev/null differ diff --git a/docs/images/troubleshooting/clipboard/toggled.png b/docs/images/troubleshooting/clipboard/toggled.png deleted file mode 100644 index 6afa0ace7..000000000 Binary files a/docs/images/troubleshooting/clipboard/toggled.png and /dev/null differ diff --git a/docs/protect-an-instance.md b/docs/protect-an-instance.md new file mode 100644 index 000000000..9b4131c17 --- /dev/null +++ b/docs/protect-an-instance.md @@ -0,0 +1,150 @@ +# how to protect your cobalt instance +if you keep getting a ton of unknown traffic that hurts the performance of your instance, then it might be a good idea to enable bot protection. + +> [!NOTE] +> this tutorial will work reliably on the latest official version of cobalt 10. +we can't promise full compatibility with anything else. + +## configure cloudflare turnstile +turnstile is a free, safe, and privacy-respecting alternative to captcha. +cobalt uses it automatically to weed out bots and automated scripts. +your instance doesn't have to be proxied by cloudflare to use turnstile. +all you need is a free cloudflare account to get started. + +cloudflare dashboard interface might change over time, but basics should stay the same. + +> [!WARNING] +> never share the turnstile secret key, always keep it private. if accidentally exposed, rotate it in widget settings. + +1. open [the cloudflare dashboard](https://dash.cloudflare.com/) and log into your account + +2. once logged in, select `Turnstile` in the sidebar +
+

+ +

+
+ +3. press `Add widget` +
+

+ +

+
+ +4. enter the widget name (can be anything, such as "cobalt") +
+

+ +

+
+ +5. add cobalt frontend domains you want the widget to work with, you can change this list later at any time + - if you want to use your processing instance with [cobalt.tools](https://cobalt.tools/) frontend, then add `cobalt.tools` to the list +
+

+ +

+
+ +6. select `invisible` widget mode +
+

+ +

+
+ +7. press `create` + +8. keep the page with sitekey and secret key open, you'll need them later. +if you closed it, no worries! +just open the same turnstile page and press "settings" on your freshly made turnstile widget. + +
+

+ +

+
+ +you've successfully created a turnstile widget! +time to add it to your processing instance. + +### enable turnstile on your processing instance +this tutorial assumes that you only have `API_URL` in your `environment` variables list. +if you have other variables there, just add new ones after existing ones. + +> [!CAUTION] +> never use any values from the tutorial, especially `JWT_SECRET`! + +1. open your `docker-compose.yml` config file in any text editor of choice. +2. copy the turnstile sitekey & secret key and paste them to their respective variables. +`TURNSTILE_SITEKEY` for the sitekey and `TURNSTILE_SECRET` for the secret key: +```yml +environment: + API_URL: "https://your.instance.url.here.local/" + TURNSTILE_SITEKEY: "2x00000000000000000000BB" # use your key + TURNSTILE_SECRET: "2x0000000000000000000000000000000AA" # use your key +``` +3. generate a `JWT_SECRET`. we recommend using an alphanumeric collection with a length of at least 64 characters. +this string will be used as salt for all JWT keys. + + you can generate a random secret with `pnpm -r token:jwt` or use any other that you like. + +```yml +environment: + API_URL: "https://your.instance.url.here.local/" + TURNSTILE_SITEKEY: "2x00000000000000000000BB" # use your key + TURNSTILE_SECRET: "2x0000000000000000000000000000000AA" # use your key + JWT_SECRET: "bgBmF4efNCKPirD" # create a new secret, NEVER use this one +``` +4. restart the docker container. + +## configure api keys +if you want to use your instance outside of web interface, you'll need an api key! + +> [!NOTE] +> this tutorial assumes that you'll keep your keys file locally, on the instance server. +> if you wish to upload your file to a remote location, +> replace the value for `API_KEYS_URL` with a direct url to the file +> and skip the second step. + +> [!WARNING] +> when storing keys file remotely, make sure that it's not publicly accessible +> and that link to it is either authenticated (via query) or impossible to guess. +> +> if api keys leak, you'll have to update/remove all UUIDs to revoke them. + +1. create a `keys.json` file following [the schema and example here](/docs//run-an-instance.md#api-key-file-format). + +2. expose the `keys.json` to the docker container: +```yml +volumes: + - ./keys.json:/keys.json:ro # ro - read-only +``` + +3. add a path to the keys file to container environment: +```yml +environment: + # ... other variables here ... + API_KEY_URL: "file:///keys.json" +``` + +4. restart the docker container. + +## limit access to an instance with api keys but no turnstile +by default, api keys are additional, meaning that they're not *required*, +but work alongside with turnstile or no auth (regular ip hash rate limiting). + +to always require auth (via keys or turnstile, if configured), set `API_AUTH_REQUIRED` to 1: +```yml +environment: + # ... other variables here ... + API_AUTH_REQUIRED: 1 +``` + +- if both keys and turnstile are enabled, then nothing will change. +- if only keys are configured, then all requests without a valid api key will be refused. + +### why not make keys exclusive by default? +keys may be useful for going around rate limiting, +while keeping the rest of api rate limited, with no turnstile in place. diff --git a/docs/run-an-instance.md b/docs/run-an-instance.md index 272fbd357..638416259 100644 --- a/docs/run-an-instance.md +++ b/docs/run-an-instance.md @@ -1,4 +1,4 @@ -# how to host a cobalt instance yourself +# how to run a cobalt instance ## using docker compose and package from github (recommended) to run the cobalt docker package, you need to have `docker` and `docker-compose` installed and configured. @@ -54,8 +54,7 @@ sudo apt install nscd sudo service nscd start ``` -## list of all environment variables -### variables for api +## list of environment variables for api | variable name | default | example | description | |:----------------------|:----------|:------------------------|:------------| | `API_PORT` | `9000` | `9000` | changes port from which api server is accessible. | @@ -89,7 +88,7 @@ requests it makes for that particular download. to use freebind in cobalt, you n in a docker container, you also need to set the `API_LISTEN_ADDRESS` env to `127.0.0.1`, and set `network_mode` for the container to `host`. -#### api key file format +## api key file format the file is a JSON-serialized object with the following structure: ```typescript diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md deleted file mode 100644 index 4c97511f1..000000000 --- a/docs/troubleshooting.md +++ /dev/null @@ -1,37 +0,0 @@ -# self-troubleshooting cobalt -``` -🚧 this page is work-in-progress. expect more guides to be added in the future! -``` -if any issues occur while using cobalt, you can fix many of them yourself. this document aims to provide guides on how to fix most complicated of them. -use wiki navigation on right to jump between solutions. - -## how to fix clipboard pasting in older versions of firefox -``` -🎉 firefox finally supports pasting by default starting from version 125. - -👍 you don't need to follow this tutorial if you're on the latest version of firefox. -``` -you can fix this issue by changing a single preference in `about:config`. - -### steps to enable clipboard functionality -1. go to `about:config`: - ![screenshot showing about:config entered into address bar](images/troubleshooting/clipboard/config.png) - -2. if asked, read what firefox has to say and press "accept the risk and continue". - ⚠ tinkering with other preferences may break your browser. **do not** edit them unless you know what you're doing. - - ![screenshot showing about:config security warning that reads: "proceed with caution. changing advanced configuration preferences can impact firefox performance or security." lower there's a pre-checked checkbox that says: "warn me when i attempt to access these preferences". lowest element is a blue button that says "accept the risk and continue"](images/troubleshooting/clipboard/risk.png) - -3. search for `dom.events.asyncClipboard.readText` - - ![screenshot showing "dom.events.asyncclipboard.readtext" entered into search on about:config page](images/troubleshooting/clipboard/search.png) - -4. press the toggle button on very right. - - ![screenshot showing "dom.events.asyncclipboard.readtext" preference on about:config page with highlighted toggle button on very right](images/troubleshooting/clipboard/toggle.png) - -5. "false" should change to "true". - - ![screenshot showing "dom.events.asyncclipboard.readtext" preference on about:config page, this one with "true" text highlighted](images/troubleshooting/clipboard/toggled.png) - -6. go back to cobalt, reload the page, press `paste` button again. this time it works! enjoy simpler downloading experience :) diff --git a/web/i18n/en/about/terms.md b/web/i18n/en/about/terms.md index a134ab84b..aa32fd652 100644 --- a/web/i18n/en/about/terms.md +++ b/web/i18n/en/about/terms.md @@ -48,9 +48,10 @@ fair use and credits benefit everyone. sectionId="abuse" /> -we have no way of detecting abusive behavior automatically, as cobalt is 100% anonymous. -however, you can report such activities to us and we will do our best to comply manually: [safety@imput.net](mailto:safety@imput.net) +we have no way of detecting abusive behavior automatically because cobalt is 100% anonymous. +however, you can report such activities to us and we will do our best to comply manually: **safety@imput.net** + +**this email is not intended for user support, you will not get a response if your concern is not related to abuse.** -please note that this email is not intended for user support. if you're experiencing issues, contact us via any preferred method on [the support page](/about/community). diff --git a/web/i18n/en/dialog.json b/web/i18n/en/dialog.json index a2688f6d7..3e6f5dece 100644 --- a/web/i18n/en/dialog.json +++ b/web/i18n/en/dialog.json @@ -1,6 +1,6 @@ { - "reset.title": "reset all settings?", - "reset.body": "are you sure you want to reset all settings? this action is immediate and irreversible.", + "reset.title": "reset all data?", + "reset.body": "are you sure you want to reset all data? this action is immediate and irreversible.", "picker.title": "select what to save", "picker.description.desktop": "click an item to save it. images can also be saved via the right click menu.", diff --git a/web/i18n/en/settings.json b/web/i18n/en/settings.json index 0537982fe..8d0034395 100644 --- a/web/i18n/en/settings.json +++ b/web/i18n/en/settings.json @@ -106,7 +106,7 @@ "advanced.debug.title": "enable debug features", "advanced.debug.description": "gives you access to a page with various info that can be useful for debugging.", - "advanced.data": "settings data", + "advanced.data": "data management", "processing.override": "default instance override", "processing.override.title": "use the instance-provided processing server", diff --git a/web/src/components/dialog/DialogContainer.svelte b/web/src/components/dialog/DialogContainer.svelte index 993efd7c1..363395ba4 100644 --- a/web/src/components/dialog/DialogContainer.svelte +++ b/web/src/components/dialog/DialogContainer.svelte @@ -1,6 +1,6 @@ {#if $settings.advanced.debug}