From 07ae17a2338fa2082e75f0095d868c5982309c42 Mon Sep 17 00:00:00 2001
From: Vishal <321vishalds@gmail.com>
Date: Sun, 28 Jul 2024 09:43:24 +0530
Subject: [PATCH] Documented (#2)
* Create README.md
* added contribution guidelines
* documented code
* Corrected diagram
---
CONTRIBUTING.md | 37 ++++++++++++
README.md | 124 ++++++++++++++++++++++++++++++++++++++
scripts/builder.js | 6 ++
scripts/screenshot.js | 4 +-
src/config.rs | 3 +
src/handlers/user.rs | 5 ++
src/main.rs | 2 +
src/models/site.rs | 2 +
src/models/template.rs | 5 ++
src/server/mod.rs | 3 +
src/services/preview.rs | 3 +
src/services/tempfiles.rs | 1 +
src/utils.rs | 7 ---
13 files changed, 193 insertions(+), 9 deletions(-)
create mode 100644 CONTRIBUTING.md
create mode 100644 README.md
delete mode 100644 src/utils.rs
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
new file mode 100644
index 0000000..3a59381
--- /dev/null
+++ b/CONTRIBUTING.md
@@ -0,0 +1,37 @@
+# Contributing
+
+First of all, a huge thank you for looking to contribute to this repository! This project was concieved and built at [FOSS Hack 4.0](https://fossunited.org/fosshack/2024), a hackathon for building open-source software. As such, everything from the architecture of this server to the code has been *hacked up* in a couple of hours. That is to say, this is messed up lol. There's tons of stuff to improve.
+
+## Improvements
+These are some of the stuff that we'd like to improve upon
+
+### Reduce memory footprint
+Any changes to reduce the overall memory footprint of the application are welcome. We're sure there are lots of `String` variables instead of string literals waiting for you to get started with. 😂 Just for context, there are 36 `.clone()` calls throughout the codebase right now! That's 36 deep copy operations waiting to be removed.
+
+Any changes improving performance in any other way are appreciated as well. Eliminating JavaScript is welcome. More on that below.
+
+### Complete overhaul?
+Sure, why not! As said earlier, this was hacked up and we're not sure this is the best approach to build this server. Right now, it works and that's all we know! If you find a way to overhaul it and make it better, please do so and open a PR!
+
+> Note: If your PR changes any of the endpoint behaviors or names, please coordinate with us by [messaging me on telegram](https://t.me/vishalds/).
+
+### Eliminating JavaScript
+The screenshot script can maybe be replaced, but we couldn't find an effective solution for that right now. The most effective workaround for this would be to switch the previews altogether to use an `` in the site. We did try this during the hackathon but couldn't quite get it to work. However, that is outside the scope of this repository. We'd still need screenshots for the previews, so any attempts to move this to rust are appreciated.
+
+As for the HTML string builder script, there currently seem to be no effective solutions to parse HTML reliably with rust alone. We briefly tried [scraper](https://docs.rs/scraper/latest/scraper/) & [select](https://docs.rs/select/latest/select/) but both of these ended up not having solutions to edit the HTML content. They only allow us to parse a string into different elements. So, we resorted to use JS and native DOM for this. Any attempts to move this to rust are appreciated as well.
+
+### Containerization
+Usually, servers like these are containerized on deployment. However, since we're not familiar with setting up containers frequently, we couldn't risk doing this at the hackathon.
+
+So, any attempts to containerize this are welcome. A good solution seems to be [docker](https://docker.com/). We're open to anything though.
+
+### Improve code quality
+If you can find places to refactor, improve code quality, or do any such thing, feel free to open a PR.
+
+## How to contribute?
+Simple, just open a PR. Let the checks run. If the build check is successful and your description conveys the intent, then we'll probably merge it.
+
+You can also open an issue if you spot one.
+
+## Merging right now...
+Right now, the evaluation for FOSS Hack is still ongoing. As such, we won't be merging any PR's until August 20, 2024. But feel free to open one, we'll merge once the evaluation finishes.
\ No newline at end of file
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..ec757b4
--- /dev/null
+++ b/README.md
@@ -0,0 +1,124 @@
+
+
+# `server` 🖥️
+
+> This is the backend server of zitefy, which manages all the user data, templates, website previews and basically everything. Written in rust, it uses the actix web framework with some other crates.
+
+## Table Of Contents
+* [What's this?](#whats-this)
+ * [Role in the stack](#role-in-the-stack)
+ * [Architectural Overview](#architectural-overview)
+* [Routes](#routes)
+* [Security](#security)
+* [Setup Guide](#setup-guide)
+ * [Prerequisites](#prerequisites)
+ * [Installation](#installation)
+ * [Build & run](#build--run)
+* [Contributing](#contributing)
+
+## What's this?
+In order for the core functionalities to work, there should be a central system where templates, users & their sites along with other stuff like assets, seo data can come together. This server is the means for doing so.
+
+### Role in the stack
+* Maintain and store user data, site data & template data
+* Provide a storage mechanism for storing site assets, seo data, etc
+* Generate previews for templates, sites & the web based editor
+
+### Architectural Overview
+This is written from the high level architecture shown below. You can find some pointers below highlighting the core idea in a nutshell. Feel free to open a new issue if you find any part of this documentation confusing or hard to understand.
+
+
+
+*In a nutshell...*
+ * Each user can register themselves on the API. Once registered or logged in, the API will generate an access token that can then be used to access the secure endpoints. Registered users can create sites from the available templates.
+ * In zitefy, only user data is kept private. This is an open platform and so, all source code & data related to all sites & templates will be exposed publicly. This is a reminder to not upload any data to a site that you don't want the world to see.
+ * Templates are uploaded and managed from the [templates]() repository. The CI/CD uploads the templates to a dedicated directory on the server and a background task that runs once every hour will update the template data in the database.
+ * The preview engine is basically two JS scripts. One is the [`scripts/builder.js`]() that builds an HTML string from the given data. Another is the [`scripts/screenshot.js`]() that takes a screenshot of the generated HTML.
+
+The intention was to keep this a pure rust codebase, but it turns out that there are no effective html parsers in rust. Neither are there any methods to take a screenshot of a webpage. So, we ended up writing two JS files for those and only those. The rest is pure rust.
+
+The server then binds with two ports. Then, [nginx](https://nginx.org/en/0) redirects traffic from both [api.zitefy.com](https://api.zitefy.com/docs/#/) & [zitefy.com](https://zitefy.com/) to the respective ports. There is a [systemd](https://systemd.io/) service that manages everything.
+
+It would be cool to containerize this server as it'd speed up CI/CD for future versions. But right now, the setup overhead seemed a bit too much for the hackathon so we skipped it.
+
+## Routes
+Routes are of 4 types: `site`, `template`, `user` & `proxy`. Yeah, unfortunately proxy is a category. Anthropic, regardless of maintaining a typescript SDK, [doesn't allow CORS](https://github.com/anthropics/anthropic-sdk-typescript/issues/410) in their API rendering it unusable for web apps. If this had worked, it would have reduced our code complexity as well, but since it doesn't, proxy is a thing.
+
+As it should be obvious, listing all the routes here seems tedious. So, they're documented in this [swagger UI](https://api.zitefy.com/docs/#/).
+
+## Security
+The focus up until now has been somehow getting this to work properly for the hackathon. Security wasn't the first thing in our mind when building this. As such, it is advised not to upload any sensitive info to zitefy.
+
+Here are the measures taken for security:
+ * All passwords are encrypted using a hashing algorithm with a secret key. Without it, the API will not allow access to user data
+ * All logins are regulated by access tokens which expire every 30 days. After 30 days, you'd have to login again to continue using the site
+ * The secret key used for hashing, along with some other stuff, are stored in a separate file not exposed in this repository.
+
+If you find a security vulnerability, please do open an issue/PR and we'd be happy to accept it!
+
+## Setup Guide
+You can follow this guide to run the server on a machine of your choice. If you want to deploy, it's advisable to do so in a linux system. There is nothing OS specific about the code itself, and so it can build and run on any platform. However, it'd be better to deploy on a linux machine since the secrets and service file are setup to be managed by systemd. When you open a PR, it'll be much easier for us to evaluate. If you're on Windows, WSL would be more than enough.
+
+### Prerequisites
+* The **Rust toolchain**, including cargo.
+
+ Install for linux/WSL by running this command
+ ```
+ $ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
+ ```
+ Alternatively, follow the instruction method for your platform [here](https://www.rust-lang.org/tools/install)
+
+* **Git CLI**
+
+ For linux, skip this step. Install by running the executable for your platform available [here](https://git-scm.com/downloads)
+
+* **MongoDB Community Server**
+
+ Install for Ubuntu by following [this guide](https://www.mongodb.com/docs/manual/tutorial/install-mongodb-on-ubuntu/). For other operating systems, follow the corresponding tutorial from the [tutorials](https://www.mongodb.com/docs/manual/installation/#mongodb-installation-tutorials) page. If you want to visualize the data better, install MongoDB Compass by following [this guide](https://www.mongodb.com/docs/compass/current/install/).
+
+* **bun**
+
+ Install for linux/WSL or macOS by running this command
+ ```
+ $ curl -fsSL https://bun.sh/install | bash
+ ```
+ Alternatively, you can follow the instructions for your platform [here](https://bun.sh/).
+
+### Installation
+1. Clone the repository to your machine.
+
+ ```
+ git clone https://github.com/zitefy/server.git
+ ```
+
+2. Install JavaScript dependencies
+
+ ```
+ bun install puppeteer
+ ```
+ ```
+ bun run puppeteer browsers install chrome
+ ```
+3. Copy the chrome executable path to line 8 in [`scripts/screenshot.js`]().
+
+### Build & run
+1. Create an env file with all your secrets. An example file would look like this
+
+ ```env
+ MONGODB_URI="mongodb://127.0.0.1:27017/"
+ API_ADDR=127.0.0.1:7878
+ SERVER_ADDR=127.0.0.1:7979
+ SECRET_KEY="xxxx"
+ ANTHROPIC_KEY='xxxx'
+ ```
+ You can get your anthropic API key [here](https://console.anthropic.com/settings/keys)
+2. Build and run the server
+
+ ```
+ cargo run
+ ```
+
+That's about it, the server should be up and running locally in your machine.
+
+## Contributing
+There's tons of stuff to contribute. Please refer to the [contributing guide](https://github.com/zitefy/server/blob/main/CONTRIBUTING.md).
diff --git a/scripts/builder.js b/scripts/builder.js
index 4013f89..b057344 100644
--- a/scripts/builder.js
+++ b/scripts/builder.js
@@ -1,3 +1,9 @@
+/*
+ This is a script that takes in the html, css & js to generate a single html string.
+ It adds the CSS to a tag, the JS to a tag.
+ It then adds the links to the specified elements, hides the unselected ones, and returns the final string.
+*/
+
const fs = require('fs');
const { JSDOM } = require('jsdom');
diff --git a/scripts/screenshot.js b/scripts/screenshot.js
index da0e91f..e174ce8 100644
--- a/scripts/screenshot.js
+++ b/scripts/screenshot.js
@@ -1,6 +1,6 @@
/*
- During initial setup, run bun run puppeteer browsers install chrome,
- and copy the executable path to line 8
+ This script takes in as input a file that contains a HTML string, renders it in a browser instance, and takes two screenshots in different sizes.
+ During initial setup, run bun run puppeteer browsers install chrome, and copy the executable path to line 8
*/
const puppeteer = require('puppeteer');
diff --git a/src/config.rs b/src/config.rs
index dddf62a..25a7533 100644
--- a/src/config.rs
+++ b/src/config.rs
@@ -27,6 +27,9 @@ impl Config {
}
}
+// Right now, this seems like the best way to manage the upload of templates from github
+// This function is very expensive, so a better method would be to write a dedicated endpoint for uploading templates from the site,
+// and adding it to the database along with it. But that's for another day.
pub async fn monitor_templates_directory(app_state: Arc) {
let templates_dir = format!("{}/.zitefy/templates", env::var("HOME").unwrap());
diff --git a/src/handlers/user.rs b/src/handlers/user.rs
index 3c4f25a..df64860 100644
--- a/src/handlers/user.rs
+++ b/src/handlers/user.rs
@@ -1,3 +1,6 @@
+// In this file, the logic should have existed in models/user.rs
+// in the heat of the moment, I've just gotten this to work.
+
use actix_files::NamedFile;
use actix_multipart::Multipart;
use actix_web::http::Error;
@@ -236,6 +239,8 @@ async fn edit(
) -> impl Responder {
let users: Collection = app_state.db.collection("users");
+ // there might be a way to refactor this better. this line repeats everywhere
+ // get_user_id_from_token() was refactored, but i still think there's more room.
let user_id = match get_user_id_from_token(&req, &app_state).await {
Ok(id) => id,
Err(response) => return response,
diff --git a/src/main.rs b/src/main.rs
index d9a0864..09260f8 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -23,6 +23,8 @@ use handlers::{user::LoginResponse, proxy_anthropic};
use models::user::{EditData, LoginData, SignupData, UserDataResponse};
use services::tempfiles::TempFileService;
+// this is very cumbersome, has to be changed.
+// right now, we don't have time to clean this up, but there should be a way.
#[derive(OpenApi)]
#[openapi(
paths(
diff --git a/src/models/site.rs b/src/models/site.rs
index e5a227e..3216bb3 100644
--- a/src/models/site.rs
+++ b/src/models/site.rs
@@ -62,6 +62,7 @@ pub struct Site {
pub user: ObjectId,
}
+// most of the names & code are self-explanatory. nothing much to document per se
impl Site {
pub async fn new(
template_id: ObjectId,
@@ -351,6 +352,7 @@ pub fn read_dir_to_string(dir: &Path) -> io::Result {
Ok(contents)
}
+// recursively reads the contents of a directory to filenames and returns them
pub fn read_dir_to_names(dir: &Path) -> io::Result> {
let mut urls = Vec::new();
for entry in fs::read_dir(dir)? {
diff --git a/src/models/template.rs b/src/models/template.rs
index 727a2e4..26fdeab 100644
--- a/src/models/template.rs
+++ b/src/models/template.rs
@@ -31,6 +31,7 @@ pub struct Template {
}
impl Template {
+ // add a template from metadata.json in the directory
pub async fn from_metadata(path: &Path) -> Result> {
let metadata_path = path.join("metadata.json");
let metadata_file = fs::File::open(metadata_path)?;
@@ -55,6 +56,7 @@ impl Template {
})).await?)
}
+ // build a site from a specified template
pub async fn build_site(site_id: ObjectId, user_id: ObjectId, template_id: ObjectId, app_state: &Arc) -> Result> {
let home_dir = std::env::var("HOME")?;
let site_dir = Path::new(&home_dir).join(".zitefy").join("sites").join(site_id.to_hex());
@@ -98,6 +100,7 @@ impl Template {
}
+// copy everything from one dir to another
fn copy_dir_all(src: &Path, dst: &Path, exclude: Option<&str>) -> io::Result<()> {
if !dst.exists() {
fs::create_dir_all(dst)?;
@@ -125,6 +128,7 @@ fn copy_dir_all(src: &Path, dst: &Path, exclude: Option<&str>) -> io::Result<()>
Ok(())
}
+// invoked by the background task to add a template to the db
pub async fn update_template_in_db(path: &Path, app_state: &Arc) -> Result<(), Box> {
let mut template = Template::from_metadata(path).await?;
template.dir_path = path.to_string_lossy().to_string();
@@ -139,6 +143,7 @@ pub async fn update_template_in_db(path: &Path, app_state: &Arc) -> Re
Ok(())
}
+// if ever we wanna implement deleting templates. code is redundant rn, so commenting to reduce build size.
// pub async fn delete_template_from_db(path: &Path, app_state: &Arc) -> Result<(), Box> {
// let metadata_path = path.join("metadata.json");
// if metadata_path.exists() {
diff --git a/src/server/mod.rs b/src/server/mod.rs
index 60cccd5..687c9eb 100644
--- a/src/server/mod.rs
+++ b/src/server/mod.rs
@@ -5,6 +5,9 @@ use std::{sync::Arc, env, path::PathBuf};
use crate::models::user::User;
+// the server at zitefy.com
+// checks if a username matches one in the db & if it has an active site.
+// if so, serves the site, otherwise redirects to the portal
pub async fn domain_server(
app_state: web::Data>,
req: HttpRequest,
diff --git a/src/services/preview.rs b/src/services/preview.rs
index f37ae30..62e6f59 100644
--- a/src/services/preview.rs
+++ b/src/services/preview.rs
@@ -13,6 +13,8 @@ pub struct Preview {
pub desktop: PathBuf,
}
+// if a path is provided, the preview images will be stored to that path, otherwise generates a temporarry file
+// useful for generating permenant and temporarry images (for template previews)
pub async fn generate_preview(
html: &str,
paths: Option,
@@ -55,6 +57,7 @@ pub async fn generate_preview(
})
}
+// just a wrapper around scripts/builder.js
pub fn build_html_string(
html: PathBuf,
css: PathBuf,
diff --git a/src/services/tempfiles.rs b/src/services/tempfiles.rs
index 292b0c8..b2ded8d 100644
--- a/src/services/tempfiles.rs
+++ b/src/services/tempfiles.rs
@@ -5,6 +5,7 @@ use uuid::Uuid;
use std::path::PathBuf;
use std::time::{SystemTime, UNIX_EPOCH};
+// basically just a setup to manage temporarry files, like preview images in the editor.
pub struct TempFile {
path: PathBuf,
expiry: u64,
diff --git a/src/utils.rs b/src/utils.rs
deleted file mode 100644
index 5497a39..0000000
--- a/src/utils.rs
+++ /dev/null
@@ -1,7 +0,0 @@
-pub fn get_url(path: &str) -> String {
- format!("https://api.site.com/resource/{}", path)
-}
-
-pub fn get_resource(type_: &str, path: &str) -> std::path::PathBuf {
- std::path::Path::new("base_dir").join(type_).join(path)
-}