Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DRAFT: Add Web page interface #132

Open
wants to merge 23 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
b86e7e6
wik: beta for web page to export to ghostfolio
JuanmanDev Dec 7, 2024
3cd61f9
feat: updated docker creation on github
JuanmanDev Dec 7, 2024
f1a382b
docs: updated readme with basic docker compose info
JuanmanDev Dec 7, 2024
a19bd63
docs: force ci
JuanmanDev Dec 7, 2024
a38bd40
ci: removed dependencies to PR on my repo
JuanmanDev Dec 7, 2024
4f659a3
feat: added github docker image to ci
JuanmanDev Dec 7, 2024
df39dc3
feat: auto referency current repo dor docker image on github actions
JuanmanDev Dec 7, 2024
ed00068
fix: username to lowercase + on all PR docker run
JuanmanDev Dec 7, 2024
94d9ed2
ci: fix repo owner incorrect code
JuanmanDev Dec 7, 2024
b27eea2
feat: added coker-github to split changes
JuanmanDev Dec 7, 2024
8833a77
ci: improved docker
JuanmanDev Dec 7, 2024
4e65f66
ci: improved size for docker image removing dev depencies on final image
JuanmanDev Dec 7, 2024
b4fe30c
feat: restored package-lock.json
JuanmanDev Dec 7, 2024
c622d3f
ci: undo docker.yml file
JuanmanDev Dec 7, 2024
7db796e
feat: clean web code
JuanmanDev Dec 7, 2024
08f389c
Merge pull request #2 from JuanmanDev/feature/docker-ci
JuanmanDev Dec 7, 2024
786f41d
docs: aded web basic info
JuanmanDev Dec 7, 2024
2bb7fab
fix: icon on buttons adn type for file
JuanmanDev Dec 7, 2024
b7e95b1
chore: update package.-lock.json to be align with package.json
JuanmanDev Dec 7, 2024
fc06d70
fix: docker url
JuanmanDev Dec 7, 2024
7c753ce
Merge pull request #3 from JuanmanDev/feature/web-improvements
JuanmanDev Dec 7, 2024
72a8fbc
feat: return env from endpoint to aling with prod changes
JuanmanDev Dec 7, 2024
8628aca
Merge pull request #4 from JuanmanDev/fix/share-envs
JuanmanDev Dec 7, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/CODEOWNERS
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@
# Each line is a file pattern followed by one or more owners.

# These owners will be the default owners for everything in the repo.
* @dickwolff
* @juanmandev
76 changes: 76 additions & 0 deletions .github/workflows/docker-github.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
name: Build and Deploy Docker Image

on:
push:
branches:
- main
- feature/docker*
- feature/add-docker*
pull_request:
branches:
- main
- feature/docker*
- feature/add-docker*

jobs:
build-and-push:
runs-on: ubuntu-latest

steps:

# Step 1: Check out the repository
- name: Check out code
uses: actions/checkout@v3

# Step 2: Set up Node.js 20
- name: Setup Node.js
uses: actions/setup-node@v3
with:
node-version: 22
cache: 'npm'

# Step 3: Install dependencies
- name: Install dependencies
run: npm install

# Step 4: Build the Docker image using the npm script
- name: Build Docker image
run: |
REPO_NAME=$(echo "${{ github.repository }}" | awk -F '/' '{print $2}' | tr '[:upper:]' '[:lower:]')
docker build -t $REPO_NAME:latest .

# Step 5: Log in to GitHub Container Registry
- name: Log in to GitHub Container Registry
uses: docker/login-action@v2
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

# Step 6: Push Docker image with branch-specific tags
- name: Push Docker image with branch-specific tags
run: |
# Get the repository name and transform it into a valid Docker image name
REPO_NAME=$(echo "${{ github.repository }}" | awk -F '/' '{print $2}' | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9._-]/-/g')
REPO_OWNER=$(echo "${{ github.repository_owner }}" | tr '[:upper:]' '[:lower:]')
IMAGE_NAME=ghcr.io/$REPO_OWNER/$REPO_NAME

# Sanitize the branch name to be Docker-compatible
SANITIZED_BRANCH_NAME=$(echo "${{ github.ref_name }}" | sed 's/[^a-zA-Z0-9._-]/-/g')

# Tag and push the Docker image with branch-specific tags
docker tag $REPO_NAME:latest $IMAGE_NAME:$SANITIZED_BRANCH_NAME
docker push $IMAGE_NAME:$SANITIZED_BRANCH_NAME

# Step 7: Push the 'latest' tag only for the main branch
- name: Push 'latest' tag for main branch
if: github.ref == 'refs/heads/main'
run: |
# Reuse the repository name transformation logic
REPO_NAME=$(echo "${{ github.repository }}" | awk -F '/' '{print $2}' | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9._-]/-/g')
REPO_OWNER=$(echo "${{ github.repository_owner }}" | tr '[:upper:]' '[:lower:]')
IMAGE_NAME=ghcr.io/$REPO_OWNER/$REPO_NAME

# Tag and push the Docker image with the 'latest' tag
docker tag $REPO_NAME:latest $IMAGE_NAME:latest
docker push $IMAGE_NAME:latest
19 changes: 6 additions & 13 deletions .github/workflows/docker.yml
Original file line number Diff line number Diff line change
@@ -1,18 +1,19 @@
name: docker
run-name: Build Docker Container

on:
push:
branches:
- main
paths:
- "src/**"
- "pages"
- "package.json"
pull_request:
branches:
- main
paths:
- "src/**"
- "pages"
- "package.json"
workflow_dispatch: # Also allow manual triggering

Expand All @@ -29,30 +30,24 @@ jobs:
uses: actions/checkout@v3
with:
fetch-depth: 0

- name: Install GitVersion
uses: gittools/actions/gitversion/[email protected]
with:
versionSpec: "5.x"

- name: Run GitVersion
uses: gittools/actions/gitversion/[email protected]
with:
useConfigFile: true

- name: Set up QEMU
uses: docker/setup-qemu-action@v3

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

- name: Login to Docker Hub
if: ${{ env.dockerhub_username != '' && env.dockerhub_token != '' }}
uses: docker/login-action@v3
with:
username: ${{ env.dockerhub_username }}
password: ${{ env.dockerhub_token }}

- name: Determine image version
# shell: bash
run: |
Expand All @@ -61,26 +56,24 @@ jobs:
else
echo "IMAGE_VERSION=${{ env.GitVersion_MajorMinorPatch }}" >> $GITHUB_ENV
fi

- name: Build and push (PR)
uses: docker/build-push-action@v5
if: ${{ github.event_name == 'pull_request' && env.dockerhub_username != '' && env.dockerhub_token != '' }}
with:
push: true
tags: |
dickwolff/export-to-ghostfolio:${{ env.IMAGE_VERSION }}
${{ github.ref_name }}/export-to-ghostfolio:${{ env.IMAGE_VERSION }}
build-args: |
IMAGE_VERSION=${{ env.IMAGE_VERSION }}
platforms: linux/amd64,linux/arm64

- name: Build and push (main)
uses: docker/build-push-action@v5
if: ${{ github.event_name != 'pull_request' && env.dockerhub_username != '' && env.dockerhub_token != '' }}
with:
push: true
tags: |
dickwolff/export-to-ghostfolio:latest
dickwolff/export-to-ghostfolio:${{ env.IMAGE_VERSION }}
${{ github.ref_name }}/export-to-ghostfolio:latest
${{ github.ref_name }}/export-to-ghostfolio:${{ env.IMAGE_VERSION }}
build-args: |
IMAGE_VERSION=${{ env.IMAGE_VERSION }}
platforms: linux/amd64,linux/arm64
platforms: linux/amd64,linux/arm64
2 changes: 1 addition & 1 deletion .github/workflows/frameworkTesting.yml
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ jobs:

- name: Add code coverage PR comment (if owner)
uses: marocchino/sticky-pull-request-comment@v2
if: github.event_name == 'pull_request' && github.event.pull_request.user.login == 'dickwolff'
if: github.event_name == 'pull_request'
with:
recreate: true
path: code-coverage-results.md
Expand Down
10 changes: 7 additions & 3 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -34,9 +34,6 @@ Thumbs.db
# Ignore built ts files
dist/**/*

# ignore package-lock
package-lock.json

# Allow sample export files
!samples/*-export.csv

Expand All @@ -57,3 +54,10 @@ ghostfolio-*.json

# Ignore output folder
e2g-output

# Ignore uploads
uploads

# Ignore nuxt files (frontend)
.nuxt
.output
24 changes: 16 additions & 8 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,20 +1,28 @@
FROM node:20-alpine3.19
FROM node:22-alpine3.19
WORKDIR /app

ARG IMAGE_VERSION

LABEL name="Export to Ghostfolio"
LABEL name="Export to Ghostfolio on Web"
LABEL description="Convert transaction history export from your favorite broker to a format that can be imported in Ghostfolio."
LABEL author="Dick Wolff"
LABEL author="Dick Wolff & JuanmanDev"
LABEL version="$IMAGE_VERSION"

COPY . .

# Install all dependencies (including dev dependencies)
RUN npm install

RUN mkdir /var/tmp/e2g-input
RUN mkdir /var/tmp/e2g-output
RUN mkdir /var/tmp/e2g-cache
# Run the generate command
RUN node --run web:generate

ENTRYPOINT [ "npm" ]
CMD ["run", "watch"]
# Remove dev dependencies
RUN npm prune --production

# Create necessary directories
RUN mkdir -p /var/tmp/e2g-input /var/tmp/e2g-output /var/tmp/e2g-cache

EXPOSE 3001

ENTRYPOINT [ "node" ]
CMD ["--run", "server"]
32 changes: 30 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,9 +119,19 @@ Login to your XTB account and from the top bar click on "Account history", then

You can run the tool on your local machine by cloning this repository. You can also run the tool inside a Docker container. See the runtime specific instructions below.

## Docker
## Docker compose

[![Docker Pulls](https://img.shields.io/docker/pulls/dickwolff/export-to-ghostfolio?style=for-the-badge)](https://hub.docker.com/r/dickwolff/export-to-ghostfolio)
Install docker

Copy the `docker-compose.yml` file on your machine/server and run the file. Remember to configure ports and environment variables for that file.

_If you have previous version are the same envs variables._

```shell
docker compose up
```

When you finnish using it, press Ctrl-C on the terminal you openned.

<details>
<summary>View instructions</summary>
Expand Down Expand Up @@ -228,6 +238,23 @@ You can now run `npm run start [exporttype]`. See the table with run commands be

The tool uses `cacache` to store data retrieved from Yahoo Finance on disk. This way the load on Yahoo Finance is reduced and the tool should run faster. The cached data is stored in `/var/tmp/e2g-cache`. If you feel you need to invalidate your cache, you can do so by removing the folder and the tool will recreate the cache when you run it the next time.


### Use the tool with Web

Run `npm install` to install all required packages.

Run `npm run web:dev` to start the web on port 3000.
Run `npm run server` to start the web on port 3001.

Recommended node version: 22.

To have web inside the server, generate the web files with `node --run web:generate`.
Them you only have to run: `npm run server`.

You can write the variables on the web server (woon't be saved), or using the same .env file than before that would be showed on the web page (recommended if you are alone).



</details>

## Import to Ghostfolio
Expand Down Expand Up @@ -258,3 +285,4 @@ The tool can be run two ways, manually and via Docker. Both entrypoints of the t
The tool uses a mock in the tests, which allow the tests to be run in a consistent and repeatable manner. This way there is no necessity for a live Yahoo Finance service. The mock was added because of inconsistencies in between test runs and rate-limiting issues with Yahoo Finance (with multiple consequetive runs, especially when running locally).

Whenever you add a new converter or create a fix for an existing one, please refer to the [Wiki](https://github.com/dickwolff/Export-To-Ghostfolio/wiki/Add-new-testdata-to-Yahoo-Finance-mock) for instructions on how to extend the mock with testdata.

17 changes: 17 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Create a docker compose file
services:
export-to-ghostfolio:
image: ghcr.io/juanmandev/export-to-ghostfolio:latest
ports:
- "3456:3001"
# Update 3456 with the port you want to use
volumes:
- ./.env:/app/.env
environment:
- GHOSTFOLIO_VALIDATE=true
# - GHOSTFOLIO_ACCOUNT_ID="YO"
# - DEGIRO_PREFERED_EXCHANGE_POSTFIX = ".AS"
# - XTB_ACCOUNT_CURRENCY = "EUR"
# - GHOSTFOLIO_URL="http://192.168.2.3:3333"
# - GHOSTFOLIO_SECRET =
# - GHOSTFOLIO_UPDATE_CASH=true
27 changes: 27 additions & 0 deletions nuxt.config.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
import { defineNuxtConfig } from 'nuxt/config';
import dotenv from 'dotenv';

dotenv.config();

// https://nuxt.com/docs/api/configuration/nuxt-config
export default defineNuxtConfig({
compatibilityDate: '2024-04-03',
modules: ['@nuxt/ui', '@nuxt/icon'],
ui: {
icons: ['heroicons', 'simple-icons']
},
runtimeConfig: {
public: {
GHOSTFOLIO_VALIDATE: process.env.GHOSTFOLIO_VALIDATE,
GHOSTFOLIO_IMPORT: process.env.GHOSTFOLIO_IMPORT,
GHOSTFOLIO_UPDATE_CASH: process.env.GHOSTFOLIO_UPDATE_CASH,
GHOSTFOLIO_ACCOUNT_ID: process.env.GHOSTFOLIO_ACCOUNT_ID,
GHOSTFOLIO_URL: process.env.GHOSTFOLIO_URL,
GHOSTFOLIO_SECRET: process.env.GHOSTFOLIO_SECRET,
serverUrl: process.env.NODE_ENV === 'production' ? '/' : 'http://localhost:3001/',
isDev: process.env.NODE_ENV !== 'production',
},
},
ssr: false,
target: 'static',
})
Loading