Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cache dump to disk #41

Open
wants to merge 122 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
122 commits
Select commit Hold shift + click to select a range
ed2b4bb
Client now properly emits ready event on hot reload
Ortovoxx Mar 11, 2021
0952695
Added directory property and added all session cached to folder which…
Ortovoxx Mar 12, 2021
3e1bde3
Creates folder if it does not exist
Ortovoxx Mar 12, 2021
fe0b745
Added hot reloading explanation to README
Ortovoxx Mar 12, 2021
f4c66ea
Added temporary file loading and parsing. Fails with roles
Ortovoxx Mar 12, 2021
1ee78f0
Added _unpatch method to guild class
Ortovoxx Mar 21, 2021
c1a6254
Added new client options
Ortovoxx Mar 21, 2021
37a23c7
Greatly simplified session and sequence client option handling
Ortovoxx Mar 21, 2021
9285dc3
Users can specify the exit events they desire or use default ones
Ortovoxx Mar 21, 2021
79644c4
Added uncaughtExceptionOnExit bool to stop loop of uncaughExceptions
Ortovoxx Mar 21, 2021
84afdb3
Added dumpCache method with sessions and client args
Ortovoxx Mar 21, 2021
382516e
Assigns new shards to hotreload and calls dumpCache asynchronously if…
Ortovoxx Mar 21, 2021
dc31079
dumpCache sync file storing function
Ortovoxx Mar 21, 2021
329bff9
Restores guilds and users
Ortovoxx Mar 21, 2021
f1f9161
Added restore cache client options
Ortovoxx Mar 21, 2021
e9770e7
Update README.md
Ortovoxx Mar 21, 2021
1eec95b
Updated client options and added typings
Ortovoxx Mar 21, 2021
ea895b6
Added new typedefs
Ortovoxx Mar 21, 2021
a8be027
Added djs-light validate options
Ortovoxx Mar 21, 2021
24fc69b
Added change for sessionData
Ortovoxx Mar 21, 2021
b9a1681
Reads session files from .sessions/sessions/{shardID}.json
Ortovoxx Mar 21, 2021
bb43047
Extra checks for validation of options and added back events
Ortovoxx Mar 21, 2021
3bc84f4
Created _loadSesssions to load sessions from disk
Ortovoxx Mar 21, 2021
63e6929
Added helper methods
Ortovoxx Mar 21, 2021
b3afb40
Added dumpCache and patchCache methods
Ortovoxx Mar 21, 2021
ca7ebd1
Added a _write method to write an array of cache data to disk
Ortovoxx Mar 22, 2021
6a915d2
Implemented dumpCache
Ortovoxx Mar 22, 2021
7fffe3b
Added better error handling for infinite uncaught exception loop
Ortovoxx Mar 22, 2021
176ae7b
Client options
Ortovoxx Mar 22, 2021
7dced34
implement _unpatch()
timotejroiko Mar 31, 2021
75f15da
Channel unpatching
timotejroiko Mar 31, 2021
c12961c
message unpatch plus fixes
timotejroiko Mar 31, 2021
422857d
wip: needs rethinking
timotejroiko Apr 1, 2021
fa3f48c
Linting
Ortovoxx Apr 1, 2021
ea0b8ee
Try catch around user supplied function
Ortovoxx Apr 1, 2021
7838707
prevent exit events from bleeding into each other
timotejroiko Apr 1, 2021
2ce1320
store caches
timotejroiko Apr 1, 2021
3ff1f9a
unpatch channels in the guild object
timotejroiko Apr 1, 2021
ff30a01
Small syntax fixes
Ortovoxx Apr 1, 2021
3713d0c
rework cache loading and storing
timotejroiko Apr 2, 2021
1655dd6
load cache on identify
timotejroiko Apr 2, 2021
50b1d95
fix conflict
timotejroiko Apr 2, 2021
652635b
Small fixes
Ortovoxx Apr 2, 2021
f53ae0c
readme
timotejroiko Apr 2, 2021
8d5f0c6
Added a last connected property to session data to avoid unnecessary …
Ortovoxx Apr 2, 2021
658c0ac
Updated cache and session types. Maybe remove from readme
Ortovoxx Apr 2, 2021
848658e
Small fixes and checks
Ortovoxx Apr 2, 2021
86fb045
Fixed _loadSession + now returns data in line with user supplied data
Ortovoxx Apr 2, 2021
accb618
Made data more consistent
Ortovoxx Apr 2, 2021
68634f4
Identify without loading cache if creating a new session
Ortovoxx Apr 2, 2021
dc630ac
Update init.js
Ortovoxx Apr 2, 2021
fa814e3
Moved loading of cache into resume event
Ortovoxx Apr 2, 2021
27f3023
Temporary fix until we load client user
Ortovoxx Apr 2, 2021
49f27b3
Checks the close sequence of the last shard and doesn't wait if it ex…
Ortovoxx Apr 3, 2021
d0db3c6
Moved session data to createShards
Ortovoxx Apr 3, 2021
b9d28ab
Stores shardCount in session object
Ortovoxx Apr 3, 2021
d806d78
shards.size -> totalShards
Ortovoxx Apr 3, 2021
57c8601
Moved cache patching and added 15s timeout for resuming
Ortovoxx Apr 3, 2021
48ab68f
Event listener removed
Ortovoxx Apr 3, 2021
c9542db
Client now properly emits ready event on hot reload
Ortovoxx Mar 11, 2021
a5b2546
Added directory property and added all session cached to folder which…
Ortovoxx Mar 12, 2021
84d1aef
Creates folder if it does not exist
Ortovoxx Mar 12, 2021
4966abf
Added hot reloading explanation to README
Ortovoxx Mar 12, 2021
f6f17d7
Added temporary file loading and parsing. Fails with roles
Ortovoxx Mar 12, 2021
51523a6
Added _unpatch method to guild class
Ortovoxx Mar 21, 2021
bac0008
Added new client options
Ortovoxx Mar 21, 2021
6129fbd
Greatly simplified session and sequence client option handling
Ortovoxx Mar 21, 2021
10e9d7f
Users can specify the exit events they desire or use default ones
Ortovoxx Mar 21, 2021
e5d0a1e
Added uncaughtExceptionOnExit bool to stop loop of uncaughExceptions
Ortovoxx Mar 21, 2021
efc65ee
Added dumpCache method with sessions and client args
Ortovoxx Mar 21, 2021
3fb04d1
Assigns new shards to hotreload and calls dumpCache asynchronously if…
Ortovoxx Mar 21, 2021
bc7900a
dumpCache sync file storing function
Ortovoxx Mar 21, 2021
4fd675b
Restores guilds and users
Ortovoxx Mar 21, 2021
81109ca
Added restore cache client options
Ortovoxx Mar 21, 2021
f66811b
Update README.md
Ortovoxx Mar 21, 2021
b226bb8
Updated client options and added typings
Ortovoxx Mar 21, 2021
10e41d8
Added new typedefs
Ortovoxx Mar 21, 2021
d3a29aa
Added djs-light validate options
Ortovoxx Mar 21, 2021
b0ef952
Added change for sessionData
Ortovoxx Mar 21, 2021
86ba965
Reads session files from .sessions/sessions/{shardID}.json
Ortovoxx Mar 21, 2021
019a28c
Extra checks for validation of options and added back events
Ortovoxx Mar 21, 2021
49777bd
Created _loadSesssions to load sessions from disk
Ortovoxx Mar 21, 2021
09940cc
Added helper methods
Ortovoxx Mar 21, 2021
bac06e4
Added dumpCache and patchCache methods
Ortovoxx Mar 21, 2021
5a7ec3c
Added a _write method to write an array of cache data to disk
Ortovoxx Mar 22, 2021
4cc9bad
Implemented dumpCache
Ortovoxx Mar 22, 2021
a526fe7
Added better error handling for infinite uncaught exception loop
Ortovoxx Mar 22, 2021
5c3234e
Client options
Ortovoxx Mar 22, 2021
39f3322
implement _unpatch()
timotejroiko Mar 31, 2021
35bd529
Channel unpatching
timotejroiko Mar 31, 2021
6fa4af2
message unpatch plus fixes
timotejroiko Mar 31, 2021
4de1442
wip: needs rethinking
timotejroiko Apr 1, 2021
7be37e3
Linting
Ortovoxx Apr 1, 2021
00d6e3c
Try catch around user supplied function
Ortovoxx Apr 1, 2021
0d6ef2d
prevent exit events from bleeding into each other
timotejroiko Apr 1, 2021
996c5d6
store caches
timotejroiko Apr 1, 2021
46fb575
unpatch channels in the guild object
timotejroiko Apr 1, 2021
f233e03
Small syntax fixes
Ortovoxx Apr 1, 2021
c5c3dbc
rework cache loading and storing
timotejroiko Apr 2, 2021
52efd4c
load cache on identify
timotejroiko Apr 2, 2021
984a73b
fix conflict
timotejroiko Apr 2, 2021
7ea9e94
Small fixes
Ortovoxx Apr 2, 2021
36236e7
readme
timotejroiko Apr 2, 2021
523a463
Added a last connected property to session data to avoid unnecessary …
Ortovoxx Apr 2, 2021
0d7bd76
Updated cache and session types. Maybe remove from readme
Ortovoxx Apr 2, 2021
c6b21de
Small fixes and checks
Ortovoxx Apr 2, 2021
0ce72a1
Fixed _loadSession + now returns data in line with user supplied data
Ortovoxx Apr 2, 2021
c91567b
Made data more consistent
Ortovoxx Apr 2, 2021
c966058
Identify without loading cache if creating a new session
Ortovoxx Apr 2, 2021
76c52aa
Update init.js
Ortovoxx Apr 2, 2021
74868b9
Moved loading of cache into resume event
Ortovoxx Apr 2, 2021
f9cf35b
Temporary fix until we load client user
Ortovoxx Apr 2, 2021
8189e36
Checks the close sequence of the last shard and doesn't wait if it ex…
Ortovoxx Apr 3, 2021
28a3e19
Moved session data to createShards
Ortovoxx Apr 3, 2021
9f7df3b
Stores shardCount in session object
Ortovoxx Apr 3, 2021
46d3d2e
shards.size -> totalShards
Ortovoxx Apr 3, 2021
fd20123
Moved cache patching and added 15s timeout for resuming
Ortovoxx Apr 3, 2021
f738d16
Event listener removed
Ortovoxx Apr 3, 2021
0627789
Cleanup
Ortovoxx Apr 3, 2021
f37f68d
Merge branch 'cache-dump' of https://github.com/Ortovoxx/discord.js-l…
Ortovoxx Apr 3, 2021
b4ee875
Fixes from rebase
Ortovoxx Apr 3, 2021
5dafdcd
Merge branch 'master' into cache-dump
timotejroiko Apr 3, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .eslintrc.json
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,7 @@
"prefer-numeric-literals":"error",
"prefer-spread":"error",
"prefer-template":"error",
"symbol-description":"error"
"symbol-description":"error",
"no-func-loop": false
}
}
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,7 @@ package-lock.json
.vscode
.eslintrc.json
.gitattributes
.pnp.js
.yarnrc.yml
yarn.lock
.sessions
39 changes: 39 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,7 @@ The following client options are available to control caching behavior:
| cacheEmojis | boolean | false | Enables caching of all Emojis at login |
| cachePresences | boolean | false | Enables caching of all Presences. If not enabled, Presences will be cached only for cached Users |
| cacheMembers | boolean | false | Enables caching of Users and Members when possible |
| hotReload | boolean or object | false | Enables hot reloading, an experimental feature that enables instantly restarting the bot |
| disabledEvents | array | [] | An array of events to ignore ([Discord events](https://github.com/discordjs/discord.js/blob/master/src/util/Constants.js#L339), not Discord.JS events). Use this in combination with intents for fine tuning which events your bot should process |

This library implements its own partials system, therefore the `partials` client option is not available. All other discord.js client options continue to be available and should work normally.
Expand Down Expand Up @@ -148,6 +149,44 @@ Voice States will be cached if the `GUILD_VOICE_STATES` intent is enabled (requi

Messages are cached only if the Channel they belong to is cached. Message caching can further be controlled via discord.js's `messageCacheMaxSize`, `messageCacheLifetime` and `messageSweepInterval` client options as usual. Additionally, the `messageEditHistoryMaxSize` client option is set to `1` by default (instead of infinity).

## Hot reloading

**THIS FEATURE IS CURRENTLY EXPERIMENTAL USE AT YOUR OWN RISK!**

When developing bots you will likely do lots of trial and error which often requires restarting your bot.
Each restart requires reconnecting to the Discord gateway on every shard which can often take a long time and eat up your daily identifies.

Hot reloading provides a way to simply resume the previous gateway sessions on startup rather than creating new ones.
This is done by storing the process data outside the process right before it exits and reloading the data into it when it starts.

This option can be overridden with a few custom methods:

```js
const cache = {
guilds: {
"581072557512458241": {} // Discord api type
}
}

const sessions = {
"0": {
id: "3e961ec59a7c24b91198d07dd3189fa0", // Session ID
sequence: 19, // The close sequence of the session
lastConnected: 1617382560867 // Time the session was closed at
}
}

new Discord.Client({
hotReload: {
cacheData: cache, // user-supplied cache data. if not present, cache will be loaded from disk
sessionData: sessions, // user-supplied session data. if not present, sessions will be loaded from disk
onExit: (sessions, cache) => {} // user-supplied data storing function. if not present, data will be stored to disk. This function can be async if the process is exited by using SIGINT (Ctrl+C), any other ways of exit are sync-only.
}
})
```

Session and cache data can also be obtained at runtime using `client.dumpCache()` and `client.dumpSessions()` for storage before a manually induced exit. In this case the user should pass an empty function to onExit to override the built-in disk storage.

## Events

Most events should be identical to the originals aside from the caching behavior plus they always emit regardless of caching state. When required data is missing, a partial structure where only an id is guaranteed will be given (the `.partial` property is not guaranteed to exist on all partials).
Expand Down
232 changes: 230 additions & 2 deletions classes.js
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,81 @@ Discord.Structures.extend("Message", M => {
}
}
}
_unpatch() {
return {
id: this.id,
type: Discord.Constants.MessageTypes.indexOf(this.type),
content: this.content,
author: this.author._unpatch(),
pinned: this.pinned,
tts: this.tts,
nonce: this.nonce,
embeds: this.embeds.map(x => x.toJSON()),
attachments: this.attachments.map(x => ({
filename: x.name,
id: x.id,
size: x.size,
url: x.url,
proxy_url: x.proxyURL,
height: x.height,
width: x.width
})),
edited_timestamp: this.editedTimestamp,
reactions: this.reactions.cache.map(x => ({
me: x.me,
emoji: {
animated: x.emoji.animated,
name: x.emoji.name,
id: x.emoji.id
},
count: x.count
})),
mentions: this.mentions.users.map(x => x._unpatch()),
mention_roles: this.mentions.roles.map(x => x._unpatch()),
mention_everyone: this.mentions.everyone,
mention_channels: this.mentions.crosspostedChannels.map(x => ({
id: x.channelID,
guild_id: x.guildID,
type: Discord.Constants.ChannelTypes[x.type.toUpperCase()],
name: x.name
})),
webhook_id: this.webhookID,
application: this.application ? {
id: this.application.id,
name: this.application.name,
description: this.application.description,
icon: this.application.icon,
cover_image: this.application.cover,
rpc_origins: this.application.rpcOrigins,
bot_require_code_grant: this.botRequireCodeGrant,
bot_public: this.application.botPublic,
team: this.application.owner instanceof Discord.Team ? {
id: this.application.owner.id,
name: this.application.owner.name,
icon: this.application.owner.icon,
owner_user_id: this.application.owner.ownerID,
members: this.application.owner.members.map(x => ({
permissions: x.permissions,
membership_state: x.membershipState,
user: x.user._unpatch()
}))
} : void 0,
owner: this.application.owner instanceof Discord.User ? this.application.owner._unpatch() : void 0
} : void 0,
activity: this.activity ? {
party_id: this.activity.partyID,
type: this.activity.type
} : void 0,
member: this.member?._unpatch(),
flags: this.flags.valueOf(),
message_reference: this.reference ? {
channel_id: this.reference.channelID,
guild_id: this.reference.guildID,
message_id: this.reference.messageID
} : void 0,
referenced_message: this.client.guilds.cache.get(this.reference?.guildID)?.channels.cache.get(this.reference?.channelID)?.messages.cache.get(this.reference?.messageID)?._unpatch()
};
}
get member() {
if(!this.guild) { return null; }
const id = (this.author || {}).id || (this._member || {}).id;
Expand Down Expand Up @@ -120,6 +195,22 @@ Discord.Structures.extend("Message", M => {
};
});

Discord.Structures.extend("User", U => {
return class User extends U {
_unpatch() {
return {
id: this.id,
username: this.username,
bot: this.bot,
discriminator: this.discriminator,
avatar: this.avatar,
system: this.system,
public_flags: this.flags?.valueOf()
};
}
};
});

Discord.Structures.extend("GuildMember", G => {
return class GuildMember extends G {
_patch(data) {
Expand All @@ -138,6 +229,16 @@ Discord.Structures.extend("GuildMember", G => {
}
}
}
_unpatch() {
return {
user: this.user._unpatch(),
nick: this.nickname,
joined_at: this.joinedTimestamp,
premium_since: this.premiumSinceTimestamp,
roles: this._roles,
pending: this.pending
};
}
equals(member) {
return member && this.deleted === member.deleted && this.nickname === member.nickname && this._roles.length === member._roles.length;
}
Expand Down Expand Up @@ -186,8 +287,8 @@ Discord.Structures.extend("Guild", G => {
this.members.add(member);
}
}
if(!this.members.cache.has(this.client.user.id)) {
this.members.fetch(this.client.user.id).catch(() => {});
if(!this.members.cache.has(this.client.user?.id)) {
Copy link
Owner

@timotejroiko timotejroiko Apr 3, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a problem: optional chaining short circuits to undefined so if client.user is falsey, then its doing .has(undefined) which will count as user not cached and attempt to fetch(undefined).

There shouldnt ever be a situation where a guild is loaded without the client having logged in, especially now that you moved the cache loading to the resume event.

this.members.fetch(this.client.user?.id).catch(() => {});
}
}
if(Array.isArray(data.presences)) {
Expand Down Expand Up @@ -216,6 +317,52 @@ Discord.Structures.extend("Guild", G => {
}
}
}
_unpatch() {
return {
unavailable: !this.available,
shardID: this.shardID,
name: this.name,
icon: this.icon,
splash: this.splash,
discovery_splash: this.discoverySplash,
region: this.region,
member_count: this.memberCount,
large: this.large,
features: this.features,
application_id: this.applicationID,
afk_timeout: this.afkTimeout,
afk_channel_id: this.afkChannelID,
system_channel_id: this.systemChannelID,
premium_tier: this.premiumTier,
premium_subscription_count: this.premiumSubscriptionCount,
widget_enabled: this.widgetEnabled,
widget_channel_id: this.widgetChannelID,
verification_level: Discord.Constants.VerificationLevels.indexOf(this.verificationLevel),
explicit_content_filter: Discord.Constants.ExplicitContentFilterLevels.indexOf(this.explicitContentFilter),
mfa_level: this.mfaLevel,
joinedTimestamp: this.joinedTimestamp,
default_message_notifications: this.defaultMessageNotifications,
system_channel_flags: this.systemChannelFlags?.valueOf(),
max_members: this.maximumMembers,
max_presences: this.maximumPresences,
approximate_member_count: this.approximateMemberCount,
approximate_presence_count: this.approximatePresenceCount,
vanity_url_code: this.vanityURLCode,
description: this.description,
banner: this.banner,
id: this.id,
rules_channel_id: this.rulesChannelID,
public_updates_channel_id: this.publicUpdatesChannelID,
preferred_locale: this.preferredLocale,
roles: this.roles.cache.map(x => x._unpatch()),
members: this.members.cache.map(x => x._unpatch()),
channels: this.channels.cache.map(x => x._unpatch()),
owner_id: this.ownerID,
presences: this.presences.cache.map(x => x._unpatch()),
voice_states: this.voiceStates.cache.map(x => x._unpatch()),
emojis: this.emojis?.cache.map(x => x._unpatch())
};
}
get nameAcronym() {
return this.name ? super.nameAcronym : void 0;
}
Expand Down Expand Up @@ -275,6 +422,18 @@ Discord.Structures.extend("GuildEmoji", E => {
this._author = data.user.id;
}
}
_unpatch() {
return {
animated: this.animated,
name: this.name,
id: this.id,
require_colons: this.requiresColons,
managed: this.managed,
available: this.available,
roles: this._roles,
user: this.author ? this.author._unpatch() : void 0
};
}
async fetchAuthor(cache = true) {
if(this.managed) {
throw new Error("EMOJI_MANAGED");
Expand All @@ -291,6 +450,28 @@ Discord.Structures.extend("GuildEmoji", E => {
};
});

Discord.Structures.extend("Role", R => {
return class Role extends R {
_unpatch() {
return {
id: this.id,
name: this.name,
color: this.color,
hoist: this.hoist,
position: this.rawPosition,
permissions: this.permissions.valueOf().toString(),
managed: this.managed,
mentionable: this.mentionable,
tags: {
bot_id: this.tags?.botID,
integration_id: this.tags?.integrationID,
premium_subscriber: this.tags?.premiumSubscriberRole
}
};
}
};
});

Discord.Structures.extend("VoiceState", V => {
return class VoiceState extends V {
_patch(data) {
Expand All @@ -300,6 +481,19 @@ Discord.Structures.extend("VoiceState", V => {
}
return this;
}
_unpatch() {
return {
user_id: this.id,
deaf: this.serverDeaf,
mute: this.serverMute,
self_deaf: this.selfDeaf,
self_mute: this.selfMute,
self_video: this.selfVideo,
session_id: this.sessionID,
self_stream: this.streaming,
channel_id: this.channelID
};
}
get channel() {
return this.channelID ? this.client.channels.cache.get(this.channelID) || this.client.channels.add({
id: this.channelID,
Expand Down Expand Up @@ -354,6 +548,40 @@ Discord.Structures.extend("Presence", P => {
}
return this;
}
_unpatch() {
return {
user: { id: this.userID },
status: this.status,
activities: this.activities.map(a => ({
name: a.name,
type: a.type,
url: a.url,
details: a.details,
state: a.state,
application_id: a.applicationID,
timestamps: a.timestamps ? {
start: a.timestamps.start ? a.timestamps.start.getTime() : null,
end: a.timestamps.end ? a.timestamps.end.getTime() : null
} : void 0,
party: a.party,
assets: a.assets ? {
large_text: a.assets.largeText,
small_text: a.assets.smallText,
large_image: a.assets.largeImage,
small_image: a.assets.smallImage
} : void 0,
sync_id: a.syncID,
flags: a.flags.valueOf(),
emoji: a.emoji ? {
animated: a.emoji.animated,
name: a.emoji.name,
id: a.emoji.id
} : void 0,
created_at: a.createdTimestamp
})),
client_status: this.clientStatus
};
}
get user() {
return this.client.users.cache.get(this.userID) || this.client.users.add((this._member || {}).user || { id: this.userID }, false);
}
Expand Down
Loading