Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add public share results feature #20

Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
2c29db3
Add feature to allow public sharing of results:
smathews-techempower Jan 3, 2020
92b6132
Add share results page:
smathews-techempower Jan 21, 2020
2031a5c
Merge branch 'master' into add-public-share-results-feature
smathews-techempower Jan 21, 2020
96d5204
Fix errors from merge conflicts, add comments
smathews-techempower Jan 21, 2020
4a3f6e8
Add comments and use the fixed testMetadata key from results files
smathews-techempower Jan 21, 2020
e16c3ae
Add missing class-level javadoc
smathews-techempower Jan 21, 2020
9f24d97
Use lowerCamelCase for consistency with the other settings
smathews-techempower Jan 22, 2020
f212d02
- Revert the local port to 80 rather than 8080
smathews-techempower Jan 22, 2020
f97fadc
Imports: use the ordering and spacing rules from Google's Java style …
smathews-techempower Jan 22, 2020
4ce7191
Add final to classes where possible
smathews-techempower Jan 22, 2020
5f15213
Split the ShareResultsHandler into 2 separate handlers for GET/POST,
smathews-techempower Jan 22, 2020
9c94bea
Use Input/Output Streams instead of Readable/Writable Channels
smathews-techempower Jan 22, 2020
e283f11
Improve paste bin results sharing:
smathews-techempower Jan 24, 2020
50a98e2
Create dedicated FileUtils, avoid using java.io.File, and use FileSys…
smathews-techempower Jan 24, 2020
7e7ed26
Fix line break in comment at incorrect place
smathews-techempower Jan 24, 2020
1437142
Remove config.yml that was accidentally committed
smathews-techempower Jan 24, 2020
9e6408f
Add tests for new share results features
smathews-techempower Jan 29, 2020
aeb48e9
Make new share results test classes final
smathews-techempower Jan 29, 2020
e6d1053
Make new classes final, remove unused class
smathews-techempower Jan 29, 2020
e4d1c13
Clarify when testMetadata was added to results json
smathews-techempower Jan 29, 2020
def434e
Use random uuids for json files during testing to prevent conflicts b…
smathews-techempower Jan 30, 2020
4e3dbfe
Send an email once a day when the share directory is full and someone…
smathews-techempower Jan 30, 2020
e9e22ad
Ensure ShareResultsMailerTest the test is not run in parallel and add…
smathews-techempower Jan 30, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions src/main/java/tfb/status/bootstrap/ServicesBinder.java
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@
import tfb.status.service.ObjectMapperFactory;
import tfb.status.service.RunCompleteMailer;
import tfb.status.service.RunProgressMonitor;
import tfb.status.service.ShareResultsMailer;
import tfb.status.service.ShareResultsUploader;
import tfb.status.service.TaskScheduler;
import tfb.status.service.TickerFactory;
Expand Down Expand Up @@ -79,6 +80,7 @@ protected void configure() {
addActiveDescriptor(RunCompleteMailer.class);
addActiveDescriptor(TaskScheduler.class);
addActiveDescriptor(ShareResultsUploader.class);
addActiveDescriptor(ShareResultsMailer.class);
addActiveDescriptor(RootHandler.class);
addActiveDescriptor(HomePageHandler.class);
addActiveDescriptor(HomeUpdatesHandler.class);
Expand Down
24 changes: 20 additions & 4 deletions src/main/java/tfb/status/config/ApplicationConfig.java
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,12 @@ public final class ApplicationConfig {
@Provides
public final UrlsConfig urlsConfig;

/**
* See {@link ShareResultsMailerConfig}
*/
@Provides
public final ShareResultsMailerConfig shareResultsMailer;

/**
* The configuration for outbound emails, or {@code null} if outbound emails
* are disabled. See {@link EmailConfig}.
Expand All @@ -81,7 +87,8 @@ public ApplicationConfig(HttpServerConfig http,
RunProgressMonitorConfig runProgressMonitor,
RunCompleteMailerConfig runCompleteMailer,
@Nullable EmailConfig email,
UrlsConfig urlsConfig) {
UrlsConfig urlsConfig,
ShareResultsMailerConfig shareResultsMailer) {

this.http = Objects.requireNonNull(http);
this.assets = Objects.requireNonNull(assets);
Expand All @@ -91,6 +98,7 @@ public ApplicationConfig(HttpServerConfig http,
this.runCompleteMailer = Objects.requireNonNull(runCompleteMailer);
this.email = email;
this.urlsConfig = Objects.requireNonNull(urlsConfig);
this.shareResultsMailer = Objects.requireNonNull(shareResultsMailer);
}

@Override
Expand Down Expand Up @@ -150,7 +158,10 @@ public static ApplicationConfig create(
@Nullable EmailConfig email,

@JsonProperty(value = "urls", required = false)
@Nullable UrlsConfig urls) {
@Nullable UrlsConfig urls,

@JsonProperty(value = "shareResultsMailer", required = false)
@Nullable ShareResultsMailerConfig shareResultsMailer) {

return new ApplicationConfig(
/* http= */
Expand Down Expand Up @@ -189,10 +200,15 @@ public static ApplicationConfig create(
/* urlsConfig= */
Objects.requireNonNullElseGet(
urls,
() -> UrlsConfig.defaultConfig()));
() -> UrlsConfig.defaultConfig()),

/* shareResultsMailer= */
Objects.requireNonNullElseGet(
shareResultsMailer,
() -> ShareResultsMailerConfig.defaultConfig()));
}

public static ApplicationConfig defaultConfig() {
return create(null, null, null, null, null, null, null, null);
return create(null, null, null, null, null, null, null, null, null);
}
}
69 changes: 69 additions & 0 deletions src/main/java/tfb/status/config/ShareResultsMailerConfig.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
package tfb.status.config;

import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.google.errorprone.annotations.Immutable;
import java.time.Duration;
import java.util.Objects;
import javax.inject.Singleton;
import org.checkerframework.checker.nullness.qual.Nullable;

/**
* The configuration for the service that sends an emails related to sharing.
*/
@Immutable
@Singleton
public class ShareResultsMailerConfig {
/**
* The minimum number of seconds that must pass after sending one email before
* sending another. This is used to prevent accidental self-spam if there are
* a lot of requests to upload results when the share directory is full.
*/
public final long minSecondsBetweenDirectoryFullEmails;

public ShareResultsMailerConfig(long minSecondsBetweenDirectoryFullEmails) {
this.minSecondsBetweenDirectoryFullEmails =
minSecondsBetweenDirectoryFullEmails;
}

@Override
public boolean equals(@Nullable Object object) {
if (object == this) {
return true;
} else if (!(object instanceof ShareResultsMailerConfig)) {
return false;
} else {
ShareResultsMailerConfig that = (ShareResultsMailerConfig) object;
return this.minSecondsBetweenDirectoryFullEmails ==
that.minSecondsBetweenDirectoryFullEmails;
}
}

@Override
public int hashCode() {
int hash = 1;
hash = 31 * hash + Long.hashCode(minSecondsBetweenDirectoryFullEmails);
return hash;
}

@JsonCreator
public static ShareResultsMailerConfig create(
@JsonProperty(
value = "minSecondsBetweenDirectoryFullEmails",
required = false)
@Nullable Long minSecondsBetweenDirectoryFullEmails) {

return new ShareResultsMailerConfig(
/* minSecondsBetweenDirectoryFullEmails= */
Objects.requireNonNullElse(
minSecondsBetweenDirectoryFullEmails,
DEFAULT_MIN_SECONDS_BETWEEN_DIRECTORY_FULL_EMAILS));
}

public static ShareResultsMailerConfig defaultConfig() {
return create(null);
}

private static final long DEFAULT_MIN_SECONDS_BETWEEN_DIRECTORY_FULL_EMAILS =
Duration.ofDays(1).toSeconds();
}
97 changes: 97 additions & 0 deletions src/main/java/tfb/status/service/ShareResultsMailer.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
package tfb.status.service;

import com.google.common.annotations.VisibleForTesting;
import com.google.common.collect.ImmutableList;
import com.google.errorprone.annotations.concurrent.GuardedBy;
import java.time.Clock;
import java.time.Instant;
import java.util.Objects;
import javax.inject.Inject;
import javax.inject.Singleton;
import javax.mail.MessagingException;
import org.checkerframework.checker.nullness.qual.Nullable;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import tfb.status.config.ShareResultsMailerConfig;

@Singleton
public final class ShareResultsMailer {
private final Logger logger = LoggerFactory.getLogger(getClass());
private final Clock clock;
private final EmailSender emailSender;
private final ShareResultsMailerConfig config;

@GuardedBy("emailTimeLock")
private volatile @Nullable Instant previousEmailTime;
private final Object emailTimeLock = new Object();

@Inject
public ShareResultsMailer(Clock clock,
EmailSender emailSender,
ShareResultsMailerConfig config) {
this.clock = Objects.requireNonNull(clock);
this.emailSender = Objects.requireNonNull(emailSender);
this.config = Objects.requireNonNull(config);
}

public void onShareDirectoryFull(long capacityBytes, long sizeBytes) {
synchronized (emailTimeLock) {
Instant now = clock.instant();
Instant previous = this.previousEmailTime;

if (previous != null) {
Instant nextEmailTime =
previous.plusSeconds(config.minSecondsBetweenDirectoryFullEmails);

if (now.isBefore(nextEmailTime)) {
logger.warn(
"Suppressing email for full share directory, "
+ "another email was sent for that account too recently, "
+ "previous email time = {}, next possible email time = {}",
previous,
nextEmailTime);
return;
}
}

this.previousEmailTime = now;
}

sendShareDirectoryFullEmail(capacityBytes, sizeBytes);
}

private void sendShareDirectoryFullEmail(long capacityBytes, long sizeBytes) {
try {
emailSender.sendEmail(
/* subject= */ SHARE_DIRECTORY_FULL_SUBJECT,
/* textContent= */
composeShareDirectoryFullEmail(capacityBytes, sizeBytes),
/* attachments= */ ImmutableList.of());
} catch (MessagingException e) {
logger.warn("Error sending email for share directory full", e);
}
}

private String composeShareDirectoryFullEmail(long capacityBytes,
long sizeBytes) {
return "Hello,"
+ "\n"
+ "\n"
+ "The share directory used for storing public uploads of "
+ "results files has reached capacity. Please audit the "
+ "directory and delete old uploads or expand the configured "
+ "capacity."
+ "\n"
+ "\n"
+ "Share directory capacity: " + capacityBytes + " bytes"
+ "\n"
+ "Share directory size: " + sizeBytes + " bytes"
+ "\n"
+ "\n"
+ "-a robot";
}

@VisibleForTesting
public static final String SHARE_DIRECTORY_FULL_SUBJECT =
"<tfb> <auto> Share directory full";
}
7 changes: 6 additions & 1 deletion src/main/java/tfb/status/service/ShareResultsUploader.java
Original file line number Diff line number Diff line change
Expand Up @@ -48,16 +48,19 @@ public final class ShareResultsUploader {
private final UrlsConfig urlsConfig;
private final FileStore fileStore;
private final ObjectMapper objectMapper;
private final ShareResultsMailer shareResultsMailer;

@Inject
public ShareResultsUploader(FileStoreConfig fileStoreConfig,
UrlsConfig urlsConfig,
FileStore fileStore,
ObjectMapper objectMapper) {
ObjectMapper objectMapper,
ShareResultsMailer shareResultsMailer) {
this.fileStoreConfig = Objects.requireNonNull(fileStoreConfig);
this.urlsConfig = Objects.requireNonNull(urlsConfig);
this.fileStore = Objects.requireNonNull(fileStore);
this.objectMapper = Objects.requireNonNull(objectMapper);
this.shareResultsMailer = Objects.requireNonNull(shareResultsMailer);
}

/**
Expand Down Expand Up @@ -191,6 +194,8 @@ public ShareResultsUploadReport upload(Path tempFile) throws IOException {
long shareDirectorySize = FileUtils.directorySizeBytes(
fileStore.shareDirectory());
if (shareDirectorySize >= fileStoreConfig.maxShareDirectorySizeBytes) {
shareResultsMailer.onShareDirectoryFull(
fileStoreConfig.maxShareDirectorySizeBytes, shareDirectorySize);
return "Share uploads has reached max capacity.";
}

Expand Down
51 changes: 51 additions & 0 deletions src/test/java/tfb/status/service/ShareResultsMailerTest.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
package tfb.status.service;

import static org.junit.jupiter.api.Assertions.assertEquals;

import com.google.common.collect.ImmutableList;
import java.io.IOException;
import java.util.List;
import javax.mail.MessagingException;
import javax.mail.internet.MimeMessage;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import tfb.status.testlib.MailServer;
import tfb.status.testlib.TestServicesInjector;

/**
* Tests for {@link ShareResultsMailer}.
*/
@ExtendWith(TestServicesInjector.class)
public final class ShareResultsMailerTest {
/**
* Verifies that the mailer sends an email with the expected title when the
* {@link ShareResultsMailer#onShareDirectoryFull(long, long)} method is
* invoked.
*/
@Test
public void shareResultsMailer_shareDirectoryFullEmail(
MailServer mailServer,
ShareResultsMailer shareResultsMailer)
throws IOException, MessagingException {

// The mailer is a singleton and may have been called by other tests
// indirectly. We just need to make sure it sent an email as a result of
// this test.
List<MimeMessage> messages =getShareDirectoryFullEmails(mailServer);
int initialMessagesSize = messages.size();

shareResultsMailer.onShareDirectoryFull(1234, 5678);

messages = getShareDirectoryFullEmails(mailServer);

assertEquals(initialMessagesSize + 1, messages.size());
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This works now, but may cause an issue if you make the tests run in parallel. One of the other tests tries to share a results file after filling the share directory to capacity, which also sends this email. I'm curious what the right solution would be. One idea is that since the email contains the capacity and size in the body, I could add code to find a message with the specific numbers passed to onShareDirectoryFull above.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's not solve this now, but please annotate the test so that it can't run in parallel and add a TODO near the part of the test that's unfriendly to parallelism.

}

private static ImmutableList<MimeMessage> getShareDirectoryFullEmails(
MailServer mailServer) throws IOException, MessagingException {

return mailServer.getMessages(
m -> m.getSubject().equals(
ShareResultsMailer.SHARE_DIRECTORY_FULL_SUBJECT));
}
}
3 changes: 3 additions & 0 deletions src/test/resources/test_config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -32,3 +32,6 @@ runCompleteMailer:
urls:
tfbStatus: https://test.tfb-status.techempower.com
teWeb: https://www.test.techempower.com

shareResultsMailer:
minSecondsBetweenDirectoryFullEmails: 0