Skip to content
This repository has been archived by the owner on Aug 23, 2020. It is now read-only.

Commit

Permalink
threadsafe Hash (#597)
Browse files Browse the repository at this point in the history
(1) Made Hash thread-safe without needing full synchronization.

All access to `bytes`, `trits` and `hashcode` done via a null check against an inner class object that is instantiated in a synchronized block with a double check. `ByteSafe` and `TritSafe` are inner classes and the data hidden in them is not available until the classes are fully instantiated (classloading ensures this).

(2)  The (1) above fixes the race condition to `hashcode()` and `equals()` and `bytes()` that was proven to be happening and might also have cause the FIFO cache overrun.

(3) Fixed `compareTo` to do an identity check and not need to make a new `Hash` every time - because Hash extends Indexable and the the indexible provided could already be a Hash.

(4) Commented on bounds check. This is easy to fix but so many tests are abusing it and they won't pass until the tests are also fixed.

(5) Added a null check in `fullread' because the RocksDB does provide null byte[] to read methods sometimes and they can be safely passed over.

(6) Added a 2nd time `read` persistence check. The class should be effectively immutable (not possible) but at least check to see if it has already been initialized and reject a second initialization.

(7) No other classes extend `Hash` . I have made it `final` and that can be changed in the future if it ever needs to be extended.

All tests pass.
Live run passed.

Signed-off-by: footloosejava <[email protected]>
  • Loading branch information
footloosejava authored and alon-e committed Apr 11, 2018
1 parent 6267c83 commit 84cc51f
Showing 1 changed file with 99 additions and 60 deletions.
159 changes: 99 additions & 60 deletions src/main/java/com/iota/iri/model/Hash.java
Original file line number Diff line number Diff line change
Expand Up @@ -8,51 +8,39 @@

import java.io.Serializable;
import java.util.Arrays;
import java.util.Objects;

public class Hash implements Serializable, Indexable {
public final class Hash implements Serializable, Indexable {

public static final int SIZE_IN_TRITS = 243;
public static final int SIZE_IN_BYTES = 49;

public static final Hash NULL_HASH = new Hash(new int[Curl.HASH_LENGTH]);

private byte[] bytes;
private int[] trits;
private int hashCode;

// constructors' bill
private final Object lock = new Object();
private ByteSafe byteSafe;
private TritSafe tritSafe;

This comment has been minimized.

Copy link
@schierlm

schierlm Apr 24, 2018

These variables should be volatile so that double checked locking works as intended? In case you don't care about races constructing two objects, I'd remove the second check inside the lock to make this clearer.

This comment has been minimized.

Copy link
@GalRogozinski

GalRogozinski via email Apr 25, 2018

Contributor

This comment has been minimized.

Copy link
@schierlm

schierlm Apr 25, 2018

I was talking about ByteSafe and TritSafe, which are not final. As they are not volatile, the compiler may optimize the second read inside the synchronized block to happen outside, therefore making the second read "ineffective".

This comment has been minimized.

Copy link
@GalRogozinski

GalRogozinski May 5, 2018

Contributor

@schierlm
The fields are immutable, thus the object is immutable (from the JMM perspective), and the instance of ByteSafe/TritSafe won't be visible to other threads until fully constructed.

Notice the use of the thread-local safe variable. It is updated inside the synchronized block. We also use it when we return.

This comment has been minimized.

Copy link
@schierlm

schierlm May 5, 2018

Consider two threads on two cores calling the trits() method at the same time while tritSafe is null. First thread enters synchronized block and initializes the variable. Second thread initializes local variable in line 116 with null (put in processor's memory cache) and waits at synchronized block for first thread. Once entering synchronized block, compiler is free to optimize the call in line 119 to access the variable in memory cache (still there from line 116) without using memory barrier. So second thread still sees null and reinitializes the variable.

In other words entering a synchronized block does not involve a memory barrier, only exiting it. So when you access a variable before the synchronized block and inside it, and you want to make sure that double-checked locking works, you need to declare it volatile. In case you don't care (performance is more important than correctness), you can remove the second check.

This comment has been minimized.

Copy link
@GalRogozinski

GalRogozinski May 5, 2018

Contributor

@schierlm
From JSR-133 (https://www.cs.umd.edu/~pugh/java/memoryModel/jsr133.pdf):

Final fields also allow programmers to implement thread-safe immutable objects without synchronization.
A thread-safe immutable object is seen as immutable by all threads, even if a data
race is used to pass references to the immutable object between threads. This can provide safety
guarantees against misuse of an immutable class by incorrect or malicious code.

So the fact that TritsSafe has final fields suffices imho.
Also note the examples (scroll down in both):
https://wiki.sei.cmu.edu/confluence/display/java/LCK10-J.+Use+a+correct+form+of+the+double-checked+locking+idiom
And
https://en.wikipedia.org/wiki/Double-checked_locking#Usage_in_Java

If what you are saying is true that means that it contradicts the official JSR and the 2 reputable sources I linked to.

This comment has been minimized.

Copy link
@GalRogozinski

GalRogozinski May 5, 2018

Contributor

Oh and according JSR-133 FAQ

Before we can enter a synchronized block, we acquire the monitor, which has the effect of invalidating the local processor cache so that variables will be reloaded from main memory. We will then be able to see all of the writes made visible by the previous release.

This comment has been minimized.

Copy link
@schierlm

schierlm May 6, 2018

So why does https://www.cs.umd.edu/~pugh/java/memoryModel/jsr-133-faq.html#dcl state that double checked locking is broken without volatile then? Seems to me that the page contradicts itself... Or does that only apply to mutable objects?

On the other hand, that valid example with all final fields on the CMU wiki is also new to me, Do you know if that idiom is valid since Java 1.5 (JSR 133) or whether that constraint was introduced later? TBH I did not really follow memory model changes after Java 1.5, so sorry if that is all outdated now :-(

Anyway, I will withdraw my complaints since obviously it is not as simple as I believed it to be :-(


public Hash(final byte[] bytes, final int offset, final int size) {
fullRead(bytes, offset, size);
}

public Hash(){}
private final class ByteSafe {
private final byte[] bytes;
private final Integer hashcode;

public Hash(final byte[] bytes) {
this(bytes, 0, SIZE_IN_BYTES);
private ByteSafe(byte[] bytes) {
Objects.requireNonNull(bytes, "ByteSafe is attempted to be initialized with a null byte array");
this.bytes = bytes;
this.hashcode = Arrays.hashCode(bytes);
}
}

public Hash(final int[] trits, final int offset) {
this.trits = new int[SIZE_IN_TRITS];
System.arraycopy(trits, offset, this.trits, 0, SIZE_IN_TRITS);
//this(Converter.bytes(trits, offset, trits.length));
}
private final class TritSafe {
private final int[] trits;

public Hash(final int[] trits) {
this(trits, 0);
private TritSafe(int[] trits) {
this.trits = Objects.requireNonNull(trits, "TritSafe is attempted to be initialized with a null int array");
}
}

public Hash(final String trytes) {
this.trits = new int[SIZE_IN_TRITS];
Converter.trits(trytes, this.trits, 0);
}

//
/*
public static Hash calculate(byte[] bytes) {
return calculate(bytes, SIZE_IN_TRITS, new Curl());
}
*/
public static Hash calculate(SpongeFactory.Mode mode, int[] trits) {
return calculate(trits, 0, trits.length, SpongeFactory.create(mode));
}
Expand All @@ -62,6 +50,7 @@ public static Hash calculate(byte[] bytes, int tritsLength, final Sponge curl) {
Converter.getTrits(bytes, trits);
return calculate(trits, 0, tritsLength, curl);
}

public static Hash calculate(final int[] tritsToCalculate, int offset, int length, final Sponge curl) {
int[] hashTrits = new int[SIZE_IN_TRITS];
curl.reset();
Expand All @@ -70,24 +59,74 @@ public static Hash calculate(final int[] tritsToCalculate, int offset, int lengt
return new Hash(hashTrits);
}


public Hash() {
}


public Hash(final byte[] source, final int sourceOffset, final int sourceSize) {
byte[] dest = new byte[SIZE_IN_BYTES];
System.arraycopy(source, sourceOffset, dest, 0, sourceSize - sourceOffset > source.length ? source.length - sourceOffset : sourceSize);
this.byteSafe = new ByteSafe(dest);
}

public Hash(final byte[] bytes) {
this(bytes, 0, SIZE_IN_BYTES);
}

public Hash(final int[] trits, final int offset) {
int[] dest = new int[SIZE_IN_TRITS];
System.arraycopy(trits, offset, dest, 0, dest.length);
this.tritSafe = new TritSafe(dest);
}

public Hash(final int[] trits) {
this(trits, 0);
}

public Hash(final String trytes) {
this.tritSafe = new TritSafe(new int[SIZE_IN_TRITS]);
Converter.trits(trytes, this.tritSafe.trits, 0);
}

private void fullRead(byte[] src) {
if (src != null) {
synchronized (lock) {
if (byteSafe != null || tritSafe != null) {
throw new IllegalStateException("I cannot be initialized with data twice.");
}
byte[] dest = new byte[SIZE_IN_BYTES];
System.arraycopy(src, 0, dest, 0, Math.min(dest.length, src.length));
byteSafe = new ByteSafe(dest);
}
}
}

public int trailingZeros() {
int index, zeros;
final int[] trits;
index = SIZE_IN_TRITS;
zeros = 0;
trits = trits();
while(index-- > 0 && trits[index] == 0) {
final int[] trits = trits();
int index = SIZE_IN_TRITS;
int zeros = 0;
while (index-- > 0 && trits[index] == 0) {
zeros++;
}
return zeros;
}

public int[] trits() {
if(trits == null) {
trits = new int[Curl.HASH_LENGTH];
Converter.getTrits(bytes, trits);
TritSafe safe = tritSafe;
if (safe == null) {
synchronized (lock) {
if (tritSafe == null) {
Objects.requireNonNull(byteSafe, "I need my bytes to be initialized in order to construct trits.");
byte[] src = bytes();
int[] dest = new int[Curl.HASH_LENGTH];
Converter.getTrits(src, dest);
tritSafe = new TritSafe(dest);
}
safe = tritSafe;
}
}
return trits;
return safe.trits;
}

@Override
Expand All @@ -104,35 +143,36 @@ public boolean equals(Object o) {

@Override
public int hashCode() {
if(bytes == null) {
bytes();
}
return hashCode;
bytes();
return byteSafe.hashcode;
}

@Override
public String toString() {
return Converter.trytes(trits());
}

public byte[] bytes() {
if(bytes == null) {
bytes = new byte[SIZE_IN_BYTES];
Converter.bytes(trits, 0, bytes, 0, trits.length);
hashCode = Arrays.hashCode(this.bytes);
ByteSafe safe = byteSafe;
if (safe == null) {
synchronized (lock) {
if (byteSafe == null) {
Objects.requireNonNull(tritSafe, "I need my trits to be initialized in order to construct bytes.");
int[] src = trits();
byte[] dest = new byte[SIZE_IN_BYTES];
Converter.bytes(src, 0, dest, 0, src.length);
byteSafe = new ByteSafe(dest);
}
safe = byteSafe;
}
}
return bytes;
return safe.bytes;
}

private void fullRead(byte[] bytes, int offset, int size) {
this.bytes = new byte[SIZE_IN_BYTES];
System.arraycopy(bytes, offset, this.bytes, 0, size - offset > bytes.length ? bytes.length-offset: size);
hashCode = Arrays.hashCode(this.bytes);
}

@Override
public void read(byte[] bytes) {
fullRead(bytes, 0, SIZE_IN_BYTES);
public void read(byte[] src) {
fullRead(src);
}

@Override
Expand All @@ -147,7 +187,7 @@ public Indexable decremented() {

@Override
public int compareTo(Indexable indexable) {
Hash hash = new Hash(indexable.bytes());
Hash hash = (indexable instanceof Hash) ? (Hash) indexable : new Hash(indexable.bytes());
if (this.equals(hash)) {
return 0;
}
Expand All @@ -157,5 +197,4 @@ public int compareTo(Indexable indexable) {
}
return (int) diff;
}
}

}

0 comments on commit 84cc51f

Please sign in to comment.