-
Notifications
You must be signed in to change notification settings - Fork 313
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Prep 0.12.0 release #564
Merged
Merged
Prep 0.12.0 release #564
Changes from 8 commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
122caa1
initial commit
suhsteve 1d0861c
remove extra paranthesis
suhsteve 064fcdc
PR comments
suhsteve 8d1fccb
PR comments
suhsteve e14afaa
Merge branch 'master' into v0.12.0
suhsteve 1e10110
add known issues section
suhsteve 4540df2
add 530
suhsteve 9ea9239
Merge branch 'master' into v0.12.0
imback82 9234d1d
Merge branch 'master' into v0.12.0
imback82 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,115 @@ | ||
# .NET for Apache Spark 0.12 Release Notes | ||
|
||
### New Features/Improvements and Bug Fixes | ||
|
||
* Expose `DataStreamWriter.ForeachBatch` API ([#549](https://github.com/dotnet/spark/pull/549)) | ||
* Support for [dotnet-interactive](https://github.com/dotnet/interactive) ([#515](https://github.com/dotnet/spark/pull/515)) ([#517](https://github.com/dotnet/spark/pull/517)) ([#554](https://github.com/dotnet/spark/pull/554)) | ||
* Support for [Hyperspace v0.1.0](https://github.com/microsoft/hyperspace) APIs ([#555](https://github.com/dotnet/spark/pull/555)) | ||
* Support for Spark 2.4.6 ([#547](https://github.com/dotnet/spark/pull/547)) | ||
* Bug fixes: | ||
* Udf bug caused by `BroadcastVariablesRegistry` ([#551](https://github.com/dotnet/spark/pull/551)) | ||
* Null checks for `TimestampType` and `DateType` ([#530](https://github.com/dotnet/spark/pull/530)) | ||
* Update `Microsoft.Data.Analysis` to v`0.4.0` ([#528](https://github.com/dotnet/spark/pull/528)) | ||
|
||
### Infrastructure / Documentation / Etc. | ||
|
||
* Improve build pipeline ([#510](https://github.com/dotnet/spark/pull/510)) ([#511](https://github.com/dotnet/spark/pull/511)) ([#512](https://github.com/dotnet/spark/pull/512)) ([#513](https://github.com/dotnet/spark/pull/513)) ([#524](https://github.com/dotnet/spark/pull/524)) | ||
* Update AppName for the C# Spark Examples ([#548](https://github.com/dotnet/spark/pull/548)) | ||
* Update maven links in build documentation ([#558](https://github.com/dotnet/spark/pull/558)) ([#560](https://github.com/dotnet/spark/pull/560)) | ||
|
||
### Breaking Changes | ||
|
||
* None | ||
|
||
### Known Issues | ||
|
||
* Broadcast variables do not work with [dotnet-interactive](https://github.com/dotnet/interactive) ([#561](https://github.com/dotnet/spark/pull/561)) | ||
|
||
### Compatibility | ||
|
||
#### Backward compatibility | ||
|
||
The following table describes the oldest version of the worker that the current version is compatible with, along with new features that are incompatible with the worker. | ||
|
||
<table> | ||
<thead> | ||
<tr> | ||
<th>Oldest compatible Microsoft.Spark.Worker version</th> | ||
<th>Incompatible features</th> | ||
</tr> | ||
</thead> | ||
<tbody align="center"> | ||
<tr> | ||
<td rowspan=4>v0.9.0</td> | ||
<td>DataFrame with Grouped Map UDF <a href="https://github.com/dotnet/spark/pull/277">(#277)</a></td> | ||
</tr> | ||
<tr> | ||
<td>DataFrame with Vector UDF <a href="https://github.com/dotnet/spark/pull/277">(#277)</a></td> | ||
</tr> | ||
<tr> | ||
<td>Support for Broadcast Variables <a href="https://github.com/dotnet/spark/pull/414">(#414)</a></td> | ||
</tr> | ||
<tr> | ||
<td>Support for TimestampType <a href="https://github.com/dotnet/spark/pull/428">(#428)</a></td> | ||
</tr> | ||
</tbody> | ||
</table> | ||
|
||
#### Forward compatibility | ||
|
||
The following table describes the oldest version of .NET for Apache Spark release that the current worker is compatible with. | ||
|
||
<table> | ||
<thead> | ||
<tr> | ||
<th>Oldest compatible .NET for Apache Spark release version</th> | ||
</tr> | ||
</thead> | ||
<tbody align="center"> | ||
<tr> | ||
<td>v0.9.0</td> | ||
</tr> | ||
</tbody> | ||
</table> | ||
|
||
### Supported Spark Versions | ||
|
||
The following table outlines the supported Spark versions along with the microsoft-spark JAR to use with: | ||
|
||
<table> | ||
<thead> | ||
<tr> | ||
<th>Spark Version</th> | ||
<th>microsoft-spark JAR</th> | ||
</tr> | ||
</thead> | ||
<tbody align="center"> | ||
<tr> | ||
<td>2.3.*</td> | ||
<td>microsoft-spark-2.3.x-0.12.0.jar</td> | ||
</tr> | ||
<tr> | ||
<td>2.4.0</td> | ||
<td rowspan=6>microsoft-spark-2.4.x-0.12.0.jar</td> | ||
</tr> | ||
<tr> | ||
<td>2.4.1</td> | ||
</tr> | ||
<tr> | ||
<td>2.4.3</td> | ||
</tr> | ||
<tr> | ||
<td>2.4.4</td> | ||
</tr> | ||
<tr> | ||
<td>2.4.5</td> | ||
</tr> | ||
<tr> | ||
<td>2.4.6</td> | ||
</tr> | ||
<tr> | ||
<td>2.4.2</td> | ||
<td><a href="https://github.com/dotnet/spark/issues/60">Not supported</a></td> | ||
</tr> | ||
</tbody> | ||
</table> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you add a known issue section and add the issue where broadcast variables do not work with dotnet-interactive and reference #561?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also if #530 goes in before 9am tomorrow, let's add it. I will update this thread.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we add #530 now?