Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update branch-21.10 with changes from rapidsai fork #1601

Open
wants to merge 42 commits into
base: branch-21.10
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
28535f3
fixed error where the ucx mem handle was getting freed twice because …
May 19, 2021
2a6977d
Merge pull request #1545 from williamBlazing/fix/ucx_mem_handle
wmalpica May 19, 2021
713b628
fixed parquet metadata for decimal
wmalpica Aug 3, 2021
68fed9b
changed warning to error for when creating a table based off of a cud…
wmalpica Aug 3, 2021
daa1385
[feature/decimal_support_using_float] cast decimal to float64 for par…
gcca Aug 3, 2021
8d6a324
[feature/decimal_support_using_float] cast decimal to float64 for orc
gcca Aug 3, 2021
9f75314
[feature/decimal_support_using_float] check float64 in schema for orc
gcca Aug 3, 2021
0a81e20
[feature/decimal_support_using_float] update changelog
gcca Aug 4, 2021
c3ab090
[feature/decimal_support_using_float] fix for check style
gcca Aug 4, 2021
9826a15
Merge pull request #1589 from gcca/feature/decimal_support_using_float
wmalpica Aug 4, 2021
8c783b9
Add argument thrust-ignore-cub-version-check
romulo-auccapuclla Aug 5, 2021
ddba597
Update changelog
romulo-auccapuclla Aug 5, 2021
6d9d3cb
Merge pull request #1590 from romulo-auccapuclla/fix/thrust-ignore-cu…
wmalpica Aug 5, 2021
e37e4bd
pre-release changes for 21.08
wmalpica Aug 13, 2021
1e59abc
Merge pull request #1592 from wmalpica/prerelease-21.08
wmalpica Aug 13, 2021
a34a089
Pin spdlog to match RAPIDS
raydouglass Aug 18, 2021
924605b
DOC Update changelog
raydouglass Aug 18, 2021
e90a645
Merge pull request #1595 from raydouglass/spdlog1.9
wmalpica Aug 18, 2021
de3b0e8
Restrict google-cloud-cpp version
raydouglass Aug 19, 2021
9efa6dd
DOC Update changelog
raydouglass Aug 19, 2021
bb48f0c
Ensure librmm comes from rapidai channel
raydouglass Aug 20, 2021
cc513aa
Fix dependencies.sh to match meta.yaml
raydouglass Aug 20, 2021
75398f7
Merge pull request #1597 from raydouglass/google-cloud-cpp-version
wmalpica Aug 20, 2021
98d8bab
Update README.md
wmalpica Aug 30, 2021
34b119e
Merge remote-tracking branch 'rapidsai/branch-21.08' into branch-21.10
raydouglass Sep 13, 2021
53f6490
remove empty file
trxcllnt Sep 18, 2021
3918ef2
remove stream param from make_strings_column
trxcllnt Sep 18, 2021
a56f634
vendor the old strings -> dtypes logic that was removed from cudf::io…
trxcllnt Sep 18, 2021
150baec
update groupby and rolling aggregation types and factory functions
trxcllnt Sep 18, 2021
cd271f5
use data_from_unique_ptr instead of the removed Table.from_unique_ptr
trxcllnt Sep 20, 2021
54fe7cd
remove unused import
trxcllnt Sep 20, 2021
a40fb00
update cudf._lib.types import names
trxcllnt Sep 21, 2021
d17342f
use rapidsai/blazingsql-testing-files instead of BlazingDB/blazingsql…
trxcllnt Sep 22, 2021
0379e2d
spread return of data_from_unique_ptr as args list to DataFrame._from…
trxcllnt Sep 22, 2021
d0205d0
fix cudf imports
trxcllnt Sep 22, 2021
52ae55c
update for new cudf::io::table_input_metadata API
trxcllnt Sep 23, 2021
3342f29
Merge pull request #1 from trxcllnt/fix/21.10-cudf-groupby-api-change
ajschmidt8 Sep 23, 2021
47ace5e
Disable uploads to `blazing` channels
ajschmidt8 Sep 24, 2021
e7241a7
Merge pull request #2 from ajschmidt8/disable-blazing-upload
ajschmidt8 Sep 24, 2021
747a240
Merge commit '2a6977d00a095b28e39a4d047572665f08b26f29' into fix_doub…
jglaser Sep 28, 2021
a35643d
Merge pull request #1600 from BlazingDB/fix_double_free
wmalpica Sep 30, 2021
8018b82
Merge pull request #3 from BlazingDB/branch-21.08
raydouglass Oct 7, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 21 additions & 22 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,36 +1,35 @@
# BlazingSQL 21.10.00 (TBD)
## New Features


## Improvements

## Bug Fixes


# BlazingSQL 21.08.00 (TBD)
# BlazingSQL 21.08.00 (August 12th, 2021)

## New Features


## Improvements
- #1571 Update ucx-py versions to 0.21
- #1554 return ok for filesystems
- #1572 Setting up default value for max_bytes_chunk_read to 256 MB


## Bug Fixes
- #1570 Fix build due to changes in rmm device buffer
- #1570 Fix build due to changes in rmm device buffer
- #1574 Fix reading decimal columns from orc file
- #1576 Fix `CC`/`CXX` variables in CI
- #1581 Fix latest cudf dependencies
- #1582 Fix concat suite E2E test for nested calls
- #1585 Fix for GCS credentials from filepath

- #1589 Fix decimal support using float64
- #1590 Fix build issue with thrust package
- #1595 Fix `spdlog` pinning
- #1597 Fix `google-cloud-cpp` pinning

# BlazingSQL 21.06.00 (June 10th, 2021)

## New Features
- #1471 Unbounded partitioned windows
- #1471 Unbounded partitioned windows
- #1445 Support for CURRENT_DATE, CURRENT_TIME and CURRENT_TIMESTAMP
- #1505 Support for right outer join
- #1523 Support for DURATION type
Expand All @@ -50,7 +49,7 @@
- #1455 Support for IS NOT FALSE condition
- #1502 Fix IS NOT DISTINCT FROM with joins
- #1475 Fix wrong results from timestampdiff/add
- #1528 Fixed build issues due to cudf aggregation API change
- #1528 Fixed build issues due to cudf aggregation API change
- #1540 Comparing param set to true for e2e
- #1543 Enables provider unit_tests
- #1548 Fix orc statistic building
Expand All @@ -65,15 +64,15 @@
## New Features
- #1367 OverlapAccumulator Kernel
- #1364 Implement the concurrent API (bc.sql with token, bc.status, bc.fetch)
- #1426 Window Functions without partitioning
- #1426 Window Functions without partitioning
- #1349 Add e2e test for Hive Partitioned Data
- #1396 Create tables from other RDBMS
- #1427 Support for CONCAT alias operator
- #1424 Add get physical plan with explain
- #1472 Implement predicate pushdown for data providers

## Improvements
- #1325 Refactored CacheMachine.h and CacheMachine.cpp
- #1325 Refactored CacheMachine.h and CacheMachine.cpp
- #1322 Updated and enabled several E2E tests
- #1333 Fixing build due to cudf update
- #1344 Removed GPUCacheDataMetadata class
Expand All @@ -82,7 +81,7 @@
- #1331 Added flag to enable null e2e testing
- #1418 Adding support for docker image
- #1434 Added documentation for C++ and Python in Sphinx
- #1419 Added concat cache machine timeout
- #1419 Added concat cache machine timeout
- #1444 Updating GCP to >= version
- #1349 Add e2e test for Hive Partitioned Data
- #1447 Improve getting estimated output num rows
Expand All @@ -99,18 +98,18 @@
- #1350 Fixed bug where there are no projects in a bindable table scan
- #1359 Avoid cuda issues when free pinned memory
- #1365 Fixed build after sublibs changes on cudf
- #1369 Updated java path for powerpc build
- #1369 Updated java path for powerpc build
- #1371 Fixed e2e settings
- #1372 Recompute `columns_to_hash` in DistributeAggregationKernel
- #1375 Fix empty row_group_ids for parquet
- #1380 Fixed issue with int64 literal values
- #1380 Fixed issue with int64 literal values
- #1379 Remove ProjectRemoveRule
- #1389 Fix issue when CAST a literal
- #1387 Skip getting orc metadata for decimal type
- #1392 Fix substrings with nulls
- #1398 Fix performance regression
- #1401 Fix support for minus unary operation
- #1415 Fixed bug where num_batches was not getting set in BindableTableScan
- #1415 Fixed bug where num_batches was not getting set in BindableTableScan
- #1413 Fix for null tests 13 and 23 of windowFunctionTest
- #1416 Fix full join when both tables contains nulls
- #1423 Fix temporary directory for hive partition test
Expand All @@ -123,7 +122,7 @@
- #1504 Fixing some conflicts in Dockerfile

## Deprecated Features
- #1394 Disabled support for outer joins with inequalities
- #1394 Disabled support for outer joins with inequalities

# BlazingSQL 0.18.0 (February 24, 2021)

Expand All @@ -136,7 +135,7 @@
- #1238 Implements MergeStramKernel executor model
- #1259 Implements SortAndSamplernel executor model, also avoid setting up num of samples
- #1271 Added Hive utility for partitioned data
- #1289 Multiple concurrent query support
- #1289 Multiple concurrent query support
- #1285 Infer PROTOCOL when Dask client is passed
- #1294 Add config options for logger
- #1301 Added usage of pinned buffers for communication and fixes various UCX related bugs
Expand All @@ -145,7 +144,7 @@
- #1303 Add support for INITCAP
- #1313 getting and using ORC metadata
- #1347 Fixing issue when reading orc metadata from DATE dtype
- #1338 Window Function support for LEAD and LAG statements
- #1338 Window Function support for LEAD and LAG statements
- #1362 give useful message when file extension is not recognized
- #1361 Supporting first_value and last_value for Window Function

Expand All @@ -168,7 +167,7 @@
- #1308 Improve the engine loggers
- #1314 Added unit tests to verify that OOM error handling works well
- #1320 Revamping cache logger
- #1323 Made progress bar update continuously and stay after query is done
- #1323 Made progress bar update continuously and stay after query is done
- #1336 Improvements for the cache API
- #1483 Improve dependencies script

Expand All @@ -182,7 +181,7 @@
- #1277 Support FileSystems (GS, S3) when extension of the files are not provided
- #1300 Fixed issue when creating tables from a local dir relative path
- #1312 Fix progress bar for jupyterlab
- #1318 Disabled require acknowledge
- #1318 Disabled require acknowledge

# BlazingSQL 0.17.0 (December 10, 2020)

Expand All @@ -203,7 +202,7 @@
- #1201 Implement string TRIM
- #1216 Add unit test for DAYOFWEEK
- #1205 Implement string REVERSE
- #1220 Implement string LEFT and RIGHT
- #1220 Implement string LEFT and RIGHT
- #1223 Add support for UNION statement
- #1250 updated README.md and CHANGELOG and others preparing for 0.17 release

Expand Down Expand Up @@ -262,7 +261,7 @@
- #1203 Changed code back so that parquet is not read a single rowgroup at a time
- #1207 Calcite uses literal as int32 if not explicit CAST was provided
- #1212 Fixed issue when building the thirdpart, cmake version set to 3.18.4
- #1225 Fixed issue due to change in gather API
- #1225 Fixed issue due to change in gather API
- #1254 Fixing support of nightly and stable on localhost
- #1258 Fixing gtest version issue

Expand Down
27 changes: 14 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,8 +79,9 @@ You can find our full documentation at [docs.blazingdb.com](https://docs.blazing
* Pascal or Better
* Compute Capability >= 6.0
* CUDA Support
* 10.1.2
* 10.2
* 11.0
* 11.2
* 11.4
* Python Support
* 3.7
* 3.8
Expand All @@ -91,18 +92,18 @@ BlazingSQL can be installed with conda ([miniconda](https://conda.io/miniconda.h
```bash
conda install -c blazingsql -c rapidsai -c nvidia -c conda-forge -c defaults blazingsql python=$PYTHON_VERSION cudatoolkit=$CUDA_VERSION
```
Where $CUDA_VERSION is 10.1, 10.2 or 11.0 and $PYTHON_VERSION is 3.7 or 3.8
*For example for CUDA 10.1 and Python 3.7:*
Where $CUDA_VERSION is 11.0, 11.2 or 11.4 and $PYTHON_VERSION is 3.7 or 3.8
*For example for CUDA 11.2 and Python 3.8:*
```bash
conda install -c blazingsql -c rapidsai -c nvidia -c conda-forge -c defaults blazingsql python=3.7 cudatoolkit=10.1
conda install -c blazingsql -c rapidsai -c nvidia -c conda-forge -c defaults blazingsql python=3.8 cudatoolkit=11.2
```

## Nightly Version
For nightly version cuda 11+ are only supported, see https://github.com/rapidsai/cudf#cudagpu-requirements
```bash
conda install -c blazingsql-nightly -c rapidsai-nightly -c nvidia -c conda-forge -c defaults blazingsql python=$PYTHON_VERSION cudatoolkit=$CUDA_VERSION
```
Where $CUDA_VERSION is 11.0 or 11.2 and $PYTHON_VERSION is 3.7 or 3.8
Where $CUDA_VERSION is 11.0, 11.2 or 11.4 and $PYTHON_VERSION is 3.7 or 3.8
*For example for CUDA 11.2 and Python 3.8:*
```bash
conda install -c blazingsql-nightly -c rapidsai-nightly -c nvidia -c conda-forge -c defaults blazingsql python=3.8 cudatoolkit=11.2
Expand All @@ -117,14 +118,14 @@ This is the recommended way of building all of the BlazingSQL components and dep
```bash
conda create -n bsql python=$PYTHON_VERSION
conda activate bsql
./dependencies.sh 21.06 $CUDA_VERSION
./dependencies.sh 21.08 $CUDA_VERSION
```
Where $CUDA_VERSION is is 10.1, 10.2 or 11.0 and $PYTHON_VERSION is 3.7 or 3.8
*For example for CUDA 10.1 and Python 3.7:*
Where $CUDA_VERSION is is 11.0, 11.2 or 11.4 and $PYTHON_VERSION is 3.7 or 3.8
*For example for CUDA 11.2 and Python 3.7:*
```bash
conda create -n bsql python=3.7
conda activate bsql
./dependencies.sh 21.06 10.1
./dependencies.sh 21.08 11.2
```

### Build
Expand All @@ -149,14 +150,14 @@ For nightly version cuda 11+ are only supported, see https://github.com/rapidsai
```bash
conda create -n bsql python=$PYTHON_VERSION
conda activate bsql
./dependencies.sh 21.08 $CUDA_VERSION nightly
./dependencies.sh 21.10 $CUDA_VERSION nightly
```
Where $CUDA_VERSION is 11.0 or 11.2 and $PYTHON_VERSION is 3.7 or 3.8
Where $CUDA_VERSION is 11.0, 11.2 or 11.4 and $PYTHON_VERSION is 3.7 or 3.8
*For example for CUDA 11.2 and Python 3.8:*
```bash
conda create -n bsql python=3.8
conda activate bsql
./dependencies.sh 21.08 11.2 nightly
./dependencies.sh 21.10 11.2 nightly
```

### Build
Expand Down
2 changes: 1 addition & 1 deletion build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ fi
# Get version number
export GIT_DESCRIBE_TAG=`git describe --tags`
export MINOR_VERSION=`echo $GIT_DESCRIBE_TAG | grep -o -E '([0-9]+\.[0-9]+)'`
export UCX_PY_VERSION="0.21"
export UCX_PY_VERSION="0.22"

# Process flags
if hasArg -v; then
Expand Down
4 changes: 2 additions & 2 deletions ci/cpu/upload.sh
Original file line number Diff line number Diff line change
Expand Up @@ -50,8 +50,8 @@ if [ "$UPLOAD_BLAZING" == "1" ]; then

test -e ${BLAZINGSQL_FILE}

echo "Upload BlazingSQL to ${CONDA_USERNAME} channel: ${BLAZINGSQL_FILE}"
anaconda -t ${MY_UPLOAD_KEY} upload -u ${CONDA_USERNAME} ${LABEL_OPTION} --force ${BLAZINGSQL_FILE}
# echo "Upload BlazingSQL to ${CONDA_USERNAME} channel: ${BLAZINGSQL_FILE}"
# anaconda -t ${MY_UPLOAD_KEY} upload -u ${CONDA_USERNAME} ${LABEL_OPTION} --force ${BLAZINGSQL_FILE}

if [ ! -z "$RAPIDS_CONDA_USERNAME" ]; then
echo "Upload BlazingSQL to ${RAPIDS_CONDA_USERNAME} channel: ${BLAZINGSQL_FILE}"
Expand Down
2 changes: 1 addition & 1 deletion ci/gpu/test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,7 @@ else
PR_BR=${PR_AUTHOR}":"${SOURCE_BRANCH}
else
echo "The fork doesn't exist"
git clone --depth 1 https://github.com/BlazingDB/blazingsql-testing-files.git --branch ${TARGET_BRANCH} --single-branch
git clone --depth 1 https://github.com/rapidsai/blazingsql-testing-files.git --branch ${TARGET_BRANCH} --single-branch
fi
fi
set -e
Expand Down
10 changes: 5 additions & 5 deletions conda/recipes/blazingsql/meta.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -27,19 +27,19 @@ requirements:
- ninja
- gtest 1.10
- aws-sdk-cpp
- google-cloud-cpp >=1.25.0
- google-cloud-cpp >=1.25.0,<1.30
- mysql-connector-cpp 8.0.23
- libpq 13
- nlohmann_json 3.9.1
- cppzmq
- python
- setuptools
- spdlog >=1.8.5,<2.0.0a0
- spdlog >=1.8.5,<1.9
- cython >=0.29,<0.30
- openjdk >=8.0, <9.0
- maven
- cudf {{ minor_version }}.*
- ucx-py 0.21.*
- ucx-py 0.22.*
- ucx-proc=*=gpu
- boost-cpp 1.72.0
- dlpack
Expand All @@ -54,11 +54,11 @@ requirements:
- netifaces
- pyhive
- sqlite 3
- spdlog >=1.8.5,<2.0.0a0
- spdlog >=1.8.5,<1.9
- {{ pin_compatible('zeromq', max_pin='x.x.x') }}
- dask-cudf {{ minor_version }}.*
- dask-cuda {{ minor_version }}.*
- ucx-py 0.21.*
- ucx-py 0.22.*
- ucx-proc=*=gpu
- {{ pin_compatible('cudatoolkit', max_pin='x.x') }}
- tqdm
Expand Down
6 changes: 3 additions & 3 deletions dependencies.sh
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ ITALICRED="\e[3;${RED}"
ENDCOLOR="\e[0m"

RAPIDS_VERSION="21.08"
UCX_PY_VERSION="0.21"
UCX_PY_VERSION="0.22"
CUDA_VERSION="11.0"
CHANNEL=""

Expand All @@ -28,13 +28,13 @@ if [ ! -z $3 ]; then
fi

echo -e "${GREEN}Installing dependencies${ENDCOLOR}"
conda install --yes -c conda-forge spdlog'>=1.8.5,<2.0.0a0' google-cloud-cpp'>=1.25' ninja mysql-connector-cpp=8.0.23 libpq=13 nlohmann_json=3.9.1
conda install --yes -c conda-forge spdlog'>=1.8.5,<1.9' google-cloud-cpp'>=1.25,<1.30' ninja mysql-connector-cpp=8.0.23 libpq=13 nlohmann_json=3.9.1
# NOTE cython must be the same of cudf (for 0.11 and 0.12 cython is >=0.29,<0.30)
conda install --yes -c conda-forge cmake=3.18 gtest==1.10.0=h0efe328_4 gmock cppzmq cython=0.29 openjdk'>=8.0,<9.0' maven jpype1 netifaces pyhive pytest tqdm ipywidgets boost-cpp=1.72.0


echo -e "${GREEN}Install RAPIDS dependencies${ENDCOLOR}"
conda install --yes -c rapidsai$CHANNEL -c nvidia -c conda-forge -c defaults dask-cuda=$RAPIDS_VERSION dask-cudf=$RAPIDS_VERSION cudf=$RAPIDS_VERSION ucx-py=$UCX_PY_VERSION ucx-proc=*=gpu cudatoolkit=$CUDA_VERSION
conda install --yes -c rapidsai$CHANNEL -c nvidia -c conda-forge -c defaults dask-cuda=$RAPIDS_VERSION dask-cudf=$RAPIDS_VERSION cudf=$RAPIDS_VERSION "rapidsai$CHANNEL::librmm=$RAPIDS_VERSION" ucx-py=$UCX_PY_VERSION ucx-proc=*=gpu cudatoolkit=$CUDA_VERSION

echo -e "${GREEN}Install E2E test dependencies${ENDCOLOR}"

Expand Down
2 changes: 1 addition & 1 deletion docsrc/source/Doxyfile
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ PROJECT_NAME = "BlazingSQL Engine"
# could be handy for archiving the generated documentation or if some version
# control system is used.

PROJECT_NUMBER = 21.06
PROJECT_NUMBER = 21.08

# Using the PROJECT_BRIEF tag one can provide an optional one line description
# for a project that appears at the top of each page and should give viewer a
Expand Down
2 changes: 1 addition & 1 deletion docsrc/source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@
language = "en"

# The full version, including alpha/beta/rc tags
version = '21.06'
version = '21.08'
release = f'v{version}'

# -- General configuration ---------------------------------------------------
Expand Down
Loading