Skip to content

Commit

Permalink
release 0.1.2, add hive_conf support
Browse files Browse the repository at this point in the history
  • Loading branch information
bennfocus committed Sep 6, 2021
1 parent e2372fc commit 13de000
Show file tree
Hide file tree
Showing 5 changed files with 27 additions and 9 deletions.
8 changes: 7 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
.PHONY: build

ROOT_DIR := $(shell dirname $(realpath $(firstword $(MAKEFILE_LIST))))

format:
Expand All @@ -11,4 +13,8 @@ lint:
cd ${ROOT_DIR}; mypy feast_hive/ tests/
cd ${ROOT_DIR}; isort feast_hive/ tests/ --check-only
cd ${ROOT_DIR}; flake8 feast_hive/ tests/
cd ${ROOT_DIR}; black --check feast_hive tests
cd ${ROOT_DIR}; black --check feast_hive tests

build:
rm -rf dist/*
python setup.py sdist bdist_wheel
11 changes: 7 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ For more details, can check [this Feast issue](https://github.com/feast-dev/feas

## Todo List
- [DONE, v0.1.1] ~~I am working on the first workable version, think it will be released in a couple of days.~~
- Allow custom hive conf when connect to a HiveServer2
- It only supports `insert into` for uploading entity_df for now, which will be a little inefficient. In next version, I will provide some extra parameters for users who are able to provide WebHDFS address.
- [DONE, v0.1.2] ~~Allow custom hive conf when connect to a HiveServer2~~
- It currently supports `insert into` for uploading entity_df, which is a little inefficient, gonna add extra parameters for people who are able to provide HDFS address in next version (for uploading to HDFS).

## Quickstart

Expand Down Expand Up @@ -48,8 +48,11 @@ provider: local
offline_store:
type: feast_hive.HiveOfflineStore
host: localhost
port: 10000 # default is `10000`
database: default # default is `default`
port: 10000 # optional, default is `10000`
database: default # optional, default is `default`
hive_conf: # optional, hive conf overlay
hive.join.cache.size: 14797
hive.exec.max.dynamic.partitions: 779
... # other parameters
online_store:
...
Expand Down
9 changes: 9 additions & 0 deletions feast_hive/hive.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
import pyarrow as pa
from pydantic import StrictBool, StrictInt, StrictStr
from pytz import utc
from six import reraise

from feast import FeatureView
from feast.data_source import DataSource
Expand Down Expand Up @@ -102,6 +103,14 @@ def cursor(self) -> ImpalaCursor:
else:
return self.real_conn.cursor()

def __enter__(self):
return self

def __exit__(self, exc_type, exc_val, exc_tb):
self.close()
if exc_type is not None:
reraise(exc_type, exc_val, exc_tb)


class HiveOfflineStore(OfflineStore):
@staticmethod
Expand Down
2 changes: 1 addition & 1 deletion setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@

setup(
name="feast-hive",
version="0.1.1",
version="0.1.2",
author="Benn Ma",
author_email="[email protected]",
description="Hive support for Feast offline store",
Expand Down
6 changes: 3 additions & 3 deletions tests/test_all.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
import contextlib
import os
import uuid
from pathlib import Path
import random
import time
import uuid
from datetime import datetime
from pathlib import Path
from tempfile import TemporaryDirectory
from typing import Union, Iterator, Tuple, Any, Dict

Expand Down Expand Up @@ -509,7 +509,7 @@ def test_hive_config(pytestconfig):
"hive.exec.max.dynamic.partitions": 779,
}
offline_store = HiveOfflineStoreConfig(**options)
with contextlib.closing(feast_hive_module.HiveConnection(offline_store)) as conn:
with feast_hive_module.HiveConnection(offline_store) as conn:
with conn.cursor() as cursor:
cursor.execute("SET hive.join.cache.size")
expected = ("hive.join.cache.size=14797",)
Expand Down

0 comments on commit 13de000

Please sign in to comment.