Skip to content
This repository has been archived by the owner on Apr 25, 2019. It is now read-only.

v0.3.0
Compare
Choose a tag to compare
@tkakantousis tkakantousis released this 02 Feb 15:43
· 91 commits to master since this release
v0.3.0
17c41f4 
    Release notes - HopsWorks - Version 0.3.0

Bug

  • [HOPSWORKS-3] - Possibility of data leakage between projects
  • [HOPSWORKS-7] - Unshare Datasets
  • [HOPSWORKS-9] - No warning page when DB is not working
  • [HOPSWORKS-10] - File upload not auto-starting while interface shows they should
  • [HOPSWORKS-13] - Cannot create project with capital letter as a project name
  • [HOPSWORKS-31] - Unclear "can edit" datasets sharing option
  • [HOPSWORKS-37] - kafka schema update not working
  • [HOPSWORKS-38] - Topic Creation Dialog - create button doesn't work sometimes
  • [HOPSWORKS-39] - Logs for jobs that fail during initialization are not available
  • [HOPSWORKS-40] - Fix metadata
  • [HOPSWORKS-41] - UI fixes
  • [HOPSWORKS-42] - Metadata search not working correctly.
  • [HOPSWORKS-43] - search in project and datasets returning results of other projects
  • [HOPSWORKS-49] - Lazy load of job logs in Jobs UI
  • [HOPSWORKS-52] - folders created by job directly in a project can't be accessed
  • [HOPSWORKS-54] - Project certs are not deleted when member is removed by a project
  • [HOPSWORKS-56] - wrong regexp in HdfsUsers result in wrong user being removed
  • [HOPSWORKS-57] - Disallow special characters from job name
  • [HOPSWORKS-58] - Project quotas are not displayed
  • [HOPSWORKS-59] - Kafka UI Advanced view tries to fetch all topics
  • [HOPSWORKS-60] - Cannot delete an imported public dataset from a project
  • [HOPSWORKS-61] - Fixes for Kafka ACLs UI, enabling job logs for shared projects
  • [HOPSWORKS-63] - Job monitor should persist after AppServer restart
  • [HOPSWORKS-65] - Close zeppelin interpreters upon project deletion
  • [HOPSWORKS-66] - Fix starting and stoping services in the the admin interface
  • [HOPSWORKS-82] - webapp.address badly configured in yarn-site.xml
  • [HOPSWORKS-83] - Tour tips do not appear
  • [HOPSWORKS-84] - Example jars TestJob dataset of tour projects is owned by superuser
  • [HOPSWORKS-86] - Sanitize Logs
  • [HOPSWORKS-87] - Telegraf log rotation policy
  • [HOPSWORKS-89] - Fix Jupyter Websockets Proxy
  • [HOPSWORKS-91] - Telegraf influxdb user
  • [HOPSWORKS-92] - Error with kompics snapshot repo
  • [HOPSWORKS-93] - Set python to project conda repo for Zeppelin Livy interpreter
  • [HOPSWORKS-95] - project disk size blocked to MB and limited to int
  • [HOPSWORKS-96] - Cleaning up a Project after an unsuccessful creation should not delete the user
  • [HOPSWORKS-97] - String interpolation in hopsworks-chef
  • [HOPSWORKS-98] - telegraf user is not created
  • [HOPSWORKS-104] - Fix tensorflow download url in tensorflow-chef
  • [HOPSWORKS-105] - Disable cuda installation in karamel-chef
  • [HOPSWORKS-106] - Fix Jupyter PYSPARK_PYTHON env variable
  • [HOPSWORKS-107] - executors jvm metrics lost in influxdb structure update
  • [HOPSWORKS-108] - hadoop-metrics2.properties not overwritten when installin hops
  • [HOPSWORKS-109] - admin.js forgoten when running hops-monitor-chef
  • [HOPSWORKS-110] - grafana dashboards not scalling
  • [HOPSWORKS-111] - Checkout specific version of Tensorflow
  • [HOPSWORKS-112] - Fix typos in hops-hadoop-chef yarn-env template
  • [HOPSWORKS-114] - iframe in admin UI breaking the session
  • [HOPSWORKS-116] - Handle tensorboard when job is in state Killed
  • [HOPSWORKS-117] - Randomize IP address space in karamel-chef
  • [HOPSWORKS-120] - Remove NN ip from URI in HDFS files
  • [HOPSWORKS-123] - foreing key to hops broken
  • [HOPSWORKS-126] - Formatting DB before starting any HDFS service
  • [HOPSWORKS-127] - User reappearing after project deletion
  • [HOPSWORKS-129] - forgoten table moddification in HOPSWORKS-63
  • [HOPSWORKS-131] - TensorFlow tour creates both jobs when a new one is created
  • [HOPSWORKS-132] - Pin VM image version
  • [HOPSWORKS-133] - Logs of jobs that fail before YARN submission are not displayed
  • [HOPSWORKS-134] - Kafka tour does not allow creation of new jobs with non default topics
  • [HOPSWORKS-137] - Kibana allows users to view logs of other projects they are members of
  • [HOPSWORKS-138] - Directory creation - give error for spaces in UI
  • [HOPSWORKS-144] - File preview does not work
  • [HOPSWORKS-146] - Upgrade zeppelin to support spark 2.2.0.
  • [HOPSWORKS-147] - [TensorflowOnSpark] Change user that writes to HDFS
  • [HOPSWORKS-148] - Spark Shuffle Jar file not tracking spark version increments
  • [HOPSWORKS-149] - Change Java Properties to Environment Variables when launching Yarn jobs from Hopsworks
  • [HOPSWORKS-151] - Jupyter Livy sessions showing up in Zeppelin View
  • [HOPSWORKS-154] - Jupyter Livy problems with PYSPARK_PYTHON
  • [HOPSWORKS-155] - [hopsworks-chef] Pydoop cannot be installed
  • [HOPSWORKS-160] - Cleaning up and fixing admin Monitoring and Control
  • [HOPSWORKS-161] - kagent return status for conda is too long for DB table
  • [HOPSWORKS-164] - Validate a request to get a keystore's password against the CN
  • [HOPSWORKS-170] - Data Scientists cant delete their own files
  • [HOPSWORKS-172] - Fix bug when adding members to project
  • [HOPSWORKS-173] - Python 3 not working in Conda
  • [HOPSWORKS-174] - Skip setting PATH env in TFoS
  • [HOPSWORKS-175] - User/Group for each service broken
  • [HOPSWORKS-177] - Jobs monitoring timer is not updating the state of jobs
  • [HOPSWORKS-180] - Vizops UI fixes
  • [HOPSWORKS-181] - Project Administration
  • [HOPSWORKS-185] - [hopslog-chef hopsmonitor-chef spark-chef] Move creation of metrics.properties and log4j.properties to hopslog-chef
  • [HOPSWORKS-186] - Kafka service uses wrong keystore and truststore password
  • [HOPSWORKS-187] - Remove creation of projects_management view
  • [HOPSWORKS-189] - Jupyter not working with per-service usernames
  • [HOPSWORKS-190] - Zeppelin broken for per-user services
  • [HOPSWORKS-191] - Fix versioning of hops-util and hops-examples in chef
  • [HOPSWORKS-192] - Kibana broken for Zeppelin and Jupyter
  • [HOPSWORKS-195] - Deleting a project can leave orphaned jupyter notebooks
  • [HOPSWORKS-196] - Search Bar has no spinner to indicate that it is busy searching
  • [HOPSWORKS-200] - Chef should add parquet dependencies in spark archive
  • [HOPSWORKS-201] - Fix logging dataset creation
  • [HOPSWORKS-203] - Make sure libcurl.so.4 is installed on the machine where ePipe runs
  • [HOPSWORKS-207] - Remove user jobs when they are removed for a project
  • [HOPSWORKS-209] - Fix ownership of files belonging to a removed project member
  • [HOPSWORKS-211] - Jupyter admin ui regression
  • [HOPSWORKS-214] - Zeppelin per-user interpreter leads to awful behaviors
  • [HOPSWORKS-216] - Job scheduer fails due to missing ExecutionId
  • [HOPSWORKS-217] - jobs not writing metrics to influxDB
  • [HOPSWORKS-218] - Jupyter with tensorflow fails with SSL enabled
  • [HOPSWORKS-219] - Stopping and Starting services in admin UI not working when SSL activated
  • [HOPSWORKS-222] - Cluster utilization bar has wrong colour
  • [HOPSWORKS-223] - Duration of a launched job continues counting
  • [HOPSWORKS-227] - Spark Executor should not export CUDA_VISIBLE_DEVICES=""
  • [HOPSWORKS-228] - Duplicate "GPUs" option for TensorFlow in Jupyter
  • [HOPSWORKS-230] - NullPointerException when two user working on same zeppelin notebook
  • [HOPSWORKS-234] - CertificateMaterializer should throw an exception if transient dir does not exist
  • [HOPSWORKS-235] - Don't delete materialized certificates when closing udfso if SSL is not enabled
  • [HOPSWORKS-239] - Redirection on viewing a big job log lead to blank page
  • [HOPSWORKS-241] - HDFS path for metrics.properties has a white-space character
  • [HOPSWORKS-245] - Fix jobUi dropdown menu to select the tensorboard
  • [HOPSWORKS-246] - Zeppelin Interpreter Conf table failed to persist/update if project with the same name is created
  • [HOPSWORKS-248] - Refactor initialization of BaseHadoopServices
  • [HOPSWORKS-250] - User Administration send to the wrong user page
  • [HOPSWORKS-253] - Hadoop files are copied to YARN containers with Jupyter and Zeppelin
  • [HOPSWORKS-255] - ApplicationService retrieves wrong project name
  • [HOPSWORKS-260] - Buttons on Jupyter console not showing up
  • [HOPSWORKS-261] - NPE for project mgmt due to quotas
  • [HOPSWORKS-262] - Spark job staging dir should be unique per application
  • [HOPSWORKS-263] - Sanitize TensorFlow configuration in JupyterConfig
  • [HOPSWORKS-266] - Removing a member from a project does not remove them from Kafka ACLs
  • [HOPSWORKS-267] - Set default livy session expiration to 1 hour for Zeppelin
  • [HOPSWORKS-268] - Multiple fixes for hops-hadoop-chef to work with SSL
  • [HOPSWORKS-269] - cannot remove a dataset shared with your project
  • [HOPSWORKS-279] - front page not scalling well with some resolutions
  • [HOPSWORKS-287] - Jupyter and Zeppelin buttons not working due to missing function
  • [HOPSWORKS-289] - Check for user's password and sessionid in one REST call
  • [HOPSWORKS-298] - Service status tooltip getting out of the window
  • [HOPSWORKS-301] - Fix broken Jobs UI and tour tips
  • [HOPSWORKS-303] - Kafka brokers addresses in HopsUtil must be retrieved from ZooKeeper
  • [HOPSWORKS-304] - Wrong update of false login
  • [HOPSWORKS-308] - When removing a file a log for dataset delete is written into ops_log
  • [HOPSWORKS-311] - Orphaned elastic result?
  • [HOPSWORKS-313] - Conda Block-Report Protocol fixes
  • [HOPSWORKS-314] - Jupyter does not lower case project name for elasticsearch
  • [HOPSWORKS-316] - Limit in number of projects should be a hard limit
  • [HOPSWORKS-318] - Security Question makes entries lowercase
  • [HOPSWORKS-319] - Adjust certificate services to new project generic user
  • [HOPSWORKS-321] - Executors tab in Spark web UI is not working
  • [HOPSWORKS-325] - Telegraf configuration is problematic
  • [HOPSWORKS-328] - Livy sessions should use projectgenericuser for Zeppelin
  • [HOPSWORKS-329] - Zeppelin notebooks cannot be deleted
  • [HOPSWORKS-330] - Remove hardcoded hive user from start-llap script launcher
  • [HOPSWORKS-331] - Set interpreter binding mode for projectgenericuser
  • [HOPSWORKS-332] - Check if superuser UGI when closing DFS client
  • [HOPSWORKS-335] - Add project generic user to Zeppelin connection list
  • [HOPSWORKS-336] - Kafka schema dropdown list does not expand
  • [HOPSWORKS-342] - Encrypted password stored in DB should be updated when user change their password
  • [HOPSWORKS-343] - Project under creation cause NPE on promectsManagement page
  • [HOPSWORKS-344] - numCreatedProjects should be annotated with NotNull
  • [HOPSWORKS-348] - Spam Requests admin page not working
  • [HOPSWORKS-360] - Cluster utilisation should not be based on price multiplicator
  • [HOPSWORKS-362] - "-" hyphen in project name causes jupyter to fail to start
  • [HOPSWORKS-364] - Allow data scientist project role to download projectgenericuser files
  • [HOPSWORKS-366] - In the ProjectManagementUI the Hive quota should be -1 by default
  • [HOPSWORKS-368] - Rollback all handlers when changing master encryption key
  • [HOPSWORKS-372] - hopsworks-cluster profile
  • [HOPSWORKS-375] - Session ID is not updated when admin user navigates away from hopsworks-admin
  • [HOPSWORKS-377] - Adding a member in a project, does not automatically add them in the Kafka ACLs
  • [HOPSWORKS-379] - Uploaded files permissions
  • [HOPSWORKS-387] - CertificateMaterializer does not work as expected with Livy/Spark and Hive interpreter
  • [HOPSWORKS-388] - FilePreview tries to display entire file
  • [HOPSWORKS-393] - Notification when trying to start Jupyter but out of Quota
  • [HOPSWORKS-396] - Certificate password file is not cleaned up in HDFS
  • [HOPSWORKS-397] - Pyspark throws NoClassDefFoundError: io/hops/leaderElection/HdfsLeDescriptorFactory
  • [HOPSWORKS-401] - Roles should be cleaned up with Hosts
  • [HOPSWORKS-404] - Changing payment type in admin UI not working
  • [HOPSWORKS-407] - Data scientists can install python deps but cannot list them
  • [HOPSWORKS-408] - java.lang.NumberFormatException in LlapClusterFacade
  • [HOPSWORKS-413] - Log level in Jupyter settings is ignored
  • [HOPSWORKS-425] - Changing user password front-end not working
  • [HOPSWORKS-433] - Fix namenode metrics collection/visualisation
  • [HOPSWORKS-436] - Miscellaneous fixes for TensorFlow Serving

Sub-task

New Feature

Improvement

  • [HOPSWORKS-14] - spark UI for Zeppeling jobs
  • [HOPSWORKS-15] - Kafka guided tour
  • [HOPSWORKS-16] - Tours notification
  • [HOPSWORKS-20] - update zeppelin to zeppelin-0.7
  • [HOPSWORKS-23] - Rest end point to send emails from application
  • [HOPSWORKS-26] - Support for TensorFlowOnSpark
  • [HOPSWORKS-29] - Refactor projects creation and removal
  • [HOPSWORKS-30] - Differentiate between applications for HDFS certificates directory
  • [HOPSWORKS-32] - show that a dataset is shared
  • [HOPSWORKS-33] - Close file uploader dialogue on success
  • [HOPSWORKS-35] - Store metrics from spark, hadoop into influxdb and turn them into tables
  • [HOPSWORKS-36] - Access Spark history server for old job executions
  • [HOPSWORKS-64] - Launch Jupyter as user 'jupyter' with a private directory that can be used by Python
  • [HOPSWORKS-67] - Add possibility to start a service on all the nodes at once
  • [HOPSWORKS-70] - Improve Kafka tour to use StructuredStreaming
  • [HOPSWORKS-71] - Configure defaultManagedExecutorService with sane values
  • [HOPSWORKS-73] - Deploy Hopsworks with different users for running services
  • [HOPSWORKS-74] - admin dashboard
  • [HOPSWORKS-78] - Reverse Websocker proxy for Jupyter
  • [HOPSWORKS-81] - Automate port-forwarding for mysqld
  • [HOPSWORKS-88] - Fix log rotation policy for auth.log
  • [HOPSWORKS-90] - Tensorflow 1.2
  • [HOPSWORKS-103] - Add support for yarn-env.sh to hops-hadoop-chef
  • [HOPSWORKS-122] - [hops-hadoop-chef] RPC TLS configuration
  • [HOPSWORKS-140] - Break up application submission for Tensorflow
  • [HOPSWORKS-159] - Whitelist system users from logging them every time they login
  • [HOPSWORKS-165] - add possibility to access HDFS and YARN UI from admin page
  • [HOPSWORKS-166] - [hopsworks-chef] Add column to service_certs table for storing password
  • [HOPSWORKS-167] - [hops-hadoop-chef] Add Hopsworks REST endpoint to core-site.xml
  • [HOPSWORKS-169] - [hopsworks-chef] Change truststore password
  • [HOPSWORKS-178] - kill job button not workin in YarnUI
  • [HOPSWORKS-179] - All services in red in admin ui
  • [HOPSWORKS-182] - [hops-hadoop-chef] SSL Attribute renaming
  • [HOPSWORKS-194] - Change chef cookbooks from dot notation to bracket notation
  • [HOPSWORKS-210] - Upgade Kafka to 0.11.0.1
  • [HOPSWORKS-212] - Jupyter tour
  • [HOPSWORKS-220] - Improve CertificateMaterializer on force delete certificates
  • [HOPSWORKS-221] - Update Hopsworks dependency of hadoop-client to 2.8.2.1
  • [HOPSWORKS-226] - Error when trying to recover code for non-existent user
  • [HOPSWORKS-238] - Dela sharing options
  • [HOPSWORKS-247] - Project creation is getting slow
  • [HOPSWORKS-249] - User system helth indicator
  • [HOPSWORKS-254] - In project creation use only superuser DFS client
  • [HOPSWORKS-271] - Make use of Dynamic executors for non-distributed TensorFlow
  • [HOPSWORKS-273] - Improve Jupyter UI for TensorFlow and Spark
  • [HOPSWORKS-275] - Download project user certificates from Hopsworks
  • [HOPSWORKS-277] - Only set HADOOP_USER_NAME for jupyter and zeppelin
  • [HOPSWORKS-286] - @AllowedRoles is confusingly named and the semantics need to change
  • [HOPSWORKS-292] - Use showdown instead of ng-showdown to set showdown params
  • [HOPSWORKS-294] - Cluster shared dataset catalog is slow to load
  • [HOPSWORKS-317] - Set Livy as the default Zeppelin interpreter
  • [HOPSWORKS-338] - Remove virtual cores configuration for TensorFlow in Jupyter.
  • [HOPSWORKS-339] - Show error message to the user if LLAP cluster fails to start
  • [HOPSWORKS-341] - Dynamic executors for TensorFlowOnSpark
  • [HOPSWORKS-345] - Salt user password stored in database
  • [HOPSWORKS-354] - list of user owned projects is not sorted
  • [HOPSWORKS-359] - Add REST endpoint for LLAP cluster admin
  • [HOPSWORKS-365] - Cluster utilization is not very pretty
  • [HOPSWORKS-367] - Replace TensorFlowOnSpark executor configuration with worker
  • [HOPSWORKS-371] - Admin user should be able to delete a project
  • [HOPSWORKS-376] - Dynamic executors for Horovod
  • [HOPSWORKS-405] - Spark jobs should not retry 4 times in case of failure when running TensorFlow
Assets 2
Loading