Merge branch 'main' into chraberturas/pandas-api-nunique

.github/PULL_REQUEST_TEMPLATE/bug.md

@@ -0,0 +1,24 @@
+# Bug Fix
+
+- Please insert link to github issue here:
+
+## What is the bug?
+
+If possible please include a summary and reproducible use-case below:
+
+```python
+>>> (insert example here)
+```
+
+## How does the change fix it?
+
+Include a short description outlining how your change fixes current behaviour
+
+## Code
+
+- [ ] Is code production ready (no stub/test functions, hardcoded IP/ tables/ hostnames, etc.)?
+
+## Testing
+
+- [ ] Have unit tests been created or existing ones updated to catch this bug in the future?
+- [ ] Has test coverage remained the same or improved?

.github/PULL_REQUEST_TEMPLATE/documentation.md

@@ -0,0 +1,11 @@
+# Documentation template
+
+## What does this change introduce?
+
+## Checklist
+
+- [ ] Have you ensured that any changes to the documentation are correctly formatted, in particular are code snippets being correctly displayed?
+- [ ] If a new class has been added has a documentation stub `.md` file associated with it been created?
+- [ ] Have you ensured that any included hyperlinks are operating correctly?
+- [ ] If any documentation page has been created has it been added to `mkdocs.yml`
+- [ ] Have you checked your changes with a spell checker? (US English)

.github/PULL_REQUEST_TEMPLATE/feature.md

@@ -0,0 +1,39 @@
+# Feature
+
+- Please insert link to associated issue here:
+
+## What does this change introduce?
+
+## General
+
+- [ ] Has an example been added to demo the new feature?
+- [ ] Have existing examples been updated or tested?
+- [ ] Have you added any new Environment Variables/Configuration Options? If yes please tick the boxes below as applicable
+    - [ ] Addition to reimporter logic within `src/pykx/pykx.q` and `src/pykx/reimporter.py`
+    - [ ] Have updated the `src/pykx/util.py` logic which is used for environment variable
+- [ ] If there have been any dependency updates have they been reflected in all files?
+    - [ ] pyproject.toml
+    - [ ] docs/getting-started/installing.md
+    - [ ] conda-recipe/meta.yaml
+    - [ ] README.md
+- [ ] If any examples have been updated has it's associated `.zip` been updated
+
+## Code
+
+- [ ] Has all temporary code used during development been removed?
+- [ ] Has all commented out (unused) code been removed?
+- [ ] Where reasonable have you ensured there is no duplication of existing code?
+- [ ] If applicable for your use-case have you ensured that the code is performant?
+
+## Testing
+
+- [ ] Have unit tests been created or existing ones updated to test this new functionality?
+
+## Documentation
+
+- [ ] Has documentation been added for all public code?
+- [ ] Has a release note been included for the new feature?
+- [ ] Has any documentation which would benefit from this feature been updated to use the most up to date functionality?
+- [ ] If a new class has been added has a documentation stub `.md` file associated with it been created?
+- [ ] If any documentation page has been created has it been added to `mkdocs.yml`
+- [ ] Have you checked your changes with a spell checker? (US English)

.github/pull_request_template.md

@@ -0,0 +1,5 @@
+Please go the the `Preview` tab and select the appropriate sub-template:
+
+* [Bug fix Pull Request](?expand=1&template=bug.md)
+* [Feature addition Pull Request](?expand=1&template=feature.md)
+* [Documentation addition Pull Request](?expand=1&template=documentation.md)

.gitignore

@@ -223,3 +223,10 @@ coverage.xml
 
 # CCLS files
 .ccls-cache/*
+
+# HDB Test files
+HDB/**
+multiColSplay/**
+singleColSplay/**
+sym
+symsEnumsSplay/**

conda-recipe/meta.yaml

@@ -27,7 +27,7 @@ requirements:
   run:
     - python
     - numpy>=1.22
-    - pandas>=1.2,<2.0
+    - pandas>=1.2
     - pytz>=2022.1
     - toml>=0.10.2
 

docs/examples/server/server.md

@@ -7,11 +7,14 @@ To follow along with this example please feel free to download this <a href="./a
 
 ## Quick start
 
-To run this example simply run the `server.py` script and it will launch a `PyKX` server on port 5000.
+To run this example simply run the `server.py` script and it will launch a `PyKX` server on port 5000 or
+you can run `server_async.py` to run an asyncronous version of the server.
 The server will print out any queries it receives as well as the result of executing the query before replying.
 
 ```bash
 python server.py
+// or
+python server_async.py
 ```
 
 ## Extra Configuration Options

docs/examples/server/server_async.py

@@ -0,0 +1,36 @@
+import asyncio
+import sys
+
+
+import pykx as kx
+
+port = 5000
+if len(sys.argv)>1:
+    port = int(sys.argv[1])
+
+
+def qval_sync(query):
+    res = kx.q.value(query)
+    print("sync")
+    print(f'{query}\n{res}\n')
+    return res
+
+
+def qval_async(query):
+    res = kx.q.value(query)
+    print("async")
+    print(f'{query}\n{res}\n')
+
+
+async def main():
+    # It is possible to add user validation by overriding the .z.pw function
+    # kx.q.z.pw = lambda username, password: password == 'password'
+    kx.q.z.pg = qval_sync
+    kx.q.z.ps = qval_async
+    async with kx.RawQConnection(port=port, as_server=True, conn_gc_time=20.0) as q:
+        while True:
+            await q.poll_recv_async()
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

docs/examples/subscriber/readme.md

@@ -26,6 +26,8 @@ q)
 ```bash
 // run the subscriber which will automatically connect
 $ python subscriber.py
+// you can also run the asnychronous example with
+$ python subscriber_async.py
 ```
 
 ### Outcome

docs/examples/subscriber/subscriber_async.py

@@ -0,0 +1,39 @@
+import pykx as kx
+
+import asyncio
+
+
+table = kx.q('([] a: 10?10; b: 10?10)')
+
+
+def assert_result(res):
+    # assert message from q process has the correct schema to be appended to the table
+    return type(res) is kx.LongVector and len(res) == 2
+
+
+async def main_loop(q):
+    global table
+    while True:
+        result = await q.poll_recv_async()
+        if assert_result(result):
+            print(f'Recieved new table row from q: {result}')
+            table = kx.q.upsert(table, result)
+            print(table)
+            result = None
+
+
+async def main():
+    global table
+    async with kx.RawQConnection(port=5001, event_loop=asyncio.get_event_loop()) as q:
+        print('===== Initital Table =====')
+        print(table)
+        print('===== Initital Table =====')
+        # Set the variable py_server on the q process pointing towards this processes IPC connection
+        # We use neg to ensure the messages are sent async so no reply is expected from this process
+        await q('py_server: neg .z.w')
+
+        await main_loop(q)
+
+
+if __name__ == '__main__':
+    asyncio.run(main())

docs/getting-started/installing.md

@@ -151,14 +151,17 @@ KX only officially supports versions of PyKX built by KX, i.e. versions of PyKX
 
 PyKX depends on the following third-party Python packages:
 
-- `pandas~=1.2`
+- `pandas>=1.2`
 - `numpy~=1.22`
-- `pytz~=2022.1`
+- `pytz>=2022.1`
 - `toml~=0.10.2`
 
 They are installed automatically by `pip` when PyKX is installed.
 
-PyKX also has an optional Python dependency of `pyarrow>=3.0.0`, which can be included by installing the `pyarrow` extra, e.g. `pip install pykx[pyarrow]`
+### Optional Python Dependencies
+
+- `pyarrow>=3.0.0`, which can be included by installing the `pyarrow` extra, e.g. `pip install pykx[pyarrow]`.
+- `find-libpython~=0.2`, which can be included by installing the `debug` extra, e.g. `pip install pykx[debug]`. This dependency can be used to help find `libpython` in the scenario that `pykx.q` fails to find it.
 
 !!! Warning
 

docs/getting-started/q_magic_command.ipynb

@@ -46,7 +46,7 @@
    "metadata": {},
    "source": [
     "####  Executing against Embedded q\n",
-    "A cell begining with `%%q` will execute q within `PyKX`'s `EmbeddedQ` module."
+    "A cell beginning with `%%q` will execute q within `PyKX`'s `EmbeddedQ` module."
    ]
   },
   {
@@ -60,6 +60,33 @@
     "til 10"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "`print` is the default method called on returned objects. To use `display` you can instead pass `--display`"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%q\n",
+    "([] a: 1 2 3)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%q --display\n",
+    "([] a: 1 2 3)"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "89ec26e4",

docs/getting-started/quickstart.md

@@ -130,6 +130,16 @@ PyKX objects can be interacted with in a variety of ways, through indexing using
     pykx.FloatVector(pykx.q('0.08123546 0.9367503 0.2782122'))
     ```
 
+* Assign objects to PyKX lists
+
+    ```python
+    >>> qarray = kx.random.random(3, 10.0, seed=10)
+    pykx.FloatVector(pykx.q('0.891041 8.345194 3.621949'))
+    >>> qarray[1] = 0.1
+    >>> qarray
+    pykx.FloatVector(pykx.q('0.891041 0.1 3.621949'))
+    ```
+
 * Create a PyKX table and manipulate using Pythonic syntax
 
     ```python

docs/index.md

@@ -8,46 +8,26 @@ For Python developers, PyKX unlocks the speed and power of kdb+ for data process
 
 For q developers, PyKX brings together Python's data science ecosystem and the power of kdb+'s vector and time-series analytics. This makes them available in both q and Python environments. You can use it to run q code within a Python environment or embed Python analytics within your q session.
 
-To begin your journey with PyKX follow the sections in the grid below.
+To begin your journey with PyKX follow the sections below.
 
-<div class="grid cards" markdown>
+## Documentation Breakdown
 
--   :material-run-fast:{ .lg .middle } __Getting Started__
+### [Getting Started](getting-started/what_is_pykx.md)
 
-    ---
+Documentation for users new to PyKX! Contains installation instructions alongside quickstart guides and sample getting started notebooks.
 
-    Documentation for users new to PyKX! Contains installation instructions
-    alongside quickstart guides and sample getting started notebooks.
-
+### [User Guide](user-guide/index.md)
 
-    [:octicons-arrow-right-24: Getting started](getting-started/what_is_pykx.md)
+Useful information allowing users to understand the key concepts behind PyKX. Including how the library is intended to be used and examples of this functionality.
 
--   :material-book-open-variant:{ .lg .middle } __User Guide__
+### [API](api/pykx-execution/q.md)
 
-    ---
+Detailed descriptions of the functions, modules and objects managed by PyKX. Using the API reference assumes you have an understanding of how PyKX is intended to be used through the getting started and user guide.
 
-    Useful information allowing users to understand the key concepts that behind PyKX.
-    Including how the library is intended to be used and examples of this fucntionality.
-
+### [Release Notes](release-notes/changelog.md)
 
-    [:octicons-arrow-right-24: User Guide](user-guide/index.md)
+The latest additions and fixes for PyKX alongside historical changes.
 
--   :material-api:{ .lg .middle } __API Reference__
+### [Roadmap](roadmap.md)
 
-    ---
-
-    Detailed descriptions of the functions, modules and objects managed by PyKX. Using
-    the API reference assumes you have an understanding of how PyKX is intended to be used
-    through the getting started and user guide.
-
-    [:octicons-arrow-right-24: API reference](api/pykx-execution/q.md)
-
--   :material-alert-decagram:{ .lg .middle } __Release Notes__
-
-    ---
-
-    The latest additions and fixes for PyKX alongside historical changes.
-
-    [:octicons-arrow-right-24: Release notes](release-notes/changelog.md)
-
-</div>
+What to look out for in the next weeks, months and years from the PyKX team.