Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Windows compatibility #417

Open
wants to merge 69 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
3e27b3d
Use wexpect for windows
aristizabal95 Feb 7, 2023
1387fc3
Specify specific click version
aristizabal95 Feb 7, 2023
52a413d
Fix typer incompatibility error
aristizabal95 Feb 13, 2023
9c71384
Handle process text being outputted as text
aristizabal95 Feb 13, 2023
b2e6c11
fix merge
aristizabal95 Feb 21, 2023
01b5795
Update requirements for windows py3
aristizabal95 Feb 21, 2023
28f7e7f
Add Data Preparator cookiecutter template
aristizabal95 Mar 3, 2023
6f9e19e
Rename cookiecutter folder
aristizabal95 Mar 3, 2023
df6e6a2
Temporarily remove possibly offending files
aristizabal95 Mar 3, 2023
a7db0cf
Remove cookicutter conditionals
aristizabal95 Mar 3, 2023
a7a6d15
Inclube back missing pieces of template
aristizabal95 Mar 3, 2023
e2f7108
remove cookiecutter typo
aristizabal95 Mar 3, 2023
581b5bb
Use project_name attribute
aristizabal95 Mar 3, 2023
fd77804
Change cookiecutter fields order
aristizabal95 Mar 3, 2023
6eebd59
Create empty directories on hook
aristizabal95 Mar 3, 2023
5ef86a2
Fix empty folders paths
aristizabal95 Mar 3, 2023
d04baf8
Create evaluator mlcube cookiecutter template
aristizabal95 Mar 6, 2023
02cec01
Fix JSON Syntax Error
aristizabal95 Mar 6, 2023
b3d7a1d
Update template default values
aristizabal95 Mar 6, 2023
7338236
Remove reference to undefined template variable
aristizabal95 Mar 6, 2023
d1cec5e
Implement model mlcube cookiecutter template
aristizabal95 Mar 6, 2023
7338755
Update cookiecutter variable default values
aristizabal95 Mar 6, 2023
3ae9226
Create medperf CLI command for creating MLCubes
aristizabal95 Mar 6, 2023
e07cde2
Provide additional options for mlcube create
aristizabal95 Mar 6, 2023
68e136a
Start working on tests
aristizabal95 Mar 7, 2023
b8e03ac
Add tests for cube create
aristizabal95 Mar 7, 2023
7896b25
Ignore invalid syntax on cookiecutter conditionals
aristizabal95 Mar 7, 2023
4f78981
Ignore more flake8 errors
aristizabal95 Mar 7, 2023
f5dab5e
Remove unused import
aristizabal95 Mar 7, 2023
a03d7f6
Empty commit for cloudbuild
aristizabal95 Mar 8, 2023
6bb60d0
Fix inconsistency with labels paths
aristizabal95 Mar 8, 2023
43b6cab
Update mlcube.yaml so it can be commented on docs
aristizabal95 Mar 8, 2023
55b5d22
Don't render noqa comments on template
aristizabal95 Mar 8, 2023
135c598
Remove flake8 specific ignores
aristizabal95 Mar 8, 2023
e9e2c32
Exclude templates from lint checks
aristizabal95 Mar 8, 2023
e95dab8
Remove specific flake8 ignores
aristizabal95 Mar 8, 2023
d059b7a
Fix labels_paht being passed in he wrong situation
aristizabal95 Mar 10, 2023
fcdaa7b
Add requirements to cookiecutters
aristizabal95 Mar 13, 2023
37f3f3c
Set separate labels as true by default
aristizabal95 Mar 14, 2023
b45fdb9
Merge branch 'main' of https://github.com/mlcommons/medperf
aristizabal95 Mar 22, 2023
fbf02b4
Merge branch 'main' of https://github.com/mlcommons/medperf
aristizabal95 Mar 31, 2023
7a33c23
Remove duplicate templates
aristizabal95 Mar 31, 2023
e9a1190
Merge branch 'main' of https://github.com/mlcommons/medperf
aristizabal95 Apr 5, 2023
f2ff354
Merge branch 'main' of https://github.com/mlcommons/medperf
aristizabal95 Apr 12, 2023
fae57fb
Update requirements
aristizabal95 Apr 19, 2023
1e022c7
Enforce UTF8 file encoding
aristizabal95 Apr 19, 2023
6739bb5
Fix wexpect side effects
aristizabal95 Apr 19, 2023
7671a74
Merge branch 'windows-support' of https://github.com/aristizabal95/me…
aristizabal95 Apr 19, 2023
bc8939b
Undo merge changes
aristizabal95 Apr 19, 2023
f3522a5
Merge branch 'main' of https://github.com/mlcommons/medperf into wind…
aristizabal95 Apr 19, 2023
38255e8
Update setup_logging
aristizabal95 Apr 19, 2023
7683ad7
Use current implementation of test_cubes
aristizabal95 Apr 19, 2023
7060d09
Remove pexpect reference
aristizabal95 Apr 19, 2023
f368555
Remove additional pexpect references
aristizabal95 Apr 19, 2023
984e864
Replace pexpect.spawn to spawn
aristizabal95 Apr 19, 2023
bc39d05
Add timeout to configure test
aristizabal95 Apr 19, 2023
a682021
Merge branch 'main' of https://github.com/mlcommons/medperf
aristizabal95 Apr 19, 2023
b27fde5
Merge branch 'main' of https://github.com/aristizabal95/medperf-2 int…
aristizabal95 Apr 19, 2023
4ea49dd
import pexpect package
aristizabal95 Apr 19, 2023
5ec824e
Formatting changes
aristizabal95 Apr 19, 2023
cc18fb3
Merge branch 'main' into windows-support
aristizabal95 Apr 21, 2023
586609f
Merge branch 'main' into windows-support
hasan7n Apr 24, 2023
09afd50
Update cli/medperf/entities/cube.py
aristizabal95 Apr 25, 2023
8aa66ea
Add configuration validation step
aristizabal95 Apr 25, 2023
0f23c9c
Merge remote-tracking branch 'upstream/main' into windows
hasan7n Aug 20, 2023
0ee1e64
add missing import
hasan7n Aug 20, 2023
f95f591
add a space before ;
hasan7n Aug 20, 2023
9e826d1
remove redundant import
hasan7n Aug 21, 2023
d4095fe
fix linter issues
hasan7n Aug 21, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions cli/medperf/__main__.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
import medperf.commands.profile as profile
from medperf.utils import (
set_custom_config,
validate_config,
set_unique_tmp_config,
init_storage,
setup_logging,
Expand Down Expand Up @@ -100,6 +101,8 @@ def main(
# Set inline parameters
inline_args = ctx.params
set_custom_config(inline_args)
validate_config()

set_unique_tmp_config()

init_storage()
Expand Down
9 changes: 7 additions & 2 deletions cli/medperf/entities/cube.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,16 @@
import os
import sys
import yaml
import pexpect
import logging
from typing import List, Dict, Optional, Union
from pydantic import Field
from pathlib import Path

if sys.platform == "win32":
import wexpect as pexpect
else:
import pexpect

from medperf.utils import combine_proc_sp_text, list_files, storage_path, verify_hash
from medperf.entities.interface import Entity, Uploadable
from medperf.entities.schemas import MedperfSchema, DeployableSchema
Expand Down Expand Up @@ -283,7 +288,7 @@ def run(
proc_out = combine_proc_sp_text(proc)
proc.close()
logging.debug(proc_out)
if proc.exitstatus != 0:
if proc.exitstatus:
raise ExecutionError("There was an error while executing the cube")

logging.debug(list_files(config.storage))
Expand Down
19 changes: 16 additions & 3 deletions cli/medperf/utils.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
from __future__ import annotations

import sys
import re
import os
import yaml
Expand All @@ -14,12 +15,16 @@
import json
from pathlib import Path
import shutil
from pexpect import spawn
from datetime import datetime
from typing import List
from colorama import Fore, Style
from pexpect.exceptions import TIMEOUT

if sys.platform == "win32":
from wexpect import spawn
else:
from pexpect import spawn

import medperf.config as config
from medperf.logging.filters.redacting_filter import RedactingFilter
from medperf.exceptions import ExecutionError, MedperfException, InvalidEntityError
Expand All @@ -28,7 +33,7 @@
def setup_logging(log_lvl):
log_fmt = "%(asctime)s | %(levelname)s: %(message)s"
log_file = storage_path(config.log_file)
handler = handlers.RotatingFileHandler(log_file, backupCount=20)
handler = handlers.RotatingFileHandler(log_file, backupCount=20, encoding="utf-8")
handler.setFormatter(logging.Formatter(log_fmt))
logging.basicConfig(
level=log_lvl,
Expand Down Expand Up @@ -82,6 +87,13 @@ def set_custom_config(args: dict):
setattr(config, param, val)


def validate_config():
"""Validates the configuration is valid for the current machine"""
if config.platform == "singularity" and sys.platform == "win32":
raise MedperfException("Windows doesn't support singularity runner")
# Add additional checks here


def storage_path(subpath: str):
"""Helper function that converts a path to deployment storage-related path"""
server_path = config.server.split("//")[1]
Expand Down Expand Up @@ -372,7 +384,8 @@ def combine_proc_sp_text(proc: spawn) -> str:
except TIMEOUT:
logging.error("Process timed out")
raise ExecutionError("Process timed out")
line = line.decode("utf-8", "ignore")
if line is not None and not isinstance(line, str):
line = line.decode("utf-8", "ignore")
if line:
proc_out += line
ui.print(f"{Fore.WHITE}{Style.DIM}{line.strip()}{Style.RESET_ALL}")
Expand Down
6 changes: 4 additions & 2 deletions cli/requirements.txt
Original file line number Diff line number Diff line change
@@ -1,18 +1,20 @@
typer~=0.6.0
typer~=0.6.0; sys_platform != "win32"
typer<0.5.0; sys_platform == "win32"
rich~=12.5.0
PyYAML>=5.4,<6
requests>=2.26.0
pydantic==1.10.2
yaspin==2.1.0
tabulate==0.8.9
pexpect==4.8.0
wexpect==4.0.0; sys_platform == "win32"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need this for windows?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here I meant to refer to pexpect

colorama==0.4.4
time-machine==2.4.0
pytest-mock==1.13.0
pyfakefs==5.0.0
mlcube @ git+https://github.com/mlcommons/mlcube@fb371c960938b495e939bf38b161199d529cf912#subdirectory=mlcube
mlcube-docker @ git+https://github.com/mlcommons/mlcube@fb371c960938b495e939bf38b161199d529cf912#subdirectory=runners/mlcube_docker
mlcube-singularity @ git+https://github.com/mlcommons/mlcube@fb371c960938b495e939bf38b161199d529cf912#subdirectory=runners/mlcube_singularity
mlcube-singularity @ git+https://github.com/mlcommons/mlcube@fb371c960938b495e939bf38b161199d529cf912#subdirectory=runners/mlcube_singularity ; sys_platform != "win32"
validators==0.18.2
merge-args==0.1.4
synapseclient==2.7.0
Expand Down
Loading