About Project [10] Contribute action functionality

[Rahuljagwani]

@pradeeban
I am proposing a rough implementation regarding Project 10.
Feel free to correct me or add any extra points. Any suggestions and feedbacks will be appreciated.

Implementation:
Back-end:

1. First of all a main python file contribute.py will have to be made containing a flask server with various methods taking api calls from frontend and performing various checks. Few methods can be:
   • Config()
   • Add()
   • Commit()
   • Push()
   • GeneratePR()
   Etc.
   From each method calls have to be made to particular batch files in order to execute git and gh commands on terminal.

2. Test.py containing hard coded api calls to test contribute.py

3. Batch files: .bat files related to each function which has to be called to use cli with customized input and output.

Front-end:
On clicking Contribute in concore-editor a modal will appear which will first take necessary inputs regarding authorization in github and performing necessary checks. It will work in such a way that contribute action can be aborted any time by user. Proper messages will be displayed to user at necessary intervals.
Api calls will be called after passing various checks at certain points and various arguments will be taken by in order to pass them to backend to perform the desired functions.

The following flowchart represents a rough flow about approach which can be taken in order to achieve the project.

Here, it is assumed that user has already created a study/directory and saved it in the local system. In step which is green in flow chart, user has to move that directory in concore folder which he has cloned.

[pradeeban]

Hi @Rahuljagwani,
Good thinking. A few clarifications/additions (correct me if there is any misinterpretation from my side).

I see contribute as more of a script (along the lines of build, debug, and run). So, a contribute and contribute.bat.

"containing a flask server" - FRI Is the Flask server we already have. So I see it is an addition of contribute method, among the rest which is already there. Then connecting the concore script in the backend to the contribute button in concore editor is the same approach as how you have already implemented the other methods.

A few caveats:
We probably don't want to mandate GitHub installations or accounts. We can have it as an option. The idea is, if someone downloads the code from the repository (without git clone, but using the download option - as in https://github.com/ControlCore-Project/concore/archive/refs/tags/2.0.1.zip), they should still be able to use the contribute option. We don't want anyone to rely on GitHub. We have users (neuroscientists) who are not keen on learning all the git commands or install git.

To handle that, we can include a bot account, and commits can be made with the "Author Name" specified in the Study/graphml creation as a comment in the PR, but with the bot as the user.

But we need to secure the credentials of the bot too. Otherwise, a malicious user may use the bot account to spam the repository. We can make some certain valid assumptions. This is the research side of the project. You can think about it (and eventually include the solution in the project proposal draft for further comments). You really don't need to know the answer to everything as well. This is a project that will need a good amount of brainstorming during the community bonding period.

There are also factors to consider on duplicate detection and how a repeated commit on the same study is considered in the centralized repository.

There are also actually no right/wrong answers to this project at this point. This is part of concore. But also addresses an interesting common problem.

[Rahuljagwani]

Thank you for reviewing the approach.
I understood what you explained and I will get back to you after researching more about this and potential approaches you mentioned.

[Rahuljagwani]

Hi @pradeeban, I did research about what was conveyed 2 days ago and tried to roughly implement the same. I have developed some working python script in this regard, which is helping in generating a study based PR. I have used the GitHub-tokens to implement the same. In this implementation user neither needs GitHub account nor git installed in system. The only major problem for now which remains is about encrypting the GitHub-token as it can be misused by any user who is downloading the source code.
For now I have created a separate private (as GitHub-tokens are used) repository for this and I am giving you the access for testing and reviewing it. This is just a rough implementation, so all the features are not developed till now, even frontend is not created effectively but this implementation will generate a PR as you mentioned.
Just for demonstration purpose, I have created a separate flask server which will be dissolved while creating actual project.

To test it:

1. Download the source code or clone the repository : https://github.com/Rahuljagwani/ContributeConcore (This is private repo, I have given you its access)
2. In the terminal, execute

pip3 install -r requirements.txt

3. Then execute,

python3 contribute.py

Make sure the Server starts.

4. Click on https://rahuljagwani.github.io/concore-editor/ (I have created a Contribute modal for demo implementation)
5. Make a Study directory and store it in your local system.
6. Click on contribute
7. Enter all the details correctly . (specially study path of directory which was just created)
8. Click on generate PR and wait for 5-7 sec.
9. A PR will be generated for https://github.com/Rahuljagwani/concore/pulls

The functionality of updating directories which is mentioned in project idea is also working.
Since this is just a rough implementation , a lot of checks and exceptions have to be added. I just want to know , if organization is expecting this kind of implementation or something else.
Let me know if some errors appear.
Please review it and let me know all the points on which more focus should be granted, and also let me know what other features or approach which can be added.
Thank You

[pradeeban]

I have yet to test. But the code and your approach look neat, and I trust it will also work. Great work!

I don't have any more comments at this point. Please feel free to email your proposal drafts with Dr. Mark Arnold and me, when they are ready. (A google doc link to our email addresses should be perfect). We will comment if we need any clarification.

I assume you are drafting two proposals ([1] An Integrated CONTROL-CORE experience and this one).

There is some interesting "near duplicate detection" research problem (a well-studied topic. You can look that up) involved here in the PRs (how will you detect two identical pull requests - are they incremental fixes to the same study?) You don't have to answer that here under this forum. Just give that some thought.

[Rahuljagwani]

Hi @pradeeban,
This is regarding securing GitHub tokens generated in above approach. There are several ways to encrypt generated GitHub token but as soon as user downloads the code, the token will not work in local system because a local environment variable must be present in local user's system due to which need of user's GitHub account pops up for making his own token. Therefore, to overcome this I thought of 2 approaches mentioned below:

1. Different server for contribute script can be made and hosted making it 24x7 active. Any time user wishes to contribute, it will work. As this is open-source code we have to display the code, therefore we can publish the code, but instead of mentioning hardcoded GitHub token in our code we can use GitHub secrets to store the token. Moreover, a proper documentation can be added for future developers regarding this. (Even the whole flask server consisting of all Concore methods can be deployed, making it easy for user)

2. A completely new python package can be developed and published to PyPi such that it can be imported to the contribute script in same server and used. In this way, the use of token will be hidden. Moreover, to publish code the approach mentioned in first method can be used.

I would like you to enlighten me on above.
Thank You

[pradeeban]

I like the approaches. We haven't thought through enough on the contribute option project. So there is a research component involved, which we can sort out during the community bonding period.

One question though - when you say a different server hosted and available all the time, where do you plan to host that? Somewhere in a cloud server where the Mediator is hosted?

[Rahuljagwani]

I like the approaches. We haven't thought through enough on the contribute option project. So there is a research component involved, which we can sort out during the community bonding period.

One question though - when you say a different server hosted and available all the time, where do you plan to host that? Somewhere in a cloud server where the Mediator is hosted?

Exactly, on the cloud server it can be hosted. I guess, we have to give a deep thought on this or you might already have some approach in your mind. If there is already something regarding hosting concore fri, I would like to know.

Moreover, I want thoughts regarding technical aspects of my proposals before deadline, it would be very helpful in deciding the correct approach. I have even mentioned working of sample deliverable in my proposal and want some reviews on that also.
Also, is there need to contact other mentors of project regarding proposal review before submitting it??
Thank You

[pradeeban]

@Rahuljagwani, your proposal is solid. Yes, you can send the proposal as an email link to all the mentors, including me (in the same email, not as multiple emails).

It is unlikely to receive comments unless there is a real need for clarification.

[SudoSu-bham]

Hi @pradeeban, I hope you're doing well. I've been exploring the Concore project with an interest in contributing. I have experience in Python, Java, and C++, and I'm eager to learn more about the Concore idea and its codebase.

I have a couple of questions regarding the project's libraries:

I noticed that the concoredocker.hpp file appears to be empty. Could you please provide some insight into why this file is empty or if there are any plans for its implementation?
As I explored the codebase, I'm want to know if concore.py and concoredocker.py are the only libraries that need to be ported to Java. Could you clarify if there are any additional libraries or components that require attention?
Since I'm relatively new to this codebase, I would greatly appreciate any guidance or further information you can provide to help me navigate and contribute effectively.
Thank you for your time and assistance. Looking forward to hear from you.

[pradeeban]

Hi @SudoSu-bham,
Thanks for your interest and thoughtful question.
If you open that empty file, it states in a comment: "// concoredocker.hpp -- this C++ include file will be the equivalent of concoredocker.py"
That is to indicate this is a TODO.

The C++ implementation was done by a volunteer student (he later participated in GSoC in an unrelated idea on concore).

So, he did not really have the time to have the whole thing ported to C++. Python is the full implementation. You can use the C++ implementation to get inspired by how this porting is to be done. But don't use it as a guideline. That is obviously an incomplete implementation, as far as functionality goes. It is minimal, it works (or worked, at the time of implementation), and quickly done (for what, we appreciate the developer).

You are on the right track. concore.py and concoredocker.py are great places to start. But it does not mean there will be only two java classes. Due to the differences in how code is implemented between languages, I believe, you will actually end up with more files (and classes) in your java implementation.

I also recommend you to read the cited paper. It is open-access, and helps you understand the concept better.

[SudoSu-bham]

I've been working on converting some methods from concoredocker.py to Java, but I'm unsure about how to test them. Could you please provide some guidance on the testing process? Also, I'm not sure where to find the concore.iport and concore.oport files.
Any help you could offer would be great.

[pradeeban]

To test, you are going to have to use your Java implementation allow a similar execution as in https://control-core.readthedocs.io/en/latest/usage.html#writing-concore-programs

https://github.com/ControlCore-Project/concore/blob/main/mkconcore.py shows where these iport and oport files are generated. Open that file and search for iport or oport in it.

https://github.com/ControlCore-Project/concore-lite gives you a full experience with minimal files. So, less distractions, if you want to see a rather minimalistic code base.

[SudoSu-bham]

Hi @pradeeban, I have ported concore.hpp to concore.java . Currently, I'm not getting much time to contribute to this project because I have ongoing internal evaluations and exams at college. However, they will end on 19th April, and I'll be able to focus more on this project afterward. I'm attaching the code below. If you could review it, I'll create a PR for the same.

// concore.java -- this java file will be the equivalent of concore.py
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.Map;

public class concore {
    private static String s = "";
    private static String olds = "";
    private static String inpath = "./in"; // must be relative path for local
    private static String outpath = "./out";
    private static int maxtime;
    
    public static int delay = 1000; // milliseconds
    public static int retrycount = 0;
    public static double simtime;
    public static Map<String, Integer> iport = new HashMap<>();
    public static Map<String, Integer> oport = new HashMap<>();

    public concore(){
        iport = mapParser("concore.iport");
        oport = mapParser("concore.oport");

    }
    public static void main(String[] args) {
        // If Windows, create script to kill this process
        // because batch files don't provide an easy way to know the PID of the last command
        // Ignored for POSIX!=Windows, because "concorepid" is handled by script
        // Ignored for Docker (Linux!=Windows), because handled by docker stop
        if (System.getProperty("os.name").toLowerCase().contains("windows")) {
            try {
                FileWriter fpid = new FileWriter("concorekill.bat");
                fpid.write("taskkill /F /PID " + ProcessHandle.current().pid() + "\n");
                fpid.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }

        default_maxtime(100);
    }

    private static Map<String, Integer> mapParser(String filename) {
        Map<String, Integer> config = new HashMap<>();
        StringBuilder portstr = new StringBuilder(); // StringBuilder to store file data        
        try (BufferedReader reader = new BufferedReader(new FileReader(filename))) {
            int ch;
            while ((ch = reader.read()) != -1) {
                portstr.append((char) ch); // Add character to string builder
            }
        } catch (IOException e) {
            e.printStackTrace();
        }
        
        portstr.setCharAt(portstr.length() - 1, ','); // Replace last character with comma
        portstr.append("}");
        int i = 0;
        String portname = "";
        String portnum = "";

        while(portstr.charAt(i) != '}'){
            if(portstr.charAt(i) == '\''){
                i++;
                while(portstr.charAt(i) != '\''){
                    portname += portstr.charAt(i);
                    i++;
                }
                config.put(portname,0);
            }
            if(portstr.charAt(i) == ':'){
                i++;
                while(portstr.charAt(i) != ','){
                    portnum += portstr.charAt(i);
                    i++;
                }
                config.put(portname,Integer.parseInt(portnum));
                portname = "";
                portnum = "";
            }
            i++;
        }
        return config;
    }

    //function to compare and determine whether file content has been changed
    public static boolean unchanged() {
        if (olds.equals(s)) {
            s = "";
            return true;
        } else {
            olds = s;
            return false;
        }
    }

    private static ArrayList<Double> parser(String f){
        ArrayList<Double> temp = new ArrayList<>();
        String value = "";
        // Changing last bracket to comma to use comma as a delimiter
        f = f.substring(0, f.length() - 1) + ",";
        for(int i=1;i<f.length();i++){
            if(f.charAt(i) != ','){
                value += f.charAt(i);
            }
            else{
                if(value.length() != 0)
                temp.add(Double.parseDouble(value));

                // reset value
                value = "";
            }
        }
        return temp;
    }

    //accepts the file name as string and returns a string of file content
    public static ArrayList<Double> read(int port, String name, String initstr) {
        String ins = "";
        try {Thread.sleep(2 * delay);}
        catch(InterruptedException e){
            e.printStackTrace();
        }
        try (BufferedReader reader = new BufferedReader(new FileReader(inpath + String.valueOf(port) + "/" + name))) {
            int ch;
            while ((ch = reader.read()) != -1) {
                ins += ((char) ch); // Add character to string
            }
        }catch(IOException e){
            ins = initstr;
            e.printStackTrace();
        } 


        while(ins.length() == 0){
            try {Thread.sleep(2 * delay);}
            catch(InterruptedException e){
                e.printStackTrace();
            }
            try (BufferedReader reader = new BufferedReader(new FileReader(inpath + String.valueOf(port) + "/" + name))) {
                int ch;
                while ((ch = reader.read()) != -1) {
                    ins += ((char) ch); // Add character to string
                }
                retrycount++;
            } catch (IOException e) {
                //observed retry count in java from various tests is not calculated yet
                retrycount++;
                System.out.println("Read error");
                e.printStackTrace();
            }
        }
        s += ins;

        ArrayList<Double> inval = new ArrayList<>();
        inval = parser(ins);
        simtime = simtime > inval.get(0) ? simtime : inval.get(0);

        // returning a string with data exluding simtime
        inval.remove(0);
        return inval;
    }

    //write method, accepts a vector double and writes it to the file
    public static void write(int port, String name, ArrayList<Double> val, int delta) {
        try (BufferedWriter writer = new BufferedWriter (new FileWriter(outpath + String.valueOf(port) + "/" + name))){
            val.add(0,simtime+delta);
            writer.write('[');
            for(int i=0;i<val.size()-1;i++){
                writer.write(val.get(i).toString());
                writer.write(',');
            }
            writer.write(val.size()-1);
            writer.write(']');
        } catch (IOException e) {
            System.out.println("skipping +" + outpath + port + " /" + name);
            e.printStackTrace();
        }
    }

    //write method, accepts a string and writes it to the file
    public static void write(int port, String name, String val, int delta) {
        try {Thread.sleep(2 * delay);}
        catch(InterruptedException e){
            e.printStackTrace();
        }
        try (BufferedWriter writer = new BufferedWriter (new FileWriter(outpath + String.valueOf(port) + "/" + name))){
            writer.write(val);
        } catch (IOException e) {
            System.out.println("skipping +" + outpath + port + " /" + name);
            e.printStackTrace();
        }
    }

    public static void default_maxtime(int defaultValue) {
        try(BufferedReader reader = new BufferedReader(new FileReader(inpath+"1/concore.maxtime"))){
            String line = reader.readLine();
            maxtime = Integer.parseInt(line);
            
        } catch (IOException e) {
            maxtime = defaultValue;
        }
    }

    //Initializing
    public static ArrayList<Double> initval(String simtimeVal) {
        //parsing
        ArrayList<Double> val = new ArrayList<>();

        //determining simtime
        simtime = val.get(0);

        //returning the rest of the values(except simtime) in val
        val.remove(0);
        return val;
    }
}

[pradeeban]

Thanks, @SudoSu-bham, for your effort. We appreciate that.
You should create a pull request. That way, your pull request gets a URL. That will also make it easier to review.

[SudoSu-bham]

Hey @Rahuljagwani , I'm trying to understand the Concore codebase and relationship between all the Python files and their respective functions. Since you've already contributed to it, could you please guide me through this? Additionally, there is a contribute.py file. Is there any documentation available for it?"

[pradeeban]

The documentation is in the ReadTheDocs. You can find contribute.py related documentation at https://control-core.readthedocs.io/en/latest/contributeactiondev.html

[SudoSu-bham]

Oh, I was unable to find that earlier. Thank-you