cEP-0026.md: Adds optimize caching cEP #146

palash25 · 2018-05-11T07:04:34Z

No description provided.

jayvdb · 2018-05-11T10:33:48Z

Needs rebase, and needs syntax fixup, otherwise your PR will be drowned in gitmate error comments.

gitmate-bot · 2018-05-11T12:01:33Z

cEP-0026.md

@@ -0,0 +1,188 @@
+# Optimize caching for the NextGen-Core
+
+| Metadata |                                                  |


The text does not comply to the set style.

Origin: MarkdownBear, Section: markdown.

The issue can be fixed by applying the following patch:

--- a/tmp/tmpn6adnkb3/cEP-0026.md +++ b/tmp/tmpn6adnkb3/cEP-0026.md @@ -1,13 +1,13 @@ # Optimize caching for the NextGen-Core -| Metadata | | -| ---------| ------------------------------------------------ | -| cEP | 26 | -| Version | 1.0 | -| Title | Optimize caching for the NextGen-Core | -| Authors | Palash Nigam <[email protected]> | -| Status | Implementation Due | -| Type | Feature | +| Metadata | | +| -------- | ----------------------------------------- | +| cEP | 26 | +| Version | 1.0 | +| Title | Optimize caching for the NextGen-Core | +| Authors | Palash Nigam <mailto:[email protected]> | +| Status | Implementation Due | +| Type | Feature | ## Abstract

gitmate-bot · 2018-05-11T12:01:33Z

cEP-0026.md

+## Implementation
+
+The implementation will be divided into three parts:
+1. Prototype for `FileFactory` class implementation.


The text does not comply to the set style.

Origin: MarkdownBear, Section: markdown.

The issue can be fixed by applying the following patch:

--- a/tmp/tmpn6adnkb3/cEP-0026.md +++ b/tmp/tmpn6adnkb3/cEP-0026.md @@ -33,10 +33,10 @@ ## Implementation The implementation will be divided into three parts: -1. Prototype for `FileFactory` class implementation. -2. Prototype for `Directory` class implementation. -3. Ignore directories functionality. -4. Cache control flags. +1\. Prototype for `FileFactory` class implementation. +2\. Prototype for `Directory` class implementation. +3\. Ignore directories functionality. +4\. Cache control flags. ### Prototype of FileFactory class implementation

gitmate-bot · 2018-05-11T12:03:57Z

cEP-0026.md

@@ -0,0 +1,188 @@
+# Optimize caching for the NextGen-Core
+
+| Metadata |                                       |


The text does not comply to the set style.

Origin: MarkdownBear, Section: markdown.

The issue can be fixed by applying the following patch:

--- a/tmp/tmpth88cx6d/cEP-0026.md +++ b/tmp/tmpth88cx6d/cEP-0026.md @@ -1,13 +1,13 @@ # Optimize caching for the NextGen-Core -| Metadata | | -| ---------| ------------------------------------- | -| cEP | 26 | -| Version | 1.0 | -| Title | Optimize caching for the NextGen-Core | -| Authors | Palash Nigam <[email protected]> | -| Status | Implementation Due | -| Type | Feature | +| Metadata | | +| -------- | ----------------------------------------- | +| cEP | 26 | +| Version | 1.0 | +| Title | Optimize caching for the NextGen-Core | +| Authors | Palash Nigam <mailto:[email protected]> | +| Status | Implementation Due | +| Type | Feature | ## Abstract

gitmate-bot · 2018-05-11T12:03:58Z

cEP-0026.md

+
+## Implementation
+
+The implementation will be divided into three parts:


The text does not comply to the set style.

Origin: MarkdownBear, Section: markdown.

The issue can be fixed by applying the following patch:

--- a/tmp/tmpth88cx6d/cEP-0026.md +++ b/tmp/tmpth88cx6d/cEP-0026.md @@ -33,6 +33,7 @@ ## Implementation The implementation will be divided into three parts: + - Prototype for `FileFactory` class implementation. - Prototype for `Directory` class implementation. - Ignore directories functionality.

gitmate-bot · 2018-05-11T12:09:09Z

cEP-0026.md

@@ -0,0 +1,189 @@
+# Optimize caching for the NextGen-Core
+
+| Metadata |                                            |


The text does not comply to the set style.

Origin: MarkdownBear, Section: markdown.

The issue can be fixed by applying the following patch:

--- a/tmp/tmpcoso1g9x/cEP-0026.md +++ b/tmp/tmpcoso1g9x/cEP-0026.md @@ -1,13 +1,13 @@ # Optimize caching for the NextGen-Core -| Metadata | | -| ---------| ------------------------------------------ | -| cEP | 26 | -| Version | 1.0 | -| Title | Optimize caching for the NextGen-Core | -| Authors | Palash Nigam <mailto:[email protected]> | -| Status | Implementation Due | -| Type | Feature | +| Metadata | | +| -------- | ----------------------------------------- | +| cEP | 26 | +| Version | 1.0 | +| Title | Optimize caching for the NextGen-Core | +| Authors | Palash Nigam <mailto:[email protected]> | +| Status | Implementation Due | +| Type | Feature | ## Abstract

gitmate-bot · 2018-05-11T12:10:19Z

cEP-0026.md

@@ -0,0 +1,189 @@
+# Optimize caching for the NextGen-Core
+
+| Metadata |                                            |


The text does not comply to the set style.

Origin: MarkdownBear, Section: markdown.

The issue can be fixed by applying the following patch:

--- a/tmp/tmpcoso1g9x/cEP-0026.md +++ b/tmp/tmpcoso1g9x/cEP-0026.md @@ -1,13 +1,13 @@ # Optimize caching for the NextGen-Core -| Metadata | | -| ---------| ------------------------------------------ | -| cEP | 26 | -| Version | 1.0 | -| Title | Optimize caching for the NextGen-Core | -| Authors | Palash Nigam <mailto:[email protected]> | -| Status | Implementation Due | -| Type | Feature | +| Metadata | | +| -------- | ----------------------------------------- | +| cEP | 26 | +| Version | 1.0 | +| Title | Optimize caching for the NextGen-Core | +| Authors | Palash Nigam <mailto:[email protected]> | +| Status | Implementation Due | +| Type | Feature | ## Abstract

Makman2 · 2018-05-12T10:35:17Z

cEP-0026.md

+performance.
+
+The current caching used by the old core can be improved upon and integrated
+with the NextGen-Core by adding a few more enhancements to the current caching


integrated with the NextGen-Core

you can integrate the old stuff into the new one? I'm wondering :D

That's not exactly what I meant. What I meant was that we try to take the improvements suggested for the old cache and try to integrate it with the new cache by putting a little spin on it. I can see how that can be confusing.

I now realise that its not important to include that, its confusing. Deleting it.

Makman2 · 2018-05-12T10:36:09Z

cEP-0026.md

+to interface with the directories, ignoring unmodified directories during
+successive runs and adding cache control flags for better usability and
+providing the user various options to perform caching according to a project's
+needs.


I think you should mention the already existing caching principle with tasks above in a little paragraph (just some basic explanations), which is fundamental for all new caching features.

Makman2 · 2018-05-12T10:36:21Z

cEP-0026.md

+
+## Implementation
+
+The implementation will be divided into four parts


Add colon ;)

Makman2 · 2018-05-12T10:41:34Z

cEP-0026.md

+provide the file contents in different forms for e.g. string, binary
+format, etc. The biggest benefit that we will get out of this class would be
+that we can replace the actual file contents in our file-dict with these
+objects which will itself be a performance improvement.


I would like to see a bit more explanation, especially that file contents are lazy-loaded.

That shouldn't be too hard to also reflect this in the prototype roughly :)

By lazy-loading you mean read the file in small chunks? Should I add a chunk argument like
file.read(chunk_size) or open(filename, buffering=chunk_size) ?

For example, or reading it only when the user really wants to acces file contents (so not loading it immediately inside __init__)

Makman2 · 2018-05-12T10:42:24Z

cEP-0026.md

+        self._content = open(self.filename)
+
+    def open(filename, newline=None):
+        return FileFactory(filename, newline=newline)


has to be a classmethod :D

And then ->

@classmethod def open(cls, ...): return cls(...)

Makman2 · 2018-05-12T10:55:18Z

cEP-0026.md

+### Prototype of Directory class implementation
+
+The objects of this class will also reside in the file-dict (need to think of
+a better name for the file-dict since it will no longer contain any files).


In the end it will become a file-set / directory-set :D

Makman2 · 2018-05-12T10:55:46Z

cEP-0026.md

+a better name for the file-dict since it will no longer contain any files).
+The objects of this class can also be used by `DirectoryBears` as discussed
+on this thread [#4999](https://github.com/coala/coala/issues/4999) by performing
+a simple type check on the entries in the dict, whether the objects belong to


belong to -> are instances of

Makman2 · 2018-05-12T10:56:24Z

cEP-0026.md

+            if os.path.isfile(child):
+                self.files += 1
+
+        self.subdirs = len(self.children) - self.files


Do you have a usecase for count_children?

Not really its only used to calculate the subdirs. I guess you are right this shouldn't be an attribute of the class but just a temporary variable so I will keep children but remove self from it.

Makman2 · 2018-05-12T10:58:38Z

cEP-0026.md

+- `--cache-compression`: This will accept the following arguments: `none`
+   for compression to be disabled and `lzma` or `gzip` to compress the cache
+   using these modules.
+- `cache-optimization`: For faster cache loading we can make use of this flag


I think it should be called --cache-optimize ;)

Makman2 · 2018-05-12T10:59:35Z

cEP-0026.md

+   using these modules.
+- `cache-optimization`: For faster cache loading we can make use of this flag
+   which will utilize `pickletools.optimize` for faster cache loading.
+- `export-cache` or `import-cache`: To transfer cache across different


I would add "API":

...: API to transfer cache ...

As it's yet to be designed how coala will store the cache ;)

shreyans800755 · 2018-05-13T20:48:50Z

cEP-0026.md

+| cEP      | 26                                        |
+| Version  | 1.0                                       |
+| Title    | Optimize caching for the NextGen-Core     |
+| Authors  | Palash Nigam <mailto:[email protected]> |


No need to show mailto.
Do something like [mailto](mailto:[email protected])

I needed to add maito because gitmate suggested it (I just copy pasted the suggested diff). It might give me errors if I try to change it now.

Can you please try it locally and check if gitmate is throwing errors ?
If so, we can file an issue about.

shreyans800755 · 2018-05-13T20:55:15Z

cEP-0026.md

+
+This class will be implemented according to the Factory design pattern and will
+act as a proxy to dealing with files. This class will have various methods to
+provide the file contents in different forms for e.g. string, binary


in different forms like string, binary...

shreyans800755 · 2018-05-13T21:01:38Z

cEP-0026.md

+This class will be implemented according to the Factory design pattern and will
+act as a proxy to dealing with files. This class will have various methods to
+provide the file contents in different forms for e.g. string, binary
+format, etc. The biggest benefit that we will get out of this class would be


Please rephrase it as something like:
With the help of this class, we can replace...
I would suggest not to keep too long sentences. You should break it into multiple sentences. Keep it concise and to the point.

shreyans800755 · 2018-05-13T21:04:39Z

cEP-0026.md

+
+The current caching used by the old core can be improved upon and integrated
+with the NextGen-Core by adding a few more enhancements to the current caching
+mechanism like `FileFactory` class to load files, `Directory` class


FileFactory & Directory are not mechanisms. Please rephrase it. 3rd party might not be aware of these classes before reading the details. Its not a good idea to put it in intro.

You can just say extra classes/method to interface with files/dirs smoothly

shreyans800755 · 2018-05-13T21:07:04Z

cEP-0026.md

+
+### Ignore directories functionality
+
+The will probably be the biggest performance boost for coala. Instead of only


Typo in the first word: The -> This

shreyans800755 · 2018-05-13T21:09:18Z

cEP-0026.md

+### Ignore directories functionality
+
+The will probably be the biggest performance boost for coala. Instead of only
+ignoring unmodified files on subsequent runs coala shall also ignore directories


Please add comma after runs

shreyans800755 · 2018-05-13T21:11:57Z

cEP-0026.md

+  `FileFactory.parent` and decrease the count of the number of files
+  (`Directory.files` in the directory-tracker dict) corresponding to that path
+  by one.
+- Once all the files have been deleted from the cache coala will move on to the


comma after deleted from the cache

shreyans800755 · 2018-05-13T21:30:18Z

cEP-0026.md

+
+### Cache control flags
+
+This feature enhancement focusses both on performance and usability. The aim


focusses -> focuses
both on -> on both

shreyans800755 · 2018-05-13T21:31:27Z

cEP-0026.md

+   and directories on subsequent coala runs and `lru` (last recently used) in
+   which cached items only persist until the next run (a count parameter can
+   also be used to know when to discard an unused cached item).
+- `--clear-cache`: Similar to `--flush-cache`.


Are we providing both --clear-cache & --flush-cache ?

I'm not sure
ping @Makman2

We can to not break API

Why can't we go ahead with only --flush-cache ? Any particular reason for --clear-cache ?

We even should ;)

shreyans800755 · 2018-05-13T21:32:23Z

cEP-0026.md

+## Documentation
+
+Necessary changes will be made to the NextGen-Core documentation once the
+features have been implemented. Besides the NextGen docs a document solely


comma after NextGen docs

li-boxuan · 2018-05-16T03:25:41Z

cEP-0026.md

+   machines (like the user's local machine and the CI servers) and speed up
+   builds.
+
+## Documentation


As #123 (comment) suggests, Documentation should better not appear in cEPs.

palash25 · 2018-05-19T05:59:12Z

cEP-0026.md

+    def __init__(self, filename, parent):
+        self.filename = filename # absolute file path
+        self.parent = os.path.abspath(os.path.join(p, os.pardir)) # parent dir
+        self._content = open(self.filename)


I just realised that I have imported linecache but not used it. Should we use it here to access the file contents instead of open do something like

def __init__(self): self.lines = linecache.getlines(self.filename) def file_as_string(self): "".join(self.lines)

If you can already provide some prototype, yeah why not :D

But please don't cache the filename itself :P

gitmate-bot · 2018-05-20T16:49:35Z

cEP-0026.md

+            return fp.read()
+
+    def file_as_string(self):
+      	return self.file_as_bytes().decode()


Line contains following spacing inconsistencies:

Tabs used instead of spaces.

Origin: SpaceConsistencyBear, Section: spacing.

The issue can be fixed by applying the following patch:

--- a/tmp/tmp5gqzy4y3/cEP-0026.md +++ b/tmp/tmp5gqzy4y3/cEP-0026.md @@ -71,7 +71,7 @@ return fp.read() def file_as_string(self): - return self.file_as_bytes().decode() + return self.file_as_bytes().decode() def file_as_list(self): self.lines = linecache.getlines(self.filename)

gitmate-bot · 2018-05-20T16:49:36Z

cEP-0026.md

+# Optimize caching for the NextGen-Core
+
+| Metadata |                                           |
+| ---------| ----------------------------------------- |


The text does not comply to the set style.

Origin: MarkdownBear, Section: markdown.

The issue can be fixed by applying the following patch:

--- a/tmp/tmp5gqzy4y3/cEP-0026.md +++ b/tmp/tmp5gqzy4y3/cEP-0026.md @@ -1,7 +1,7 @@ # Optimize caching for the NextGen-Core | Metadata | | -| ---------| ----------------------------------------- | +| -------- | ----------------------------------------- | | cEP | 26 | | Version | 1.0 | | Title | Optimize caching for the NextGen-Core |

palash25 · 2018-05-20T16:49:56Z

cc @Makman2 @shreyans800755 @fneu please review.

gitmate-bot · 2018-05-20T16:54:44Z

cEP-0026.md

@@ -0,0 +1,189 @@
+# Optimize caching for the NextGen-Core
+
+| Metadata  |                                           |


The text does not comply to the set style.

Origin: MarkdownBear, Section: markdown.

The issue can be fixed by applying the following patch:

--- a/tmp/tmppxdxm84o/cEP-0026.md +++ b/tmp/tmppxdxm84o/cEP-0026.md @@ -1,13 +1,13 @@ # Optimize caching for the NextGen-Core -| Metadata | | -| --------- | ----------------------------------------- | -| cEP | 26 | -| Version | 1.0 | -| Title | Optimize caching for the NextGen-Core | -| Authors | Palash Nigam <mailto:[email protected]> | -| Status | Implementation Due | -| Type | Feature | +| Metadata | | +| -------- | ----------------------------------------- | +| cEP | 26 | +| Version | 1.0 | +| Title | Optimize caching for the NextGen-Core | +| Authors | Palash Nigam <mailto:[email protected]> | +| Status | Implementation Due | +| Type | Feature | ## Abstract

gitmate-bot · 2018-05-20T16:57:30Z

cEP-0026.md

+            return fp.read()
+
+    def file_as_string(self):
+      	return self.file_as_bytes().decode()


Line contains following spacing inconsistencies:

Tabs used instead of spaces.

Origin: SpaceConsistencyBear, Section: spacing.

The issue can be fixed by applying the following patch:

--- a/tmp/tmp5gqzy4y3/cEP-0026.md +++ b/tmp/tmp5gqzy4y3/cEP-0026.md @@ -71,7 +71,7 @@ return fp.read() def file_as_string(self): - return self.file_as_bytes().decode() + return self.file_as_bytes().decode() def file_as_list(self): self.lines = linecache.getlines(self.filename)

gitmate-bot · 2018-05-20T16:57:31Z

cEP-0026.md

+# Optimize caching for the NextGen-Core
+
+| Metadata |                                           |
+| ---------| ----------------------------------------- |


The text does not comply to the set style.

Origin: MarkdownBear, Section: markdown.

The issue can be fixed by applying the following patch:

--- a/tmp/tmp5gqzy4y3/cEP-0026.md +++ b/tmp/tmp5gqzy4y3/cEP-0026.md @@ -1,7 +1,7 @@ # Optimize caching for the NextGen-Core | Metadata | | -| ---------| ----------------------------------------- | +| -------- | ----------------------------------------- | | cEP | 26 | | Version | 1.0 | | Title | Optimize caching for the NextGen-Core |

gitmate-bot · 2018-05-20T17:17:53Z

cEP-0026.md

@@ -0,0 +1,189 @@
+# Optimize caching for the NextGen-Core
+
+| Metadata  |                                           |


The text does not comply to the set style.

Origin: MarkdownBear, Section: markdown.

The issue can be fixed by applying the following patch:

--- a/tmp/tmppxdxm84o/cEP-0026.md +++ b/tmp/tmppxdxm84o/cEP-0026.md @@ -1,13 +1,13 @@ # Optimize caching for the NextGen-Core -| Metadata | | -| --------- | ----------------------------------------- | -| cEP | 26 | -| Version | 1.0 | -| Title | Optimize caching for the NextGen-Core | -| Authors | Palash Nigam <mailto:[email protected]> | -| Status | Implementation Due | -| Type | Feature | +| Metadata | | +| -------- | ----------------------------------------- | +| cEP | 26 | +| Version | 1.0 | +| Title | Optimize caching for the NextGen-Core | +| Authors | Palash Nigam <mailto:[email protected]> | +| Status | Implementation Due | +| Type | Feature | ## Abstract

gitmate-bot · 2018-05-20T17:21:12Z

cEP-0026.md

@@ -0,0 +1,189 @@
+# Optimize caching for the NextGen-Core
+
+| Metadata  |                                           |


The text does not comply to the set style.

Origin: MarkdownBear, Section: markdown.

The issue can be fixed by applying the following patch:

--- a/tmp/tmppxdxm84o/cEP-0026.md +++ b/tmp/tmppxdxm84o/cEP-0026.md @@ -1,13 +1,13 @@ # Optimize caching for the NextGen-Core -| Metadata | | -| --------- | ----------------------------------------- | -| cEP | 26 | -| Version | 1.0 | -| Title | Optimize caching for the NextGen-Core | -| Authors | Palash Nigam <mailto:[email protected]> | -| Status | Implementation Due | -| Type | Feature | +| Metadata | | +| -------- | ----------------------------------------- | +| cEP | 26 | +| Version | 1.0 | +| Title | Optimize caching for the NextGen-Core | +| Authors | Palash Nigam <mailto:[email protected]> | +| Status | Implementation Due | +| Type | Feature | ## Abstract

gitmate-bot · 2018-05-20T17:23:29Z

cEP-0026.md

+            return fp.read()
+
+    def file_as_string(self):
+      	return self.file_as_bytes().decode()


Line contains following spacing inconsistencies:

Tabs used instead of spaces.

Origin: SpaceConsistencyBear, Section: spacing.

The issue can be fixed by applying the following patch:

--- a/tmp/tmp5gqzy4y3/cEP-0026.md +++ b/tmp/tmp5gqzy4y3/cEP-0026.md @@ -71,7 +71,7 @@ return fp.read() def file_as_string(self): - return self.file_as_bytes().decode() + return self.file_as_bytes().decode() def file_as_list(self): self.lines = linecache.getlines(self.filename)

gitmate-bot · 2018-05-20T17:23:30Z

cEP-0026.md

+# Optimize caching for the NextGen-Core
+
+| Metadata |                                           |
+| ---------| ----------------------------------------- |


The text does not comply to the set style.

Origin: MarkdownBear, Section: markdown.

The issue can be fixed by applying the following patch:

--- a/tmp/tmp5gqzy4y3/cEP-0026.md +++ b/tmp/tmp5gqzy4y3/cEP-0026.md @@ -1,7 +1,7 @@ # Optimize caching for the NextGen-Core | Metadata | | -| ---------| ----------------------------------------- | +| -------- | ----------------------------------------- | | cEP | 26 | | Version | 1.0 | | Title | Optimize caching for the NextGen-Core |

gitmate-bot · 2018-05-20T17:25:59Z

cEP-0026.md

+            return fp.read()
+
+    def file_as_string(self):
+      	return self.file_as_bytes().decode()


Line contains following spacing inconsistencies:

Tabs used instead of spaces.

Origin: SpaceConsistencyBear, Section: spacing.

The issue can be fixed by applying the following patch:

--- a/tmp/tmp5gqzy4y3/cEP-0026.md +++ b/tmp/tmp5gqzy4y3/cEP-0026.md @@ -71,7 +71,7 @@ return fp.read() def file_as_string(self): - return self.file_as_bytes().decode() + return self.file_as_bytes().decode() def file_as_list(self): self.lines = linecache.getlines(self.filename)

gitmate-bot · 2018-05-20T17:26:00Z

cEP-0026.md

+# Optimize caching for the NextGen-Core
+
+| Metadata |                                           |
+| ---------| ----------------------------------------- |


The text does not comply to the set style.

Origin: MarkdownBear, Section: markdown.

The issue can be fixed by applying the following patch:

--- a/tmp/tmp5gqzy4y3/cEP-0026.md +++ b/tmp/tmp5gqzy4y3/cEP-0026.md @@ -1,7 +1,7 @@ # Optimize caching for the NextGen-Core | Metadata | | -| ---------| ----------------------------------------- | +| -------- | ----------------------------------------- | | cEP | 26 | | Version | 1.0 | | Title | Optimize caching for the NextGen-Core |

shreyans800755 · 2018-05-20T20:51:25Z

cEP-0026.md

+   and directories on subsequent coala runs and `lru` (last recently used) in
+   which cached items only persist until the next run (a count parameter can
+   also be used to know when to discard an unused cached item).
+- `--clear-cache`: Similar to `--flush-cache`.


Why can't we go ahead with only --flush-cache ? Any particular reason for --clear-cache ?

shreyans800755 · 2018-05-20T20:58:00Z

cEP-0026.md

+their project's needs. The following cache flags can be implemented to
+provide different caching modes:
+
+- `--cache-protocol`: This will accept three different arguments, `none` if


Can you please mention the default cache protocol if user doesn't specify it ?

Makman2 · 2018-05-20T21:03:21Z

cEP-0026.md

+This class will follow the Factory design pattern and will act as a proxy to
+dealing with files. This class will have various methods to provide the file
+contents in different forms like string, binary format, etc. With the help of
+this class that we can replace the actual file contents in our file-dict with


Makman2 · 2018-05-20T21:07:54Z

cEP-0026.md

+
+    @classmethod
+    def open(cls, filename, newline=None):
+        return cls(filename, newline=newline)


newline is not present in __init__^^ There's parent instead

Makman2 · 2018-05-20T21:25:17Z

cEP-0026.md

+
+    def file_as_list(self):
+        self.lines = linecache.getlines(self.filename)
+        self.linecount = len(self.lines)


hm a bit inconsistent that this function suddenly uses linecache while the others don't ;) but not so crucial right now.

Makman2 · 2018-05-20T21:26:25Z

cEP-0026.md

+### Prototype of Directory class implementation
+
+The objects of this class will also reside in the file-set (formerly known as
+the file-dict) The objects of this class can also be used by `DirectoryBears` as


missing period before "The"

Makman2 · 2018-05-20T21:26:38Z

cEP-0026.md

+The objects of this class will also reside in the file-set (formerly known as
+the file-dict) The objects of this class can also be used by `DirectoryBears` as
+discussed on this thread [#4999](https://github.com/coala/coala/issues/4999) by
+performing a simple type check on the entries in the dict, whether the objects


dict --> file-set :)

Makman2 · 2018-05-20T21:36:03Z

cEP-0026.md

+- `--cache-protocol`: This will accept three different arguments, `none` if
+   the user wants to run coala without caching enabled, `primitive` to store
+   all the cache entries without removing the unchanged ones on subsequent runs
+   (memory wise quite inefficient but this will prove to be useful in project's


project's --> projects

Makman2 · 2018-05-20T21:36:39Z

cEP-0026.md

+   the user wants to run coala without caching enabled, `primitive` to store
+   all the cache entries without removing the unchanged ones on subsequent runs
+   (memory wise quite inefficient but this will prove to be useful in project's
+   with small incremental builds that need fast linting), `untrack` the


especially with many incremental builds

You can maybe say "sufficiently small", up to you 👍

Makman2 · 2018-05-20T21:38:19Z

cEP-0026.md

+   opposite of the `primitive` mode which will stop tracking unchanged files
+   and directories on subsequent coala runs and `lru` (last recently used) in
+   which cached items only persist until the next run (a count parameter can
+   also be used to know when to discard an unused cached item).


Could you make a sublist with bullets for the different caching modes? I mean like

- `none`: blablabla - `primitive`: blablbla - ...

That gives a reader a better overview 👍

Makman2 · 2018-05-20T21:39:34Z

cEP-0026.md

+- `--cache-compression`: This will accept the following arguments: `none`
+   for compression to be disabled and `lzma` or `gzip` to compress the cache
+   using these modules.
+- `cache-optimize`: For faster cache loading we can make use of this flag


--> --cache-optimize (dash dash cache optimize 😄 )

Makman2 · 2018-05-20T21:40:33Z

cEP-0026.md

+  `Directory.subdirs` of their parent directory will also be decreased by 1.
+- This process will continue for all the directories and the paths having both
+  the values (`Directory.files` and `Directory.subdirs`) as zero will have
+  their corresponding `Directory` objects deleted from the cache.


Still a bit vague, but I know this is currently one of our harder parts in this cEP, so we can initially merge it and improve it later on.

shreyans800755 · 2018-05-23T19:57:11Z

cEP-0026.md

+
+    def file_as_list(self):
+        self.lines = linecache.getlines(self.filename)
+        self.linecount = len(self.lines)


Do we really need linecount variable ?
It is implicit (at least right now). I think we can skip it for now, and add in the implementation if it is indeed required.

shreyans800755 · 2018-05-23T19:59:25Z

cEP-0026.md

+
+    def hash(self):
+        # hash the absolute file path to get a unique hash
+        return persistent_hash(self.filename)


We might wanna store it as data member depending on the complexity of this function call.
Although it can be taken care of while implementing. So no need to change it right now. Just keep this in mind.

shreyans800755 · 2018-05-23T20:00:59Z

cEP-0026.md

+class Directory:
+    def __init__(self, path):
+        self.path = path
+        self.parent = os.path.abspath(os.path.join(p, os.pardir))


Where is this p coming from in the first param of os.path.join ? Do you mean path ?

Yeah it should be path. Sorry about that. Will change it.

shreyans800755 · 2018-05-25T22:08:16Z

ack 7272ff6

gitmate-bot · 2018-05-25T22:08:18Z

Sorry @shreyans800755, you do not have the necessary permission levels to perform the action.

Makman2 · 2018-05-26T11:16:13Z

ack 7272ff6

Makman2 · 2018-05-26T11:16:55Z

@gitmate-bot ff

gitmate-bot · 2018-05-26T11:16:57Z

Hey! I'm GitMate.io! This pull request is being fastforwarded automatically. Please DO NOT push while fastforward is in progress or your changes would be lost permanently ⚠️

gitmate-bot · 2018-05-26T11:17:45Z

Automated fastforward with GitMate.io was successful! 🎉

gitmate-bot added process/pending review size/S labels May 11, 2018

jayvdb requested a review from ksdme May 11, 2018 10:16

jayvdb added process/wip and removed process/pending review labels May 11, 2018

gitmate-bot added process/pending review and removed process/wip labels May 11, 2018

gitmate-bot reviewed May 11, 2018

View reviewed changes

jayvdb requested review from Makman2, shreyans800755 and fneu May 12, 2018 06:33

palash25 changed the title ~~cEP-0026.md: Adds optimize caching cEP~~ [WIP] cEP-0026.md: Adds optimize caching cEP May 12, 2018

palash25 changed the title ~~[WIP] cEP-0026.md: Adds optimize caching cEP~~ cEP-0026.md: Adds optimize caching cEP May 12, 2018

Makman2 requested changes May 12, 2018

View reviewed changes

shreyans800755 suggested changes May 13, 2018

View reviewed changes

li-boxuan reviewed May 16, 2018

View reviewed changes

palash25 commented May 19, 2018

View reviewed changes

gitmate-bot reviewed May 20, 2018

View reviewed changes

shreyans800755 reviewed May 20, 2018

View reviewed changes

Makman2 requested changes May 20, 2018

View reviewed changes

shreyans800755 suggested changes May 23, 2018

View reviewed changes

cEP-0026.md: Adds optimize caching cEP

7272ff6

gitmate-bot added process/approved and removed process/pending review labels May 26, 2018

Makman2 approved these changes May 26, 2018

View reviewed changes

gitmate-bot merged commit 7272ff6 into coala:master May 26, 2018

palash25 deleted the caching-cep branch May 26, 2018 11:20

palash25 mentioned this pull request May 26, 2018

optimize_caching.md: Change project description coala/projects#578

Merged

2 tasks

		@@ -0,0 +1,188 @@
		# Optimize caching for the NextGen-Core

		\| Metadata \| \|


		## Implementation

		The implementation will be divided into three parts:

		@@ -0,0 +1,189 @@
		# Optimize caching for the NextGen-Core

		\| Metadata \| \|


		### Ignore directories functionality

		The will probably be the biggest performance boost for coala. Instead of only


		### Cache control flags

		This feature enhancement focusses both on performance and usability. The aim

cEP-0026.md: Adds optimize caching cEP #146

cEP-0026.md: Adds optimize caching cEP #146

Conversation

palash25 commented May 11, 2018

jayvdb commented May 11, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

palash25 May 13, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

palash25 May 19, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

palash25 commented May 20, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

palash25 May 13, 2018 •

edited

Loading

palash25 May 19, 2018 •

edited

Loading