Incorporate quantifier example #38

haberchr · 2024-04-25T18:38:02Z

No description provided.

…eration results for comparison pre-changes

…, X)

…rom the universe from which a QuantifierModel was generated. Cleaned up generation scripts.

…ents as name is based on properties of referent

…odel dataclass args, successfully generated output

…tonicityMeasurer (still not implemented)

…rammar. QuantifierModel initialized only with name string. Minor bug fixes

…ity measurement

…grammar creation of index primitive rules. Added default weight value of 2.0 for primitive indices

…perators for succinctness

…enerated expressions - can't seem to get more than two int rules to appear in gen exps

…ng property for GrammaticalExpression

… contains for Meaning, folder read of expressions + pkled Universe, generation with inclusive M sizes

…ial... etc.) into 'generate_expressions.py'

Learn quant

… as a function, print grammar upon reading it

…load grammar import, print grammar in serial generations

shanest

Great stuff Chris; very exciting progress! A few smaller things here and there, mostly for cleaning things up, documenting, and things of that sort. Let me know if anything's confusing / unclear :)

shanest · 2024-04-30T00:49:52Z

src/examples/learn_quant/README.md

+
+- `scripts`: a set of scripts for generating `QuantifierModels` and measuring various properties of individual models and sets of models.  These are explained in more detail in the [Usage](#usage) section below.
+- `outputs`: outputs from the generation routines for creating `QuantifierModel`s and `QuantifierUniverse`s
+- `referents.csv`: this file defines the set of points of communication (which will become `Referent`s in ULTK terms).


I think this is out of date now?

shanest · 2024-04-30T00:50:28Z

src/examples/learn_quant/README.md

+- `referents.csv`: this file defines the set of points of communication (which will become `Referent`s in ULTK terms).
+- `meaning.py`: this file defines the meaning space (a `Universe` in ULTK terms) of referents that are individual models of quantifiers (`QuantifierModel`s)
+- `quantifier.py`: defines the subclasses of `ultk`'s `Referent` and `Universe` classes that add additional properties and functionality for modeling quantifier learning
+- `grammar.yml`: defines the Language of Thought grammar (an ULTK `Grammar` is created from this file in one line in `grammar.py`) for this domain, using the five semantic features identified in Haspelmath 1997.


"using the five..." is a copy/paste vestige :)

shanest · 2024-04-30T00:51:04Z

src/examples/learn_quant/README.md

+
+    This script generates the _shortest_ expression (ULTK `GrammaticalExpression`s) for each possible `Meaning` (set of `Referent`s) in the LoT defined in `grammar.py`. In particular, ULTK provides methods for enumerating all grammatical expressions up to a given depth, with user-provided keys for uniqueness and for comparison in the case of a clash.  By setting the former to get the `Meaning` from an expression and the latter to compare along length of the expression, the enumeration method returns a mapping from meanings to shortest expressions which express them.
+
+2. `python -m learn_quant.scripts.generation_text`: generates a `GrammaticalExpression` from the quantifier modeling `Grammar`


generation_text should be generation_test

shanest · 2024-04-30T00:52:53Z

src/examples/learn_quant/conf/config.py

+    name: str
+
+@dataclasses.dataclass
+class UniverseConfig:


Maybe some comments on the fields, i.e. what they are? Two thoughts: (i) Might be good to have better names for m_size and x_size, and (ii) what exactly is depth? That feels like a property of expressions, but does it mean something like max size of a model?

OK, so having read everything now, the point is: depth is used by generate_unique_expressions, which is part of the grammar, to generate expressions. IT's not a property of the Universe, so that argument should be moved elsehwere in the Config, i.e. to a GrammarConfig sub-part or some such

shanest · 2024-04-30T00:53:08Z

src/examples/learn_quant/conf/config.yaml

@@ -0,0 +1,17 @@
+defaults:


chef's kiss!

shanest · 2024-04-30T01:06:02Z

src/examples/learn_quant/quantifier.py

+        object.__setattr__(self, 'B', frozenset([i for i, x in enumerate(self.name) if x in ['1','2']]))
+        object.__setattr__(self, 'M', frozenset([i for i, x in enumerate(self.name) if x in ['0','1','2','3']]))
+
+    @classmethod


shanest · 2024-04-30T01:07:28Z

src/examples/learn_quant/quantifier.py

+                "B": len(self.B), 
+                }
+
+    def to_numpy(self, quantifier_index=None, in_meaning=False):


nice! add type hints and docstring

shanest · 2024-04-30T01:08:25Z

src/examples/learn_quant/quantifier.py

+                raise ValueError("quantifier_index must be a one-dimensional one-hot vector.")
+        else:
+            appended_value = 0
+            if in_meaning:


Not convinced this in_meaning stuff should be part of this method, versus handled elswhere in the data processing, i.e. wherever this is being called from. Because when supplying to the model, the label 1/0 is going to be separate from this array anywyas

To be more explicit: the combination of model + quantifier label are inputs (i.e. x) to a model, and the truth value (which I think is what in_meaning is doing) is the goal/output/y value

shanest · 2024-04-30T01:08:45Z

src/examples/learn_quant/quantifier.py

+        x_size = max(self.x_size, other.x_size)
+        return QuantifierUniverse(referents = self.referents + other.referents, prior= self._prior + other._prior, x_size=x_size)
+
+    def get_names(self) -> list:


shanest · 2024-04-30T01:09:59Z

src/examples/learn_quant/scripts/generate_expressions.py

+
+    return expressions_by_meaning
+
+def generate_expressions(quantifiers_grammar: QuantifierGrammar, 


Please run black for formatting too :). The repo is setup to do that automatically when PRs are merged, but I'm having a hard time reading this method actually

shanest

Great progress Chris! A few more things, again mostly minor. There are some details about the monotonicity calculation that I'm now wondering about. Happy to try to figure those out asynch or wait until our next meeting; let me know!

shanest · 2024-04-30T20:09:17Z

src/examples/learn_quant/monotonicity.py

+    for expression_id, quantifier_expression in enumerate(expressions):
+        print("Calculating monotonicity for: ", quantifier_expression)
+        metrics[str(quantifier_expression)]["monotonicity"] = (
+            upward_monotonicity_entropy(submembership, membership[:, expression_id])


why is submembership the first argument here?

As opposed to the second you mean? I can switch them?

In the original method upward_monotonicity_entropy that this is modified from, the first argument was all_models, which was an array containing all the quantifiermodels in the universe, not just the submodels. But maybe some other logic has changed there?

Ah I see. I might have been testing something and the code drifted into the test. I'll check it out

shanest · 2024-04-30T20:10:56Z

src/examples/learn_quant/monotonicity.py

+    return (1.0 - cond_ent / q_ent)[0, 0]
+
+
+def calculate_monotonicity(universe, expressions, down=False):


type annotations and docstring

shanest · 2024-04-30T20:11:31Z

src/examples/learn_quant/meaning.py

+import argparse
+
+
+def create_universe(m_size, x_size):


type annotations and docstring

shanest · 2024-04-30T20:11:52Z

src/examples/learn_quant/meaning.py

+
+    possible_quantifiers = []
+
+    for x in combinations_with_replacement([0, 1, 2, 3], r=m_size):


very minor style point: best to use a nicer name even for something like x, e.g. for combination in ...

shanest · 2024-04-30T20:13:27Z

src/examples/learn_quant/monotonicity.py

+        names_array.append(quantifier_model.name)
+    truth_matrix = np.array(truth_array)
+    names_vector = np.array(names_array)
+    return truth_matrix, names_vector


There's a mismatch here between type annotation and return type (which is tuple[np.ndarray, np.ndarray]). I'd recommend setting up VSCode with mypy so that it automatically alerts you to these things and they can be spotted quickly :). Also: why does names need to be an array?

Do you recommend one mypy extension over another?

I use the MS one, but don't have any strong/antecedent views here (probably chose it just b/c VSCode is also an MS product, honestly don't remember)

shanest · 2024-04-30T20:14:45Z

src/examples/learn_quant/monotonicity.py

+    return Context(objects, properties, bools)
+
+
+def get_sub_structures(concept_lattice: Context, name: list[str]) -> set[str]:


Let's walk through the logic of the monotonicity calculation in our next meeting and discuss the -set(name) part. In general, the substructure relation in our definition is reflexive, i.e. every structure is a sub-structure of itself; but thtere might be reasons in the way you have things setup to exclude that

shanest · 2024-04-30T20:15:35Z

src/examples/learn_quant/monotonicity.py

+        return num_arr[num, :]
+
+    def has_true_pred(num_arr, y):
+        return np.any(y * num_arr)


Is it true that these get_preds and has_true_pred are no longer needed, i.e. replaced by your lattice-based methods?

Yeah, but I do wonder if we should not do the lattice methods because they might be slower than the logic you've written, I'm not yet sure

The one big issue I foresee is that the methods I have here rely on very strong assumptions about the nature of the quantifiermodels that are no longer true (but were true in Carcassi et al 2021): we're assuming all models are the same size and that there are only two sets ($A \cap B$ and $A \setminus B$). This let us represent models by binary vectors of a fixed length. Your "names" are similar, except now they're 5-ary vectors instead of binary vectors; I'd have to sit down and think whetehr the logic here can be used with those or if it needs to be modified (and, if so, whether that's simple to do or not)

shanest · 2024-04-30T20:18:39Z

src/examples/learn_quant/quantifier.py

+                raise ValueError("quantifier_index must be a one-dimensional one-hot vector.")
+        else:
+            appended_value = 0
+            if in_meaning:


To be more explicit: the combination of model + quantifier label are inputs (i.e. x) to a model, and the truth value (which I think is what in_meaning is doing) is the goal/output/y value

shanest · 2024-04-30T20:21:39Z

src/examples/monotonicity.csv

@@ -0,0 +1,792 @@
+"subset_eq(A, A)",1
+"subset_eq(A, difference(A, B))",1.0


I think something is slightly off here: I don't think this one should be upward monotone, and I think you're only measuring upward monotonicity right now (correct me if this is wrong). For instance: $A \subseteq A \setminus B$ is true only if $A \cap B = \emptyset$, so that $A \setminus B = A$. But in that case, it's possible for there to be models with $B \subseteq B'$ and $A \cap B \neq \emptyset$, in which case thsi quantifier would be false, so it shuoldn't be upward monotone

This might be onnected to the substructure stuff as first argument to the monotonicity calculation?

You are correct, I had not done worked to get anything other than upward monotonicity implemented yet.
Thanks for finding that and spelling that out. I'll see if I can figure out how to rectify this

… merge conflict with Shane's new PR

haberchr and others added 30 commits October 30, 2023 19:37

Initial work on quantifier example

ac25c5b

Subclass referents to make quantifiers, create universe with itertools

3c07573

Merge main into learn_quant

db19d07

Add weights to grammar, add timer and results of timer with small gen…

0c56e06

…eration results for comparison pre-changes

Add small generated expressions AFTER change with times

7ea182f

Merge branch 'main' into learn_quant

5557404

Generation results after Shane's fix

a5697eb

Add generation times after edits

e58b3d9

Ensure elements of M may be any index in alien set in defined range(0…

d7b6e6e

…, X)

Remove X from QuantifierModel - names will not encode anymore the X f…

ffaf43d

…rom the universe from which a QuantifierModel was generated. Cleaned up generation scripts.

Don't generate depth=1

5fc0038

Ensure universe has uniquely named referents - therefor, unique refer…

cda6fef

…ents as name is based on properties of referent

Generated at M=4 X=5 D=3 and still only outputs one expression

4dfa9da

Converted sets to frozensets, set eq=True frozen=True for QuantifierM…

72df8b0

…odel dataclass args, successfully generated output

Add x_size to a subclass of Universe (QuantifierUniverse), begin Mono…

dbf7bec

…tonicityMeasurer (still not implemented)

Update monotonicity measurer

a6881cc

Rules are added for primitives dynamically with new class QuantifierG…

922de09

…rammar. QuantifierModel initialized only with name string. Minor bug fixes

Ensure that truth_values are only calculated dynamically in monotonic…

95585ba

…ity measurement

Fix minor bugs; add iter method to grammar

8b61e53

Add tutorial beginning. TODO: monotonicity. Added fix to indexing in …

62e56b4

…grammar creation of index primitive rules. Added default weight value of 2.0 for primitive indices

Minor progress on tutorial check in, minor changes toward using set o…

e02d2b9

…perators for succinctness

Update tutorial with monotonicity of specific expressions; check in g…

35db091

…enerated expressions - can't seem to get more than two int rules to appear in gen exps

Add initial log file of upward monotonicity

f69dfbc

Comments

70ce61f

Minor changes

5e4e34e

Merge 'main' into 'learn_quant'

af0d87b

Ultk name change, add from_sets QuantifierModel method, add antimeani…

479e2ee

…ng property for GrammaticalExpression

Allow for merging universes, draw function for GrammaticalExpression,…

a8415ba

… contains for Meaning, folder read of expressions + pkled Universe, generation with inclusive M sizes

Roll in different generation procedures (inclusive universes, time tr…

41abd08

…ial... etc.) into 'generate_expressions.py'

No longer needed - now consolidated in 'generated_expressions.py'

11bbf7c

haberchr and others added 10 commits February 4, 2024 16:42

Merge branch 'main' into learn_quant

9f7a17f

Match main branch - correct language test

26e4ca1

Merge branch 'main' into learn_quant

36850ec

Merge pull request #3 from haberchr/learn_quant

0702a8e

Learn quant

Fix func bug

e14367b

Fix tuple(referents) bug to catch up with main, refactor monotonicity…

1203109

… as a function, print grammar upon reading it

Add hydra commanands for generation, monotonicity measurement

45fa077

Fix path issue, print after dynamically adding primitives for indices

5475df3

Create function for converting QuantifierModel to a numpy array

50f6057

change name of antimeaning to complement, remove extraneous 1st time …

1344441

…load grammar import, print grammar in serial generations

shanest requested changes Apr 30, 2024

View reviewed changes

Incorporate comments from PR review, run black

9e6ef27

shanest requested changes Apr 30, 2024

View reviewed changes

haberchr added 4 commits May 8, 2024 20:01

stage fix to calculate_lattice, add docstrings

1859344

Merge in working frozendict change from generic-meaning

d1859d2

Fix change from 'referents' to 'mapping' in meaning

f444a4b

Add binarizing functions, temporary work-in of frozendict, will solve…

0ca5463

… merge conflict with Shane's new PR

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Incorporate quantifier example #38

Incorporate quantifier example #38

haberchr commented Apr 25, 2024

shanest left a comment

shanest Apr 30, 2024

shanest Apr 30, 2024

shanest Apr 30, 2024

shanest Apr 30, 2024

shanest Apr 30, 2024

shanest Apr 30, 2024

shanest Apr 30, 2024

shanest Apr 30, 2024

shanest Apr 30, 2024

shanest Apr 30, 2024

shanest Apr 30, 2024

shanest Apr 30, 2024

shanest left a comment

shanest Apr 30, 2024

haberchr Apr 30, 2024

shanest Apr 30, 2024

haberchr Apr 30, 2024

shanest Apr 30, 2024

shanest Apr 30, 2024

shanest Apr 30, 2024

shanest Apr 30, 2024

haberchr Apr 30, 2024

shanest Apr 30, 2024

shanest Apr 30, 2024

shanest Apr 30, 2024

haberchr Apr 30, 2024

shanest Apr 30, 2024

shanest Apr 30, 2024

shanest Apr 30, 2024

shanest Apr 30, 2024

haberchr Apr 30, 2024


		This script generates the _shortest_ expression (ULTK `GrammaticalExpression`s) for each possible `Meaning` (set of `Referent`s) in the LoT defined in `grammar.py`. In particular, ULTK provides methods for enumerating all grammatical expressions up to a given depth, with user-provided keys for uniqueness and for comparison in the case of a clash. By setting the former to get the `Meaning` from an expression and the latter to compare along length of the expression, the enumeration method returns a mapping from meanings to shortest expressions which express them.

		2. `python -m learn_quant.scripts.generation_text`: generates a `GrammaticalExpression` from the quantifier modeling `Grammar`


		return expressions_by_meaning

		def generate_expressions(quantifiers_grammar: QuantifierGrammar,

		return (1.0 - cond_ent / q_ent)[0, 0]


		def calculate_monotonicity(universe, expressions, down=False):


		possible_quantifiers = []

		for x in combinations_with_replacement([0, 1, 2, 3], r=m_size):

		return Context(objects, properties, bools)


		def get_sub_structures(concept_lattice: Context, name: list[str]) -> set[str]:

		@@ -0,0 +1,792 @@
		"subset_eq(A, A)",1
		"subset_eq(A, difference(A, B))",1.0

Incorporate quantifier example #38

Are you sure you want to change the base?

Incorporate quantifier example #38

Conversation

haberchr commented Apr 25, 2024

shanest left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shanest left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment