Interesting Observations

A list of observations collected from papers, blogs, talks, and conversations.

Complementary learning systems: example-based and statistical-based

The theory of complementary learning systems in cognitive neuroscience: a combination of episodic (in the hippocampus) and statistical (in the neocortex) learning is important for human task solving. The hippocampus rapidly incorporates new observations to solve tasks in an example-based way, while the neocortex gradually learns abstract rules with greater power and generality.

Source: https://www.ncbi.nlm.nih.gov/pubmed/7624455, https://www.cell.com/trends/cognitive-sciences/fulltext/S1364-6613(16)30043-2

People can learn concepts from positive examples alone

Also, rapid learning from small sample sizes is impossible without a good prior. Prior is the the mechanism by which background knowledge can be brought to bear on a problem.

Source: Word learning as Bayesian inference. Xu Fei, Tenenbaum JB.

Smell and taste is quite hard to imagine, while vision is easy to imagine

no source that I can remember but it shall be from a seminar

Pure deep learning is not necessarily bad at some reasoning task than neuro-symbolic hybrid models

Perhaps the key point is we need to play the right and smart trick and figure out better solutions? And not really about the change of the paradigm to neuro-symbolic hybrid model?

Source: Object-based attention for spatio-temporal reasoning: Outperforming neuro-symbolic models with flexible distributed architectures

A model become to behavior with some mismatch of the real world, long before it has a catastrophic failure

(talks in the session of IBM-MIT What’s next in AI: AI we can trust, 2020 Nov). It perhaps suggested that we need some sort of monitor that can detect and intervene with the updating or re-making of the model very smartly.

Source: Mentioned by Armando Solar-Lezama in a presentation.

Contrastiveness and selectivity are the two major properties of the way humans explain things

Constrastiveness means to use counterfactuals, selectivity means only a few examples are used.

Source: Explanation in artificial intelligence: Insights from the social sciences. Tim Miller.

DNN cannot learn the identity function

Regularity also does not help. How can we make DNN to favor simple models that is simple like a short computer program?

Source: Neural Networks and Occam’s Razor. Daniel Selsam.

Noise is not an annoyance for learning; noise should really helps learning

This is my own observation. Normal ML methods treat random perturbations to be a bad thing. Is it really so? I think noise should actually be what makes learning work.

Source: No reference, yet. This is only my guess. Please tell me if you have any related papers.

The formulation of a game already determines the optimal solution

One may also need to consider the meta-search of the formulation of problem.

search: search the optimal solution given the problem formulation. meta-search: search for different formulation for the same problem.

Source: Search vs meta-search. Daniel Selsam.

True data is complex, some say multi-mode, etc. We need subgroups

Source: A blog by Mario Boley

“There are many students who can start well, but very few that can end well”

Source: By my supervisor Martin Ester, through a weekly meeting.

Reasoning and algorithmic layers are not doing well in learning proper representations

Great negative result.

Source: Assessing SATNet’s Ability to Solve the Symbol Grounding Problem