Markov Decision Process, MDP

High level overview of RL

At the heart of RL theory, it is Markov Decision Process.

The general rule we follow is that anything that cannot be changed arbitrarily by the agent is considered to be outside of it and thus part of its environment.

Reinforcement learning, RL is a framework that let an agent to make suitable decisions to achieve best goal. Underneath math problem to solve is a Markov Decision Process, MDP. RL is different from both supervised and unsupervised learning.

Elements of RL

Apart from Agent and Environment, following elements also play central
roles: Policy, Reward Signal, Value Function, and Model of environment.

Policy, is a map from current states to actions to take. It might be
deterministic or stochastic.

Python是个很精巧的语言，但是常见的Cython解释器生成的代码相对来说还是比较慢的，这里主要是跟动态语言的一些特性有关系。但是，我会介绍一下非常常见的手段，可以通过简单的变化提升代码速度：无成本的加速技巧。

:palm_tree: Python到底慢在哪里？

其实，巨大部分的场景我们觉得Python慢是在循环的时候。那么在循环里，我们可以注意如下：

1、避免使用 . 操作

比如如下操作：

一切过程，无论是自然过程还是人工过程，都可以被等价成计算（computation）

1、新科学的基础

简单的规则，可以形成复杂的行为。

如果人也是规则计算的一部分，自由意志又是如何与之协调的？

数学、物理、生物学、社会学、哲学、计算机科学、艺术的关系。这个系统真的解决了这些领域的基本问题吗？？

1. Concepts

Definition. A directed graph or digraph is a set of nodes and a
collection of directed edges. Each directed edge connects an ordered
pair of nodes.

Definition. A directed path is a path in a digraph is a sequence of nodes in which there is a directed edge pointing from each node in the sequence to its successor in the sequence. A directed cycle is a directed path with at least one edge whose first and last nodes are the same. A simple cycle is a cycle with no repeated edges or nodes. The length of a path is its number of edges.

With above, we can define that a node a is reachable from node b if there is a directed path from a to b.

2. Data Structure

When working with graph, search is an important topic. For example, search for connectivity, search for shortest path. There are two basic strategies to do search in graph: Depth-first(DFS) and Breadth-first(BFS). Note that in this blog, all the discussions are based on undirected graph. But the strategy can be used to all kind of graphs given they share similar data structures.

What kind of problems we are solving?

The basic idea of search in general is to walk through the data structure and collection information we need. In terms of Graph, only two elements matters: nodes (vertices) and edges. Walking through a graph, really means iterating the nodes in a way.

Data Structure

The next questions to ask is that how can I solve a question by looping through the least nodes? Well to answer this question, we need to decide a data structure to represent graph.

Graph is a mathematical object to model pairwise connections between objects. There are a lot of applications:

如何阅读

Layer 1 Read

作者退休前是AHL的基金经理，经历过2008年的股灾。

• 这本书的目的
• 作者想要解决的问题
• 作者如何组织材料
• 作者的关键概念有哪些
• 作者的结论是
• 我学到了什么
• 与我有什么关系

Layer 2 Read

• 这一章在讲什么
• 如何联系到作者的写作目的
• 如何联系到上一章
• 关键概念
• 解决了什么问题
• 结论

Layer 3 read: 精读感兴趣的地方，甚至动手操作。

在我们想要结束一段的时候，通常会用V -> I的方式结束。以下我们用C调说明。

比如，G7 -> C:

目前，abcjs 还不支持非浏览器环境的使用，所以，目前似乎没办法变成 Hexo Tag。

Reference

• https://dustinpfister.github.io/2018/01/03/hexo-plugins/
• https://blog.bitbadger.solutions/2017/writing-a-hexo-tag-plugin.html
• https://hexo.io/api/