Text Generator (Markov Chain)

Markov Chains allow the prediction of a future state based on the characteristics of a present state. Suitable for text, the principle of Markov chain can be turned into a sentence generator.

In the textual context, a first-order Markov chain considers that a given word will be followed by certain words with specific probabilities, calculated from a training corpus.

The process involves three main steps:

— Corpus analysis: Parsing the source text and creating a dictionary of word-to-word transitions. The algorithm maintains a term tree where each node (word) points to its possible successors.

— Transition matrix construction: Calculating the frequencies and probabilities of each transition; each node is associated with a respective frequency value.

— Random generation: Using a pseudo-random generator to select the next word based on the calculated probabilities. This generates a string of words that are likely related.

Parameters

The number of words per sentence counts the words of 3 letters and more and thus ignores most of stop-words. This parameter is an average value, some sentences will be longer and others shorter.

Order 1: Each word depends only on the previous word

Order 2: Each word depends on the two previous words

Order n: Each word depends on the n previous words

The higher the order, the more the generated text respects the local structure of the original corpus, but requires more training data to avoid over-specialization.

The sentences generated by the text generator are often meaningless but seem to follow the grammar of the used language.

Several techniques improve coherence:

— Increasing the order: Moving from order 1 to order 2 or 3

— Corpus filtering: Using high-quality, thematically coherent texts

— Transition weighting: Favoring frequent transitions

— Syntactic constraints: Applying basic grammatical rules

— Post-processing: Correcting punctuation and capitalization after generation

To get better sentences, use artificial intelligences based on natural language semantics (ChatGPT, etc.), these AIs use the principle of Markov chains to determine the next word.