Arman Rysmakhanov

We study in-context binary classification in a simple attention model trained to minimise a logistic loss. Each prompt has a latent logistic teacher, and the learner must infer its direction from Bernoulli-labeled context examples. We derive an asymptotic prediction for the test logit in a high-dimensional regime with context length $\ell \propto d$ and total training prompts $n \propto d^2$. Our formula cleanly separates calibration, test-time context noise, and non-isotropic parameter-estimation noise as interpretable effects on in-context performance. Our analysis also identifies the ridgeless separability regime and addresses the effects of context length, pretraining-set size, regularisation, and sampling temperature on in-context classification performance. The theory's predictions closely agree with finite-dimensional simulations and fully trained linear-attention models, while experiments with standard softmax attention show qualitatively related behaviour beyond the regime assumed by our analysis.

Linear attention is a tractable model for understanding the mechanisms governing in-context learning in transformers. For linear regression tasks, recent asymptotic analyses have characterised its learning and generalisation behaviour. We extend this theory to nonlinear single-index targets, $y = f(x^\top w) + \varepsilon$. Our main result establishes a nonlinearity-noise equivalence: linear attention extracts only the linear Hermite component of $f$, while the remaining nonlinear structure contributes to the generalisation error as effective noise. This reduction allows results from the corresponding linear theory to be transferred to nonlinear tasks. We illustrate its implications for finite pretraining data and for the transition from task memorisation to task generalisation as task diversity increases. These results identify a limitation of the reduced linear-attention model and provide a tractable starting point for studying nonlinear in-context learning.

Preferential attachment graphs model networks whose growth produces highly uneven degree distributions, describing many real-world systems. Their adjacency spectra are important because they allow graph-theoretic questions to be studied through the eigenvalues of matrices. We analyze the adjacency matrix of a dense Barabási-Albert (B-A) multigraph, where the number of edges added at each step is proportional to the final number of vertices. First, we compute the large-$n$ limit of the expected adjacency matrix and show that it is described by a rank-one limiting kernel, viewed as a continuous analogue of the adjacency matrix. After centering and scaling, the fluctuations form a random matrix with a computable variance profile. Using the quadratic vector equation approach, we derive the limiting bulk spectral distribution. We also determine the asymptotic location of the leading eigenvalue generated by the rank-one mean component.

Zeckendorf's Theorem says that for all $k \ge 3$, every nonnegative integer has a unique $k$-Zeckendorf representation as a sum of distinct $k$-bonacci numbers, where no $k$ consecutive $k$-bonacci numbers are present in the representation. Anderson and Bicknell-Johnson extend this result to the multidimensional context: letting the $k$-bonacci vectors $\vec{\mathbf{X}}_i \in \mathbb{Z}^{k-1}$ be given by $\vec{\mathbf{X}}_0 = \vec{\mathbf{0}}$, $\vec{\mathbf{X}}_{-i} = \vec{\mathbf{e}}_i$ for $1 \le i \le k-1$, and $\vec{\mathbf{X}}_n = \sum_{i=1}^{k} \vec{\mathbf{X}}_{n-i}$ for all $n \in \mathbb{Z}$, they show that for all $k \ge 3$, every $\vec{\mathbf{v}} \in \mathbb{Z}^{k-1}$ has a unique $k$-bonacci vector Zeckendorf representation, a sum of distinct $k$-bonacci vectors where no $k$ consecutive $k$-bonacci vectors are present in the representation. Their proof provides an inductive algorithm for finding such representations. We present two improved algorithms for finding the $k$-bonacci vector Zeckendorf representation of $\vec{\mathbf{v}}$ and analyze their relative efficiency. We utilize a projection map $S_n : \mathbb{Z}^{k-1} \to \mathbb{Z}_{\ge 0}$, introduced in Anderson and Bicknell-Johnson work, that reduces the study of $k$-bonacci vector representations to the setting of $k$-bonacci number representations, provided a lower bound is established for the most negatively indexed $k$-bonacci vector present in the $k$-bonacci vector Zeckendorf representation of $\vec{\mathbf{v}}$. Using this map and a bijection between $\mathbb{Z}^{k-1}$ and $\mathbb{Z}_{\ge 0}$, we further show that the number of and gaps between summands in $k$-bonacci vector Zeckendorf representations exhibit the same properties as those in $k$-Zeckendorf representations and that $k$-bonacci vector Zeckendorf representations exhibit summand minimality.

For a finite subset $A$ of a group $G$, we define the right quotient set and the left quotient set of $A$ as $$\begin{aligned} AA^{-1} &:= \{a_1a_2^{-1} : a_1, a_2 \in A\}, \\ A^{-1}A &:= \{a_1^{-1}a_2 : a_1, a_2 \in A\} \end{aligned}$$ respectively. While the right and left quotient sets are equal if $G$ is abelian, subtleties arise when $G$ is a non-abelian group, where the cardinality difference $|AA^{-1}| - |A^{-1}A|$ may take on arbitrarily large values. Using the results of Martin and O'Bryant on the cardinality differences of sum sets and difference sets in $\mathbb{Z}$, we prove that in the infinite dihedral group, $D_\infty \cong \mathbb{Z} \rtimes \mathbb{Z}/2\mathbb{Z}$, every integer difference is achievable. Further, we prove that in $F_2$, the free group on 2 generators, an integer difference is achievable if and only if that integer is even, and we explicitly construct subsets of $F_2$ that achieve every even integer. We further determine the minimum cardinality of $A \subset G$ so that the difference between the cardinalities of the left and right quotient sets is nonzero, depending on the existence of order 2 elements in $G$. To prove these results, we construct difference graphs $D_A$ and $D_{A^{-1}}$ which encode equality in the right and left quotient sets respectively. We observe a bijection from edges in $D_A$ to edges in $D_{A^{-1}}$ and count connected components in order to obtain our results on cardinality differences $|AA^{-1}| - |A^{-1}A|$.

This article serves as a companion to the video essay by Sheafification of G on computing large Fibonacci numbers quickly. We follow the same progression of methods—from naïve recursion to sophisticated Fourier transforms—and elaborate the concepts that were impractical to discuss in video format.