Meaning is an orphan. A notion can be written, spoken, drawn, or conveyed by a sharp look that makes further scholarship quite unnecessary. These are entirely different phenomena. They arrive by means unalike and recent scholarship would have it that they are perceived very differently. And yet, something survives the journey enough for us to reconcile disparate messages into the same thing. And if said thing is something we need machines to understand, it seems very suspicious that the way to go about it is to divide the world into text, images and audio.

Modalities are a rather dubious concept in machine learning. It's quite possible that the way we understand them may be a principal reason we seem so close and yet so far from true intelligence. It has a certain promiscuity to it in that its meaning is highly mobile across at least 4 distinct readings.

The diagram distinguishes four meanings of modality. These are levels of description, rather than consecutive stages in a process.

1. Physical substrate or transducer. What produced the signal? Examples include photons falling on a sensor and pressure waves reaching a microphone.
2. Encoding. How is the data represented? Examples include a grid of floating point numbers, a sequence of discrete tokens, and a graph of nodes and edges.
3. Statistical structure. What regularities does the data have? The diagram shows the same spatial pattern shifted one cell to the right, illustrating translation equivariance. It also shows the ordered tokens “the bird sings”, illustrating sequence order in text.
4. Semantic domain. What is the data about? A generated picture of a songbird, a schematic bird call, and the sentence “a bird sings” share the same subject. The waveform is an illustration, not a recording of the pictured bird.

01

Physical substrate / transducer

What produced the signal?

photons → sensor

pressure → microphone

02

Encoding

How is it represented?

floats

tokens

graph

03

Statistical structure

What regularities does it have?

translation equivariance

sequence order

04

Semantic domain

What is it about?

“a bird sings”

the same subject

This seems hilariously vague the more you think about it. A spectrogram is audio on one level and an image on another. Text is image in one sense, audio in another, and something else entirely in your brain. Training frontier models seems to be a rather absurd endeavour in coaxing a machine to equivocate across these categories fluently.

This arose, perhaps, because we individuate the senses by organ, by stimulus, by phenomenological character or by content. Quine contends that our sorting of the world into kinds rests on an innate sense of similarity, a "quality space" that evolution tuned to our sensory apparatus that is good for induction. A great big accident of the primaeval ooze. Ordinary talk of the “senses” or “modalities” is about as well-defined as the rules of engagement in a sixteenth-century European skirmish.

You'll Know What We Did Last Summer(s)

20-25 year old Californian women

What does this actually represent? The trivial reading is that it is a sequence of symbols. The playbook here being training an embedding model to place these symbols in vector space close to where it similar statements co-occur. The more pretentious reading is that this is a distribution that can be represented by some Gaussian mixture.

If you hold this reading (and we do), it doesn't make any sense to embed these symbols by statistically torturing existing text corpora. The immediate objection is that this is a very nice piece of navel-gazing, but a very poor experimental strategy to arrive at anything approximating a model.

Contrary to our own expectations, we did in fact arrive at a lot more than an embedding model a year later (see Giedi 2).

It Works in a Browser

Suppose we want to predict how a population would answer a survey. We might ask how the prediction changes if the population is older, if we consider only graduates, or if we combine two audiences in different proportions. Then we might ask an entirely different survey question. We would like to retain what the model already knows about the population through all of this, without requiring it to undertake a fresh demographic education every time someone changes the subject.

This is the application we have in mind. Encode a population, operate on its representation, then give the resulting vector to a model trained to answer the question we care about. Survey prediction, audience comparison and estimating demand for a service could all use this arrangement: the population supplies a common input, while each predictor learns its own task.

The small example below lets you try the population operations with the actual tiny model running in your browser. We supply two populations of 800 synthetic adults, with different age compositions. Blend them, keep the older adults, then keep the graduates. The model performs these operations on a sixteen-dimensional population vector; its statistical readout estimates how many people remain.

Move the slider and press Run Experiment. Compare Model with Exact, then reverse the order of the restrictions: selecting older graduates should find the same people either way.

For these eight synthetic groups, we can calculate the answers exactly from the supplied counts, and ordinary arithmetic is entitled to question our expenditure. Its role here is to tell us how well the learned operations work. The coloured grids show the population state after each step; that state is also what we can hand to another model for a new prediction task. Whether we can teach those sixteen numbers to support both uses is the research question. The browser gives us a small working instance of the machinery; the experiments that follow examine how much it deserves our confidence.

The Art of the Possible

What would we expect the ideal embedding model for this task to do? Allow us to consensually pump your intuition so we can get you to our preferred conclusions.

Given some description of a demographic, $D$, and our ideal encoder, $E$, we have:

Statistics

As any human reader would, we would expect $z$ to have some statistical introspectability. After all, it is indeed an encoding of a very sophisticated distribution. Consider these:

1. "20-25 year old Californian women" is a relatively large population. Even if you do not know the precise number, you would expect it to be greater than the number of 20 year old Californian women and lesser than the number of Californian women. This is elementary logic.
2. A moderately informed person would further move abductively to conclude that there are probably less of them than there are Londoners, but not much less than there are Irishmen.
3. A regrettable familiarity with California and a first-order encounter with the economy would further induce one to conclude that this population is very likely to be students, renters, unmarried, and presently not very happy with Vaswani et al.

These are things that are variously true by construction, reasonable expectations, and workable assumptions pending verification. This implies that $z$ should encode attributes in metric space required to support such inferences.

Algebra

A population is also a set that accomodates certain operations. We can reasonably conclude that $z$ would admit some Boolean algebra, counting, and some very sophisticated operations that are unique to collections of people.

1. "20-30 year old Californian women" $-$ "20-25 year old Californian women" = "26-30 year old Californian women"
2. A blend of 70% "20-25 year old Californian women" and 30% "26-30 year old Californian women" should support statistical inferences exampled in the previous set while reflecting the blend.
3. Boolean logic should support the construction of sophisticated demographic subsets. "20-25 year old Californian women and 50-60 year old Eritrean women" is a high-variance subset — one half of which is of dubious construction owing to paucity of data.

Lastly, there is the semantic aspect of the matter that text embedding models already do quite well. California and Eritrea are very different places. In our ideal world, we retain that distinction whole.

Our question is precise enough to invite falsification now.

The principal artifact we would hope out of this research is an embedding model that is capable of taking descriptions of demographics and encoding them into a vector space that can support a hypothetical Large Population Model (LPM). That gives us two things to establish: whether the vectors support the computations we ask of them, and whether what they learn is of any use beyond those computations.

A Rather Small Encoder

The first point of comparison must of necessity be with existing embedding models.

Before asking our encoder to support more ambitious operations, we should establish whether its vectors are useful for learning about populations at all. We begin with a small text encoder as a base and compare it with MiniLM.

Some sense of proportion is helpful here. MiniLM is a 22M parameter model that uses deep self-attention. Our small encoder has 251K parameters, roughly one ninetieth as many, and produces a 96-dimensional embedding to MiniLM's 384. There was always reason to suspect that a smaller encoder would do. As the task is narrow in the more semantically rich space of text, it need only be proficient in encoding descriptions of populations. Gemma 2 2B spends almost 1/4^th of its parameters on simply embedding information; but it has to contend with multilinguality, a much larger domain, and the need to support a wide range of tasks.

There are three training regimes from which we can source a model.

1. Task-only training. The model trains only on the downstream task, without prior statistics or algebra training. This tests whether the additional training provides benefits beyond learning directly from task labels.
2. Statistics training. The model is pretrained to support statistical computation, followed by training on the downstream task.
3. Algebra training. The model is pretrained to support statistical computation, population operations, and responses to changed input facts, followed by training on the downstream task.

We compare the statistics- and algebra-trained text encoders with MiniLM at predicting responses to survey questions from CES. For this comparison, the encoders are frozen and an affine readout is trained on each encoder's vectors. This tests their usefulness for population-dependent prediction.

At 128 labelled groups, our statistics-trained encoder scores 7.99pp of mean absolute error against MiniLM's 7.76pp. The shortfall is a mere 0.23pp.

Delightfully, this proximity in performance carries over to changes in label budget. From 64 through 512 labelled groups, we never exceed 0.27pp of error. At 16 labels, both population pretraining conditions have slightly lower point estimates. MiniLM leads from 32 labels onward, but the shortfall remains vanishingly small.

Finally, we can ask what hapens if we allow the small encoder to adapt to the downstream task. Doing so gives a similarly competitive picture. At 128 labels, statistics remains approx. 0.19pp behind a frozen MiniLM on held respondents with known combinations, and 0.28pp behind on held geography. On held combinations, statistics is about 0.18pp ahead. It's worth noting that algebra does not seem to furnish any consistent advantage on ordinary survey responses.

Developing the Population Representation

Of course, having now established that our more specialized encoder fares decently against a much larger model at more superficial tasks, we must set ourselves to the question of precisely what it is that these "improvements" are doing for us.

To answer this question, we invent populations with exactly-known compositions. Each composition has eight groups formed by combining sex, age band, and degree status. A singular example would, to give an illustrative one, say, "blend these two populations, then keep adults aged 50 or older." We further provide numerical facts about the original populations and an operation. It must then represent the resulting population well enough for a small predictor to estimate four quantities:

1. Mean event risk: invented group-specific event probabilities.
2. Risk variance: how much those probabilities differ between people.
3. Profile entropy: how evenly the population is spread across its eight groups.
4. Expected event count: a population-size-dependent total under a second invented event law.

Note that these are entirely contrived mathematical targets, with no measurable in the real world. We freeze our encoder, train a small 676-parameter prediction head using 128 labelled population contexts, then evaluate on 512 held-out ones. The graphic below averages across 3 seeds and three label draws.

Statistics training reduces error by 10.17% for Contextual and 8.66% for Independent, relative to their random-feature controls. Algebra adds a further 6.57% and 13.68% reduction, respectively.

Of course, it should strike you that an embedding that is capable of supporting a single algebraic operation may be too shallow to support multiple algebraic operations. This is, we could even speculate, a limitation that will plague us on account of the nature of floating point independent of how many clever tricks we employ in the construction of the embedding.

It's difficult to test general sequence reasoning for a number of reasons we won't go into here. We can attempt smaller, more developmental diagnostics that give us an idea of the direction but not necessarily the scope of improvement. We again take populations of 8 groups. Each group contains 200, 400, 800 or 1200 people, distributed randomly among those groups. The model receives a population description, eight group proportions and their scaled counts. Our encoder processes it to a 16-dim vector. A learned operator then changes those numbers when instructed to restrict or blend populations. Finally, the previously designed statistical readout predicts population size.

The resulting score is a mean absolute population-count error, measured in people. Operation training substantially reduces error in every class. It also keeps the 5-restriction error close to single-restriction error. However, as can be seen quite readily in the figure, the error is still quite significant. The finding here remains that training our encoder in explicit operations makes these learned operators a lot more usable for repeatedly stacking operations.

Man proposes, gradient descent — that unfeeling golden god — disposes. An apparent capacity to meet our requirement to produce the capabilities we demand of it may simply be a lookup table of exceptional size and trickery.

Here we hold the population facts and operations fixed, then express the same task in two different ways. One uses the construction that is known through training and the other changes the wording.

The wording comparison sneaks in the final H-Algebra model (the construction of which will be discussed soon). Its training includes correct answers under two sets of supplied facts and the difference between those answers. We will discuss what this extra supervision actually buys us.

The two wordings have broadly similar errors for restrictions and alternating operations, with larger differences on direct queries. These are errors against the correct answer for each wording; they do not measure whether the two predictions agree. Nor are all the reserved constructions entirely untouched by earlier development. This is a useful wording diagnostic, with rather less entitlement to the word "invariance" than we might have liked.

If our model was feeling particularly parsimonious, it could theoretically build an internal representation that is ill-formed. It would be task-effective but model its world in such a way that it is incapable of separating populations. In this case, the defect would manifest in a failure to answer questions about population A if facts about population B were to be perverted in some way.

Return now to the earlier encoders. If we ask about population A, changing facts about an unrelated population B should leave the answer alone. Our Contextual encoder receives the description and both reference tables. Our Independent encoder receives the description and only the selected reference's facts. These are the two architectures in the earlier pretraining comparison.

The distinction matters. Under statistics training, changing B moves Contextual's predicted count for A by about 81 people on average. Adding algebra training raises that to about 95.

Independent has zero partner sensitivity because B's facts never enter its computation. This is a rather satisfying problem to solve by declining to give the model an opportunity to have it. It doesn't tell us whether the answer about A is correct. For that, we must ask whether the model uses the information that does belong to A.

Is our model committing a fallacy of division? Two populations could have identical demographic percentages but different combinations of characteristics. The model should ideally be able to distinguish these differences.

The concern here is a related error of inferring intersections from aggregate facts. In a population of 100 people, knowing that 50 are women and 50 are graduates tells us remarkably little about how many are women with degrees. It could be none; it could be all 50. A representation that retains each percentage while losing their association has discarded something necessary for selection.

Our diagnostic is stricter than this example. We change the eight-cell joint distribution while holding the total and all one- and two-way margins fixed. Only the three-way association changes. The model receives the changed facts; we ask whether a new predictor built on its frozen vectors changes its answer by the correct amount. A predictor that ignores the intervention gives us a useful no-change reference.

At 128 labels, all three training regimes are close to that reference, and statistics and algebra have slightly worse point estimates. Ordinary prediction improvements have not bought us a convincing response to joint information. These are post-hoc diagnostics on the earlier cohort, but they identify a concrete weakness worth addressing.

Showing a model two correct facts may not teach the model how a change in the facts connects them. If we teach that explicitly, will that improve performance?

We add a loss on the difference between the answers under paired factual views, alongside teaching each answer separately. The architecture stays fixed. The training now asks the model to account for movement between two populations, as well as for their endpoints.

The one-seed pilot lowers profile-change error in every displayed operation family. Yet two and five restrictions still do worse than predicting no change. Improvement over a previous model and evidence that the new model follows the facts are distinct accomplishments. This pilot gives us reason to retain change supervision, with little reason to declare the matter settled.

In the wild, the model will invariably see contradictory information. This is a feature of reality it must contend with. But is the model itself internally consistent? It is conceivable to have a model that is quite accurate in 80% of evaluation items while the remaining 20% represents a rather dramatic failure to reconcile its own internal representations. These would be trivial errors and inconsistencies in cell counts, total counts, and proportions.

Our synthetic populations spare us contradictory evidence. Any disagreement between their predicted totals, cell counts and proportions is therefore entirely the model's own contribution.

Algebra training substantially reduces the disagreement. But the predicted total still differs from the sum of predicted cell counts by about 46 people on average in both earlier architectures. Separate readouts are being asked to discover a relationship we already know exactly.

Consider the model predicting a population of 100 people, and then giving us subgroup counts that add up to 120. Somewhere, somehow, twenty people were spontaneously ushered into existence. We could prevent this by having the model predict the subgroup counts first, and then arrive at a total.

This leads us to a mass representation. A single bias-free linear decoder maps the population vector to eight cell masses. We obtain the total by summing them and, for a positive total, obtain proportions by division. The statistics now share an arithmetic origin. They can still be wrong, but they cannot independently invent incompatible totals.

We can also design the model so that blending embeddings blends their predicted counts in the same proportions. For a linear count decoder $C$ and blend weight $\alpha$,

Blending becomes a fixed weighted sum of vectors. Restriction remains learned: an instruction selects a linear transformation of the population state. It must learn which mass to retain, but its action on that state is constrained.

In a matched one-seed pilot, linear restrictions reduce two-restriction count error from about 83 to 69 people, and five-restriction error from 135 to 105. Both variants already use the mass decoder and fixed blending. This comparison supports the choice of restriction operator within that design; it does not isolate the benefit of the whole mass architecture against every earlier encoder.

Rather surprisingly, idempotency is difficult for models to learn. In many cases, it has to be jackhammered into the training data so it can be learnt in the first place. Selecting older adults twice should not be any different from selecting them once. Selecting graduates, then older adults should be no different from selecting them in the opposite order.

We test an additional penalty for these laws and for complementary partitions: the masses selected by a condition and its complement should add back to the original. It is an appealing prescription. The model would presumably benefit from being told how to behave.

None of the four comparisons meets the study's support rule. Mixed-operation and joint-response errors worsen on average. Empty-group error improves by about 24%, but one of the three source seeds reverses direction. The explicit law penalty does not earn its place in the final model.

Here and in the later interval plots, support requires both a positive multiple-comparison-adjusted interval and improvement in every source seed. The intervals resample paired synthetic worlds conditional on the fitted models; they do not measure all the uncertainty in retraining a model. An attractive average alone is insufficient.

From Size Scaling to H-Algebra

While text embedding models are content to place their embeddings on a unit sphere, magnitude is a valuable axis of information for us to retain in some capacity. A village and a city may have identical proportions, but their counts should differ by several orders of magnitude.

Giving a model population size as an input does not ensure that it learns the required scaling. When we move to much smaller or larger populations, we may discover that it has learnt a tolerable approximation within its training range and very little arithmetic beyond it.

This brings about a rather trivial intervention. If we expect our vector to represent population mass, doubling a population while preserving composition should actually double the vector.

We ask the original mass model to encode each population at a fixed reference size of 1,200, preserving its composition, then multiply the resulting vector by the actual root size divided by 1,200. We call this inference-scaled algebra. No weights change. Since decoding and the instruction-conditioned restriction operators are linear in the population state, this route makes their outputs scale with root size as well.

The intervention reduces small-size count error by 96.63% and large-size error by 52.75% in the earlier confirmation cohort. Ordinary-size improvement is unsupported. A remarkably large portion of the extrapolation problem was amenable to multiplication.

This repair may be our undoing though. Scaling the vector could perturb relationships that the model needs to interpret demographic facts.

Held-description counts, three distinct restrictions and empty groups improve, but joint-fact response error grows by 16.43%. We have repaired size behaviour while making the model worse at responding to a change in composition. This is an awkward bargain for a population representation.

The model is, unfortunately, trained under one arrangement and then unceremoniously obliged to operate under a different one. In homogeneous training, reference-size encoding and explicit scaling are first-class citizens from initialization. This, combined with statistics, operation and factual-change training, constitutes H-Algebra. The explicit algebraic-law penalty is omitted.

For description $D$, eight joint proportions $p$, root size $N$ and reference size $N_0=1200$, its encoding has the form

The learned function constructs a representation at reference size; the explicit multiplier supplies its mass. During training, the model must now learn composition and operations in the same arrangement it will use at inference.

At familiar sizes, mixed-operation cell-count error falls from 16.43 people per cell under inference scaling to 5.59 with H-Algebra, a 65.97% reduction. This is a comparison of training routes at ordinary sizes. We still need to establish what survives changes in size, composition, wording and operation sequence.

H-Algebra Tiny Architecture

H-Algebra Tiny is the architecture behind Giedi Embedding 0.1-tiny. In the experiments and plot labels, H-Algebra names its fully trained configuration. The resulting mass model has 252,787 parameters and a 16-dimensional state. A small two-layer Transformer encodes the text. A fusion network combines its output with the selected reference population's eight joint proportions and a fixed reference-size input. The explicit size multiplier then produces the population vector.

Restrictions apply instruction-conditioned linear operators to that state; blends take weighted sums. The shared linear mass decoder supplies cell counts, from which totals and proportions follow. The state can also be handed to a new model without the native statistical readout.

There is a considerable qualification to the phrase "embedding model" here. Unlike the text-only encoder compared with MiniLM, H-Algebra receives numerical facts about its reference population. It learns to represent and operate on supplied facts. It does not conjure a demographic census from a sentence. In our eight-cell world, exact table arithmetic is available throughout; it supplies the answers against which we judge the learned representation.

What Survived the Examination

Once we have a candidate model, we should look beyond its most flattering anecdotes. We should:

1. ask direct statistical questions;
2. execute mixed sequences;
3. change the supplied facts;
4. vary population sizes;
5. introduce sparse populations;
6. and use unfamiliar mixture weights.

The final native evaluation uses three trained seeds and 512 synthetic context pairs per cohort. Alternative wordings and programs are views of those same populations, so they do not multiply the number of independent examples. Here native means execution using the source model's own operators and statistical decoder. A new predictor consuming the vector is a separate examination.

Alongside the mixed-operation improvement, held-description total-count error falls from 61.92 to 51.44 people relative to inference scaling. These are different quantities: the first averages errors across eight cells, while the second measures the error in a population total. Their numerical scales should not be mistaken for a ranking of difficulty.

The joint-fact test is particularly encouraging. Relative to inference scaling, H-Algebra reduces profile-change error by about 73% at both familiar and unfamiliar sizes. Training in the homogeneous arrangement avoids the factual-response regression of the inference-only intervention.

At root sizes of 3, 11, 37 and 113, mean cell-count error is about 0.35 people. At sizes of 3,200, 6,400 and 12,800 it is about 62.90. The relative gains over original algebra are large, but a model that scales its predictions also scales its mistakes. Homogeneity secures the size relationship; it does not make an imperfect composition estimate exact.

Sparse populations concentrate mass in fewer cells. Unfamiliar blend weights change how we combine populations. H-Algebra has lower descriptive cell errors than the alternatives under both shifts: about 9.13 and 5.79 people per cell, respectively. These are useful additional views rather than further registered superiority claims.

What happens if we demand answers to empty groups (the 5 year old high schoolers and vegetarian steakhouse critics)? In our smaller experimental world, this means selections with exactly zero mass. Attractive averages can conceal considerable embarrassment for a model merely playing the part.

H-Algebra does not establish improvement over inference scaling on either three distinct restrictions or empty groups. Even the average 38.51% improvement over an always-zero reference on triple restrictions fails the support rule: one seed is worse than that reference. Repeated selection of one condition was an encouraging developmental result. Composing three different conditions asks something harder of the representation.

Twelve of the fifteen registered native comparisons pass. Some compare with deliberately weak zero-output or no-response references, so twelve should not be read as twelve victories over competitive models. The full matrix leaves the failed comparisons visible. We have a substantially better learned population calculator with a discernible boundary around what it reliably executes.

New Tasks and Downstream Models

If our model survives the many trials and tribulations we fling at it in a lab, what precise utility does it serve outside of it? We would furnish a new model with our learned representation and teach it new tasks. Event rates, expected deaths, population entropy, survivors. Here, our three training regimes return as competing sources of representation.

The new targets follow synthetic laws; "deaths" and "survivors" name calculations in that world, rather than validated forecasts about real people. We give identical 752-parameter consumers the population vectors and their log magnitudes, then train them on 128 labelled contexts involving unfamiliar mixed-operation families. The main score averages errors across four targets after standardizing them, so population counts cannot dominate simply by having larger units.

H-Algebra reduces this error relative to H-Statistics by 57.04% with a frozen source and 54.67% with adaptation. The frozen statistics-only improvement over task-only random features does not meet the support rule. In that arm, H-Statistics also has an untrained identity restriction: part of the algebra advantage is having learned to execute the population operations that generate the consumer's input. This is evidence for the usefulness of the trained system under this interface, rather than an isolated claim about every possible use of its vectors.

It's also possible, of course, that if we allow the encoder to change during the exercise, the new task may teach it something that our plans never intended or our means ever did. We should also ask how much of this novel utility is indeed novel as opposed to latent.

We must perform a comparison of our population encoder frozen and not.

At 128 labels, frozen H-Algebra has standardized error 0.2412, against 0.2363 for its fine-tuned counterpart. The latter can adapt 252,256 parameters, compared with the frozen arm's 752 parameters. These separately fitted arms do not establish equivalence, but they do establish the point we needed: a small consumer can use the representation without relearning its encoder.

Even if we do have better arithmetic, it does not follow that we have a better input to another model — which would be an anticlimax of an embedding model.

H-Algebra improves over inference-scaled algebra by 12.34% in the frozen arm and 17.04% in the fine-tuned arm. Superiority over original algebra is unsupported in both. The large native arithmetic improvement has translated into a more modest downstream result, with no established victory over every earlier representation.

And, of course, we would need to know how pretraining scales with our labels.

More labels lower H-Algebra's error, but it is not the best representation at every budget. At 16 labels, inference-scaled algebra has the lowest point estimate in both arms. Original algebra is slightly ahead of H-Algebra at 32. By 128, H-Algebra has the lowest displayed mean. A ranking at one budget is insufficient to choose a representation for every data-poor task.

The individual targets give the average some character. With frozen vectors, H-Algebra's expected death-count error is about 1.88 people, against original algebra's 3.79; four-period survivor error is 15.47, against 28.22. Original algebra has lower event-rate and entropy errors. These descriptive panels average both factual views, whereas the registered aggregate comparison uses the base view. They help explain the tradeoffs without supplying extra primary victories.

What happens if we normalize the vector to unit length and provide an explicit measure of its original magnitude? Does the utility of the embedding decline, or do downstream models find it easier to disentangle?

Both interfaces retain log magnitude. We compare the raw vector plus that scalar with the unit vector plus the same scalar, keeping the consumer's capacity fixed.