Live · refreshes every 20 s · all training runs · the GB10

Coppelius training on the RTX 3090.

A 57.7-million-parameter language model training on a rented RTX 3090 (RunPod) through the Alpha compiler alone: a host executable and sm_86 kernels the compiler lowers from Coppelius's own programs, with no CUDA runtime in the path. Every figure below is read from the run's own records: the result files, the device's semaphore timestamps, the per-launch profile of the profiled build and the samplers on the machine.

Run rtx3090 · polled 2026-09-27 22:30:06 UTC

finished

2,000 of 2,000 steps · 899.4 min since the start

Launch profiles · submission 334 of each profiled build, timed by the device's own semaphores

Where a step's time goes

One training step of each profiled build, every launch bracketed by timestamped releases.

Run	Step	Launches
barrier-guard-step1	27.43 ms	658

barrier-guard-step1: kernel families
Family	ms a step	Share	Launches
GEMM (tiled)	11.05	40.3%	243
attention-key	2.39	8.7%	16
adamw	2.05	7.5%	1
attention-query	1.91	7.0%	16
attention-forward	1.37	5.0%	16
embedding-scatter	1.16	4.2%	1
transpose	1.04	3.8%	64
cross-entropy-backward	0.92	3.3%	1
layernorm-parameters	0.88	3.2%	33
cast-down	0.70	2.6%	66
gated-gelu-backward	0.64	2.3%	16
layernorm-backward	0.62	2.3%	33
layernorm-forward	0.55	2.0%	33
gated-gelu-forward	0.48	1.7%	16
add	0.44	1.6%	32
inverse-rope-qkv	0.34	1.2%	16
value-transpose	0.27	1.0%	16
qk-rope	0.20	0.7%	16
token-to-head	0.15	0.5%	16
cross-entropy	0.15	0.5%	1
copy	0.09	0.3%	4
gather	0.02	0.1%	1
loss-target-gather	0.01	0.0%	1

barrier-guard-step1: tiled products (µs a launch, TFLOP/s)
Phase	Shape	µs	TFLOP/s	× a step
ffn-gate	`tiled-cc-m1408-n512-k1024`	51.7	28.6	32
layer	`tiled-rr-m1024-n1408-k512`	37.0	39.9	32
layer	`tiled-rr-m1024-n1536-k512`	58.0	27.8	16
ffn-down	`tiled-rc-m1024-n1408-k512`	56.2	26.3	16
layer	`tiled-rr-m1024-n512-k1408-add`	53.5	27.6	16
ffn-down	`tiled-cc-m512-n1408-k1024`	52.1	28.4	16
qkv	`tiled-cc-m1536-n512-k1024`	51.7	31.2	16
qkv	`tiled-rc-m1024-n512-k1536`	42.9	37.5	16
ffn-gate	`tiled-rc-m1024-n512-k1408-add`	42.9	34.4	16
ffn-gate	`tiled-rc-m1024-n512-k1408`	39.9	37.0	16
layer	`tiled-rr-m1024-n512-k512-add`	24.8	21.6	16
attention-output	`tiled-cc-m512-n512-k1024`	23.6	22.8	16
attention-output	`tiled-rc-m1024-n512-k512`	21.4	25.1	16
output	`tiled-rr-m1024-n12288-k512`	300.0	43.0	1
output-gradient	`tiled-rc-m1024-n512-k12288`	254.9	50.5	1
output-gradient	`tiled-cc-m12288-n512-k1024`	190.6	67.6	1

Tokens per second40,010end to end, last 20 invocations; 40,010/s during the updates on the GPU

Loss on an unseen context6.545mean of the last 10; the first ten averaged 8.040

Loss after ten steps6.360invocation 199

Best unseen-context loss5.648a fresh model: 9.416

AdamW steps2000of 2,000 in one process

Tokens trained on2,048,0001,024 a step, a fresh context each

Milliseconds a step25.61the latest group of steps

GPU utilization0%last minute's average

Power28.6 Wpeak 29.1 W

Temperature25 °Cmax 25 °C

Prompt probe

What the model says next

A predict invocation of a saved checkpoint on a context whose text ends with the prompt at rows 255, 511 and 1023; each row gives the model's most likely next token after a different stretch of preceding text. Probes run between training runs: the GPU runs one invocation at a time.

No probe yet.

Loss

Cross-entropy at every AdamW step

Mean over the 1,024 rows of each step's context; the band is the 10th–90th percentile across rows. The first step of an invocation is scored on a context the model has not trained on (dots), so its fall is learning rather than memorization. Dashed: ln 12,288 = 9.416, a model that knows nothing.

every stepp10–p90 across rowsfirst step (unseen context)unseen, 20-invocation average

Fitting the context

Loss fall from step 1 to step 10 within each invocation.

Per-row spread at the last step

Minimum, median and maximum per-row loss.

minmedianmax

Sampled rows before training

Cross-entropy at rows 255, 511 and 1023 on the state each invocation starts from, and row 1023 after its ten steps.

25551110231023 after

Prediction entropy

Row 1023's next-token entropy before and after each invocation, nats; ln 12,288 when every token is equally likely.

beforeafter

Invocation inspector

One invocation, every field

Invocation

Loading invocation 199…

Device timing

On the GPU's own clock

Every step's time comes from its submission's two semaphore releases, stamped by the RTX 3090's own clock; the host records them once the run ends, so while it runs the figures are the polls' wall clock. Each point is a group of 10 steps.

Device time per invocation, seconds

ten updatesforward passescheckpoint load and gatherwall clock

Tokens per second

Ten steps on 1,024 tokens per invocation: over the wall clock (checkpoint in and out included), and over the ten update submissions on the GPU.

during the updates, GPU clockend to end, wall clock

One AdamW update, milliseconds

Mean of the ten update submissions; band: their range.

Submission timeline, invocation 199

0 submissions; each bar is one submission's device time in milliseconds, log scale.

Host doorbell to wake-up, milliseconds

Per invocation, over the host-stamped submissions.

medianmax

Optimizer internals

The token embedding, read back from the device

L2 norms of the first 1,024 scalars of the embedding's last-step gradient, its parameters and AdamW's first moment after each invocation, and a count of non-finite values in those windows.

Gradient

Parameters

AdamW first moment

Non-finite values

GPU telemetry

nvidia-smi, per minute

Sampled every second; line: the minute's average, band: its range. The RTX 3090 has 24 GB of its own memory; the arena lives there.

Utilization, %

Power, W

Temperature, °C

SM clock, MHz

Memory-controller utilization, %

Performance state and clock events

P-state (0 is maximum performance) and whether any clock-event reason (throttling) was active.

P-stateclock event active

Host telemetry

The pod's x86-64 CPU running the host

Load average, 1 min

Memory available, GB

Host process resident memory, MB

Host process CPU, %

Channel health

Every invocation's exit, notifier and numerics

One cell per planned invocation: exit 0 with a clear error notifier, an all-zero host status block and a finite loss; a failure or a non-finite loss; pending.

Checkpoint lineage

Every invocation, newest first

Each invocation resumes the checkpoint the one before wrote. Predictions are the argmax at rows 255, 511 and 1023 on the state the invocation started from; truth is the real next token.

k	Steps	Unseen loss	Step 10	Wall s	Device s	Checkpoint SHA-256
199	1991–2000	6.832	6.360	0.26	0.26	`—…`
198	1981–1990	6.251	6.730	0.26	0.26	`—…`
197	1971–1980	6.359	6.304	0.26	0.26	`—…`
196	1961–1970	6.497	6.371	0.26	0.26	`—…`
195	1951–1960	6.250	6.162	0.26	0.26	`—…`
194	1941–1950	6.538	6.585	0.26	0.26	`—…`
193	1931–1940	7.059	6.480	0.25	0.25	`—…`
192	1921–1930	6.328	6.355	0.26	0.26	`—…`
191	1911–1920	6.625	6.131	0.26	0.26	`—…`
190	1901–1910	6.711	6.755	0.26	0.26	`—…`
189	1891–1900	6.140	6.224	0.26	0.26	`—…`
188	1881–1890	6.289	6.355	0.26	0.26	`—…`
187	1871–1880	6.449	6.456	0.25	0.25	`—…`
186	1861–1870	6.492	6.418	0.26	0.26	`—…`
185	1851–1860	6.377	6.351	0.26	0.26	`—…`
184	1841–1850	6.570	6.511	0.25	0.25	`—…`
183	1831–1840	6.726	6.301	0.25	0.25	`—…`
182	1821–1830	6.455	6.120	0.25	0.25	`—…`
181	1811–1820	6.704	6.789	0.25	0.25	`—…`
180	1801–1810	6.635	6.410	0.26	0.26	`—…`
179	1791–1800	6.431	5.891	0.26	0.26	`—…`
178	1781–1790	6.439	6.376	0.26	0.26	`—…`
177	1771–1780	6.252	6.861	0.25	0.25	`—…`
176	1761–1770	6.153	6.347	0.25	0.25	`—…`
175	1751–1760	6.048	6.310	0.25	0.25	`—…`
174	1741–1750	5.648	6.415	0.26	0.26	`—…`
173	1731–1740	6.535	5.876	0.26	0.26	`—…`
172	1721–1730	6.347	6.398	0.26	0.26	`—…`
171	1711–1720	6.752	6.398	0.26	0.26	`—…`
170	1701–1710	6.388	6.498	0.25	0.25	`—…`
169	1691–1700	6.678	6.510	0.25	0.25	`—…`
168	1681–1690	6.353	6.650	0.25	0.25	`—…`
167	1671–1680	6.391	6.436	0.25	0.25	`—…`
166	1661–1670	6.604	6.439	0.26	0.26	`—…`
165	1651–1660	6.564	6.443	0.26	0.26	`—…`
164	1641–1650	6.773	6.425	0.26	0.26	`—…`
163	1631–1640	6.478	6.366	0.25	0.25	`—…`
162	1621–1630	7.489	6.347	0.26	0.26	`—…`
161	1611–1620	6.300	6.715	0.26	0.26	`—…`
160	1601–1610	6.548	6.510	0.26	0.26	`—…`
159	1591–1600	6.819	6.217	0.26	0.26	`—…`
158	1581–1590	6.733	6.558	0.26	0.26	`—…`
157	1571–1580	6.646	6.815	0.26	0.26	`—…`
156	1561–1570	6.983	6.411	0.26	0.26	`—…`
155	1551–1560	6.489	6.802	0.26	0.26	`—…`
154	1541–1550	6.586	6.489	0.26	0.26	`—…`
153	1531–1540	6.435	6.944	0.26	0.26	`—…`
152	1521–1530	7.178	6.175	0.26	0.26	`—…`
151	1511–1520	6.488	7.174	0.26	0.26	`—…`
150	1501–1510	6.378	6.515	0.26	0.26	`—…`
149	1491–1500	6.248	6.429	0.26	0.26	`—…`
148	1481–1490	6.436	6.568	0.26	0.26	`—…`
147	1471–1480	6.466	6.547	0.26	0.26	`—…`
146	1461–1470	5.889	6.129	0.26	0.26	`—…`
145	1451–1460	6.553	6.366	0.26	0.26	`—…`
144	1441–1450	6.699	6.432	0.26	0.26	`—…`
143	1431–1440	6.538	6.732	0.26	0.26	`—…`
142	1421–1430	6.438	6.430	0.26	0.26	`—…`
141	1411–1420	6.557	6.365	0.26	0.26	`—…`
140	1401–1410	6.381	6.761	0.26	0.26	`—…`
139	1391–1400	6.520	6.191	0.26	0.26	`—…`
138	1381–1390	6.674	6.472	0.26	0.26	`—…`
137	1371–1380	6.045	6.708	0.26	0.26	`—…`
136	1361–1370	6.484	5.767	0.26	0.26	`—…`
135	1351–1360	6.612	7.243	0.26	0.26	`—…`
134	1341–1350	6.450	6.557	0.26	0.26	`—…`
133	1331–1340	6.741	6.661	0.26	0.26	`—…`
132	1321–1330	6.584	6.573	0.26	0.26	`—…`
131	1311–1320	6.363	6.606	0.26	0.26	`—…`
130	1301–1310	6.689	6.593	0.26	0.26	`—…`
129	1291–1300	6.464	6.600	0.26	0.26	`—…`
128	1281–1290	6.942	6.657	0.26	0.26	`—…`
127	1271–1280	6.424	7.139	0.26	0.26	`—…`
126	1261–1270	6.754	6.944	0.26	0.26	`—…`
125	1251–1260	6.400	6.440	0.26	0.26	`—…`
124	1241–1250	6.383	6.363	0.26	0.26	`—…`
123	1231–1240	6.461	6.552	0.26	0.26	`—…`
122	1221–1230	6.734	6.612	0.26	0.26	`—…`
121	1211–1220	6.479	6.646	0.26	0.26	`—…`
120	1201–1210	6.847	6.343	0.26	0.26	`—…`
119	1191–1200	6.658	6.762	0.26	0.26	`—…`
118	1181–1190	6.773	6.637	0.26	0.26	`—…`
117	1171–1180	6.681	6.911	0.26	0.26	`—…`
116	1161–1170	6.621	6.875	0.26	0.26	`—…`
115	1151–1160	7.038	6.463	0.26	0.26	`—…`
114	1141–1150	6.456	7.005	0.26	0.26	`—…`
113	1131–1140	6.807	6.533	0.26	0.26	`—…`
112	1121–1130	6.826	6.694	0.26	0.26	`—…`
111	1111–1120	6.720	6.515	0.26	0.26	`—…`
110	1101–1110	6.966	6.310	0.26	0.26	`—…`
109	1091–1100	6.659	6.790	0.25	0.25	`—…`
108	1081–1090	6.786	6.704	0.26	0.26	`—…`
107	1071–1080	6.726	6.943	0.26	0.26	`—…`
106	1061–1070	6.659	7.018	0.26	0.26	`—…`
105	1051–1060	6.676	6.902	0.25	0.25	`—…`
104	1041–1050	6.772	6.882	0.25	0.25	`—…`
103	1031–1040	6.963	6.790	0.25	0.25	`—…`
102	1021–1030	6.691	6.705	0.26	0.26	`—…`
101	1011–1020	6.825	7.007	0.26	0.26	`—…`
100	1001–1010	6.772	6.904	0.26	0.26	`—…`
99	991–1000	6.588	6.622	0.26	0.26	`—…`
98	981–990	6.424	6.517	0.26	0.26	`—…`
97	971–980	7.040	6.935	0.26	0.26	`—…`
96	961–970	6.554	6.598	0.26	0.26	`—…`
95	951–960	6.932	6.362	0.26	0.26	`—…`
94	941–950	6.491	6.828	0.26	0.26	`—…`
93	931–940	6.824	6.405	0.26	0.26	`—…`
92	921–930	6.701	6.629	0.26	0.26	`—…`
91	911–920	6.818	6.584	0.26	0.26	`—…`
90	901–910	6.850	6.776	0.26	0.26	`—…`
89	891–900	6.931	6.808	0.26	0.26	`—…`
88	881–890	6.714	6.883	0.26	0.26	`—…`
87	871–880	7.104	6.773	0.26	0.26	`—…`
86	861–870	6.676	6.914	0.25	0.25	`—…`
85	851–860	6.933	7.094	0.26	0.26	`—…`
84	841–850	6.936	6.973	0.26	0.26	`—…`
83	831–840	6.813	6.814	0.26	0.26	`—…`
82	821–830	7.129	6.854	0.26	0.26	`—…`
81	811–820	7.054	7.047	0.26	0.26	`—…`
80	801–810	6.963	6.798	0.26	0.26	`—…`
79	791–800	6.530	6.902	0.26	0.26	`—…`
78	781–790	7.011	7.006	0.26	0.26	`—…`
77	771–780	6.962	6.911	0.26	0.26	`—…`
76	761–770	7.110	6.698	0.26	0.26	`—…`
75	751–760	7.398	7.025	0.26	0.26	`—…`
74	741–750	6.927	7.062	0.26	0.26	`—…`
73	731–740	6.629	6.841	0.26	0.26	`—…`
72	721–730	6.767	6.980	0.26	0.26	`—…`
71	711–720	6.952	7.039	0.26	0.26	`—…`
70	701–710	6.983	6.773	0.26	0.26	`—…`
69	691–700	7.070	6.998	0.26	0.26	`—…`
68	681–690	6.765	6.892	0.26	0.26	`—…`
67	671–680	6.878	6.814	0.25	0.25	`—…`
66	661–670	7.271	6.900	0.26	0.26	`—…`
65	651–660	7.393	7.081	0.26	0.26	`—…`
64	641–650	7.426	7.258	0.26	0.26	`—…`
63	631–640	6.926	7.009	0.26	0.26	`—…`
62	621–630	7.000	6.838	0.26	0.26	`—…`
61	611–620	6.891	7.261	0.26	0.26	`—…`
60	601–610	7.121	7.082	0.26	0.26	`—…`
59	591–600	7.018	7.982	0.26	0.26	`—…`
58	581–590	7.250	7.087	0.26	0.26	`—…`
57	571–580	6.979	7.138	0.26	0.26	`—…`
56	561–570	7.217	6.966	0.26	0.26	`—…`
55	551–560	7.097	7.042	0.26	0.26	`—…`
54	541–550	7.249	7.200	0.26	0.26	`—…`
53	531–540	7.057	7.034	0.26	0.26	`—…`
52	521–530	7.228	6.887	0.26	0.26	`—…`
51	511–520	7.144	7.348	0.26	0.26	`—…`
50	501–510	6.911	6.908	0.26	0.26	`—…`
49	491–500	6.940	6.788	0.26	0.26	`—…`
48	481–490	7.084	6.928	0.26	0.26	`—…`
47	471–480	6.982	7.251	0.26	0.26	`—…`
46	461–470	7.604	7.203	0.26	0.26	`—…`
45	451–460	7.261	7.223	0.27	0.27	`—…`
44	441–450	7.154	7.098	0.26	0.26	`—…`
43	431–440	7.377	7.111	0.26	0.26	`—…`
42	421–430	7.554	7.321	0.26	0.26	`—…`
41	411–420	7.479	7.250	0.26	0.26	`—…`
40	401–410	7.322	7.501	0.26	0.26	`—…`
39	391–400	7.298	7.680	0.26	0.26	`—…`
38	381–390	7.182	7.330	0.26	0.26	`—…`
37	371–380	7.368	7.194	0.26	0.26	`—…`
36	361–370	7.206	7.531	0.26	0.26	`—…`
35	351–360	7.397	7.266	0.26	0.26	`—…`
34	341–350	7.462	7.080	0.26	0.26	`—…`
33	331–340	7.222	7.835	0.26	0.26	`—…`
32	321–330	7.439	7.208	0.26	0.26	`—…`
31	311–320	7.490	7.242	0.26	0.26	`—…`
30	301–310	7.336	7.275	0.26	0.26	`—…`
29	291–300	7.409	7.184	0.26	0.26	`—…`
28	281–290	7.339	7.378	0.26	0.26	`—…`
27	271–280	7.426	7.392	0.26	0.26	`—…`
26	261–270	7.309	7.547	0.26	0.26	`—…`
25	251–260	7.163	7.520	0.26	0.26	`—…`
24	241–250	7.534	7.700	0.26	0.26	`—…`
23	231–240	7.470	7.518	0.26	0.26	`—…`
22	221–230	7.318	7.453	0.26	0.26	`—…`
21	211–220	7.770	7.464	0.26	0.26	`—…`
20	201–210	6.825	7.562	0.26	0.26	`—…`
19	191–200	7.139	6.823	0.26	0.26	`—…`
18	181–190	7.425	7.159	0.26	0.26	`—…`
17	171–180	7.638	7.471	0.26	0.26	`—…`
16	161–170	7.521	7.612	0.26	0.26	`—…`
15	151–160	7.348	7.769	0.26	0.26	`—…`
14	141–150	7.569	7.390	0.26	0.26	`—…`
13	131–140	7.472	7.379	0.26	0.26	`—…`
12	121–130	7.438	7.528	0.26	0.26	`—…`
11	111–120	7.277	7.476	0.26	0.26	`—…`
10	101–110	7.534	7.319	0.26	0.26	`—…`
9	91–100	7.383	7.357	0.26	0.26	`—…`
8	81–90	7.410	7.565	0.26	0.26	`—…`
7	71–80	7.718	7.624	0.26	0.26	`—…`
6	61–70	7.830	7.798	0.26	0.26	`—…`
5	51–60	7.842	7.680	0.26	0.26	`—…`
4	41–50	7.846	7.929	0.26	0.26	`—…`
3	31–40	8.031	7.861	0.26	0.26	`—…`
2	21–30	8.358	8.003	0.26	0.26	`—…`
1	11–20	8.628	8.530	0.26	0.26	`—…`
0	1–10	9.349	8.590	0.26	0.26	`—…`

Run

What is running

machine: NVIDIA GeForce RTX 3090
driver: 580.126.20
compute: 8.6
executable: rtx3090-barrier-guard-2k-from-step1.elf
model: Coppelius: 16 blocks, width 512, 8 heads, context 1024, FFN 1408, vocab 12288, 57.7M parameters
optimizer: AdamW, one step per context, 2000 steps in one process, checkpoint every 1000
steps done: 2000
started: 2026-09-27 07:30:40 UTC
last GPU sample: 2026/09/27 22:29:22.541, 0 %, 2