Repository navigation
feat: taraengine as taracode's native backend - #27
Merged
Merged
Conversation
The fake's default tier recommendations are literal data again. Reading them from internal/models would make the fake import internal/llm, and tests in the models package import the fake, which would form an import cycle. The model-name guard therefore skips the fake's directory, which mirrors the engine's catalog.
…the recommendation
…y what Ollama cannot do
…board runs the catalog
A real server gives every tool call an id of its own, so a conversation that replays several replies never holds two calls with one id. The fake numbered the calls of each reply from call_0; it now runs the numbers on for the life of the server (call_0, call_1, call_2, ...), and a test pins it.
…e's calls The detection probes give up dialling after 1.5 s, below the engine probe's 2 s, so a host that drops the packets (a powered-off box, a VPN that is down, a firewall that drops) stops detection with the not-running error, as a refused port does. net/http dials apart from the request's context, so only a dial bound below the caller's deadline shows that nothing answered. With vendor taraengine the engine client's dial gives up after 3 s, inside the session's 5 s bound on reading a model, so the session stops with "nothing answers at <host>" and prints no capability warning. Only the dial is bounded: a load keeps its 15-minute backstop. The probe test's comment says what an idle connection is: one nothing will close.
…dvice At startup, an engine with nothing pulled and a model without tool calls get the suggestion table's advice (pull a model, `taracode doctor` names one; choose a model with tools, `taracode doctor` lists them) instead of the box that says taracode cannot connect. A connection failure, and an error nothing explains, still get that box. A refusal the server explained shows its hint once: the dimmed line under the title carries the message without the hint, and the hint is the first suggestion. The same holds for a server nothing answers at. A session on Ollama says how to get a larger window there: set OLLAMA_CONTEXT_LENGTH on the Ollama server. The changelog's migration note says the same.
…s unknown The engine infers an import's capabilities while the import is loaded, lists none before that and forgets them on every unload. Show reports such capabilities unknown (inferred, and none listed), and the session takes them as it takes a server without capability data: no tools gate, thinking left as configured, the window capped by the native one, and no warning. An import can be started, and switched back to with /model, and the evals' check takes it too. A catalog model without tools is still refused, and an import whose capabilities are listed counts as any other model. The fake engine marks an import's capabilities as inferred.
…g one Before a load on the engine, which can take up to the engine's load_timeout, the session start and /model print "Loading <model> at a <n>-token window..." in the dim style, to the session's output like its other notes. The generic path loads nothing and prints nothing. When the configured model is not pulled and the session falls back to the first pulled one, a second line on the engine says "Pull it with: taraengine pull <model>". Other servers get no such line. The warning about a retired hosts: section says where taracode goes instead: the host in host:, or taraengine at http://127.0.0.1:8272 when none is set.
…loaded one A server that answered and refused (a wrong key, a 500) reads "refused: <message>", with the key line for a 401; "unreachable:" stays for a server nothing answers at. The OpenAI model list has names only, so a model whose size and window are both unknown is listed by its name alone instead of "0.0 GB ctx 0". The engine lists a sleeping model among the loaded ones, with its window: when the configured model sleeps, the Model line says "(sleeping at a <n>-token window)".
…rol characters off the terminal An error chunk the engine sends mid-answer is a *llm.ServerError (status 500, its message and hint, the status text when it sent no message), so it gets the same rendering as a refusal before the answer. A stream line longer than the 8 MiB bound is its own error, "taraengine: a line of the stream is longer than 8 MiB", not a server that stopped mid-answer. Text the engine supplies reaches the terminal without its control characters (tab and newline kept): refusal messages and hints on /v1 and /api, the error chunk, and doctor's engine report (version, runtime, platform, health detail, machine, runtime note, recommendation and a faulty model's log tail, which echoes an imported model's metadata). A refusal body that is not the engine's JSON, a proxy's error page say, is cut to 300 bytes. "nothing answers at <host>" masks the password of a host URL's user info. The first 503's body is read to its end before the retry, so the retry reuses the connection even when the body comes after the status.
…or and imports scripts/engine-ci.sh selects ^TestAgainstARealEngine$ and fails unless the output holds that test's PASS line: go test passes when no test matches, so a renamed test or a changed build tag would otherwise run nothing. Its header says the port is fixed at 18272. The real-engine test runs as steps and stops at the first that fails. It adds a health call without a key, which detection relies on, a doctor call whose machine names a tier and whose recommendations are not empty, and a Show of the imported model before its first load, whose capabilities are unknown. A test of CleanHardware uses a documentation address (192.0.2.10), a comment on the engine's URL forms uses the engine's port, and a test comment says what its 5 s bound is for.
CONTRIBUTING installs taraengine with go install, installs its runtime, starts it and pulls a model, the way scripts/engine-ci.sh does, until the release brings the cask. The generic path is described exactly: no window control, no thinking on or off, no evals. The Makefile's lab-smoke names TARACODE_KEY, as eval-lab does, and the scoreboard comment says that on the engine a model that does not fit fails to load instead of spilling to the CPU, so the board stops at it; the eight catalog models fit a 32 GB card. SECURITY no longer lists taraengine among the third-party servers out of scope; its reports go to its own policy.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
…ssword The engine lists no capabilities for an import it has not loaded and marks them inferred; the session starts such a model. Doctor now says so too: the Model line reads "(capabilities show once it loads)" and the capabilities column "unknown until loaded", where it said the model has no tool calls. A model the engine marks inferred but lists capabilities for shows them as any other. Doctor's exit status and its other lines are as they were. llm.RedactedHost, the masking "nothing answers at <host>" already used, is exported. Doctor's Server line and the connection box's Host and curl lines print the host through it, so the password of a user:password in the host URL never reaches the screen; a host without user info prints as it is.
… length is not waited for The message of a refusal on the control API is cut to 300 bytes, at a rune boundary, as the body of a /v1 refusal that is not the engine's JSON is. pkg/client passes on up to a mebibyte of a body as the message, a proxy's whole error page. One helper does the cut for both; a message that fits is unchanged. The first 503's body is read to its end before the retry only when the response declares a length of at most 64 KiB, as the engine's 503s do, so the retry still reuses the connection. A body of unknown length that may never end is closed at once, and the retry goes out after its delay instead of when the turn's context ends.
…drops control characters When model detection finds nothing answering at the engine (an *llm.UnreachableError, which only the engine client returns), New returns that error at once: before the saved and the configured model are trusted and without the "Could not detect models" line. The session used to go on with the configured name and stop at its next call to the engine, a dial's wait later. Any other detection error keeps the fallbacks. The first-run advice prints the tier and the model's name, which come from the engine's doctor, without their control characters; a recommendation of nothing but control characters is no recommendation.
The /status Host line, the config warning that names the host of a retired hosts: section, and the error evals give for a server that is not an engine print the host through llm.RedactedHost, so the password of a user:password in the host URL is masked there as it is in doctor and the connection box. The session and the evals keep the host as configured; a host without user info prints as it is.
Merged
6 tasks done
# Conflicts: # CHANGELOG.md
# Conflicts: # CHANGELOG.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
taraengine becomes taracode's native backend. taracode finds it at
http://127.0.0.1:8272when no host is set,loads the model at the resolved context window, maps
/thinkto the engine's thinking switch and levels, and readsthe engine's own timings for tokens per second. The native Ollama client goes: Ollama keeps working on the generic
OpenAI-compatible path, like vLLM and llama.cpp, and each Ollama session says in one line what that path cannot do.
This PR merges after the board proof, with
go.modon taraengine v0.2.0. The user-facing files (README.md,install.sh, the.goreleaser.yamltexts,docs/evals/and the regenerated board) follow in the release PR withtaraengine v0.3.0, because code.tara.vision serves
install.shand the/evalsboard frommain.Changes by area:
internal/llm/taraengine, the engine client over taraengine'spkg/client: one retry after 2 s ona 503, loads with a 15-minute backstop, chat over
/v1with temperature, top_p, max_tokens, the thinking switchand levels, usage and timings; stream errors arrive as the engine's refusals, and engine text reaches the terminal
without control characters. Typed errors:
ServerError,UnreachableError,ErrStreamCut./api/healthfirst (2 s); Ollama by/api/version; a host that refuses or never answers theconnection stops detection with a clear error;
vendor: taraengineskips detection; the default host ishttp://127.0.0.1:8272./modelload the model at the resolved window and say so; a failed switchrestores the previous model and its settings; first-run and not-pulled advice name
taraengine pull <model>; animported model starts before the engine has loaded it once.
tools, no connection).
configured model loads at, the recommended model, and a warning below engine 0.2.0; a refusal reads as a refusal,
not as an unreachable server.
back byte for byte; the board keeps a row per model and engine; a memory figure the engine measured replaces the
probe.
per-request window, keep_alive and format options. Existing configs still load; doctor notes a
keep_alivethatno longer does anything.
internal/llm/taraenginetest).Enginejob runs a tagged integration test against a real taraengine built at thego.modpin(
scripts/engine-ci.sh): health with and without the key, version, models, doctor, load, a streamed chat withtimings, unload, a not-pulled refusal, and the key on
/apiand/v1.config.example.yaml, the issue and PR templates, and theCHANGELOG's
[Unreleased]section with the migration notes.Related Issue
None.
Type of Change
to the generic OpenAI-compatible path; see the CHANGELOG's Migration section
How Has This Been Tested?
Describe the tests you ran:
make eval-labwith gemma4:12b against a taraengine 0.1.0 lab engine wrote aresults file with the
serverblock, tier 16, the default mark,gpu_percent100 andgpu_memory_mib; a sessionon glm-4.7-flash called a tool and answered from its result;
taracode doctorread the engine, the machine and themodels
-race, the coverage gate, golangci-lint and govulncheckscripts/engine-ci.shpasses against taraengine v0.1.0 in a linux/arm64 container; theEnginejob's amd64 run is this PR's CIChecklist
gofmt -s -l .shows no files (code is formatted)go vet ./...passesmake testpassesmake buildsucceeds