# MechLens SDK > Record comparable mechanistic-interpretability experiments from your existing Python code. The MechLens SDK (`pip install mechlens`) records a baseline and an intervention run from existing Python experiment code and uploads them to MechLens Cloud, which checks that the runs are comparable and helps publish a finding. Record the same fields in both runs, use the same project and stable example IDs, and change only the intervention. # Getting started # MechLens SDK The MechLens SDK records model experiments from the Python code you already have and sends them to MechLens Cloud, where you compare a reference run against interventions and controls, review the evidence, and publish a finding. Your code still loads the model, runs the intervention, and calculates the metrics. The SDK adds only a few calls: ``` import mechlens run = mechlens.start_run( project="refusal-steering", name="baseline", model={"id": "google/gemma-2-2b-it", "revision": "main"}, question="Does steering this feature reduce refusals?", config={"seed": 42}, ) run.log_example(id="lockout-001", input=prompt, output=answer, refused=False) run.log_metrics({"refusal_rate": 0.25}) print(run.finish().url) ``` The SDK depends only on `httpx` and `python-dotenv`. It does not need Torch, TransformerLens, or any SAE library. ## How it fits together 1. **Record the conditions** over the same examples: a reference, and the interventions and controls compared with it. 1. **Cloud compares each with the reference** field by field: model, dataset, seed, config, metric definitions, and which examples match. 1. **You review the evidence**: metric changes with their intervals, beside what you expected, and the examples whose behavior changed. 1. **You write a finding**, with a conclusion and its limitations, and publish it as a page anyone with the link can view. ## Where to go next - New here? Start with the [Quickstart](https://blaketylerfullerton.github.io/mechlens-sdk/quickstart/index.md). - Getting a warning on the comparison page? Read [Recording a comparable pair](https://blaketylerfullerton.github.io/mechlens-sdk/comparable-runs/index.md). - Wondering what "unknown" or "differs" means? See [What the checks mean](https://blaketylerfullerton.github.io/mechlens-sdk/integrity-checks/index.md). - Looking up an argument? See the [API reference](https://blaketylerfullerton.github.io/mechlens-sdk/reference/index.md). # Quickstart This walkthrough takes you from an empty environment to a reference run, an intervention and a control side by side in MechLens Cloud. ## 1. Install the SDK ``` pip install mechlens ``` The SDK needs Python 3.10 or later. ## 2. Connect to your workspace In MechLens Cloud, open **Experiments → New experiment** and click **Generate API key**, or create one under **Account → API keys**. The full key is shown only once. ``` export MECHLENS_API_KEY="ml_your_key_here" ``` Runs upload to MechLens Cloud. To use another deployment, such as a local Cloud during development, also set `MECHLENS_API_URL`: ``` export MECHLENS_API_URL="http://localhost:5175" ``` Warning Anyone with the key can upload runs to your workspace. Keep it out of notebooks and commits. Revoke it from **Account → API keys** if it leaks. ## 3. Declare the experiment Describe the experiment once. Everything shared by the conditions goes here, so they are comparable by construction. Here `generate`, `is_refusal` and `prompts` stand in for your own code and data. ``` import mechlens exp = mechlens.experiment( question="Does negatively steering SAE feature 4521 reduce refusals on benign prompts?", claim="Steering feature 4521 at -4.0 lowers the refusal rate on borderline-benign prompts; a random feature does not.", deviations=["Refusals scored by substring matching, not human review"], model={"id": "google/gemma-2-2b-it", "revision": "main"}, dataset={"id": "borderline-benign-prompts", "revision": "v1", "split": "test"}, sae={ "release": "gemma-scope-2b-pt-res-canonical", "sae_id": "layer_12/width_16k/canonical", "hook_point": "blocks.12.hook_resid_post", }, generation={"max_new_tokens": 128, "temperature": 0.0, "do_sample": False}, metric_definitions={ "refusal_rate": { "description": "Share of prompts whose response declines to help.", "example_field": "refused", }, }, config={"seed": 42}, ) ``` `deviations` lists every way the experiment knowingly falls short of the ideal protocol. Each one becomes a candidate limitation when you write the finding. `example_field` says which per-example field a metric averages. Cloud uses it to put a confidence interval on every difference in that metric. Without it, you get the difference alone; `finish()` warns when a metric looks like the mean of an example field you haven't linked. ## 4. Record the conditions Run the same examples, with the same IDs, through every condition. The reference changes nothing; every other condition describes what it changes. - An **intervention** changes the model in the way your claim is about. - A **control** is one your claim predicts should *not* have the effect, such as steering a random feature instead. Declare what you expect each condition to do, relative to the reference, before you see the results. If the claim depends on a comparison between two non-reference conditions, such as the intervention against its control, declare that too with `compare_to`. ``` def record(run, answer_fn): refusals = 0 for example in prompts: answer = answer_fn(example["prompt"]) refused = is_refusal(answer) refusals += refused run.log_example(id=example["id"], input=example["prompt"], output=answer, refused=refused) run.log_metrics({"refusal_rate": refusals / len(prompts)}) with exp.reference() as run: record(run, generate) with exp.intervention( "steer-4521", intervention={"type": "sae_feature_steering", "feature_id": 4521, "steering_strength": -4.0}, expect={"refusal_rate": "decrease"}, compare_to=["steer-random-feature"], ) as run: record(run, lambda prompt: generate_with_steering(prompt, feature_id=4521, strength=-4.0)) with exp.control( "steer-random-feature", intervention={"type": "sae_feature_steering", "feature_id": 9113, "steering_strength": -4.0}, expect={"refusal_rate": "no_effect"}, ) as run: record(run, lambda prompt: generate_with_steering(prompt, feature_id=9113, strength=-4.0)) print(exp.url) ``` Leaving each `with` block saves the run locally and uploads it. The project and experiment are chosen for you and printed when the first run starts; pass `project=` and `id=` to `experiment()` to choose your own. ### Sweeps For a dose-response, record one condition per level of the same `series`. The name defaults to `@`, and Cloud plots the series as a curve: ``` for strength in (-1.0, -2.0, -4.0, -8.0): with exp.intervention( series="steering_strength", level=strength, intervention={"type": "sae_feature_steering", "feature_id": 4521, "steering_strength": strength}, expect={"refusal_rate": "decrease"}, ) as run: record(run, lambda prompt: generate_with_steering(prompt, feature_id=4521, strength=strength)) ``` ### Conditions in separate jobs Conditions don't have to run in one process. Run the same declaration in each job, a job array or a restarted notebook, and record only that job's condition; they all join one experiment, because its ID is derived from the project, question and claim. Changing the claim starts a new experiment. Recording a condition name again counts as a rerun: Cloud uses the latest run and lists the earlier ones as superseded, so reruns stay visible. ## 5. Review the draft and write the finding Open `exp.url`. Cloud compares every condition with the reference, and shows: - whether each condition evaluated the same model, dataset, and examples as the reference - each metric's value per condition, its difference from the reference with a confidence interval, and your declared expectation beside it - the examples whose recorded behavior changed, side by side - a field-by-field table of [comparison checks](https://blaketylerfullerton.github.io/mechlens-sdk/integrity-checks/index.md) - a **draft finding**: your claim, the protocol, and candidate limitations from your declared deviations and from every check Cloud couldn't settle Add the limitations that apply, write the conclusion in your own words, and publish when you are ready to share. Cloud never writes the conclusion for you. ### Reproducing a paper Add the paper and the values it reports. Use the metric names you log and the condition names you record, so Cloud can show the paper's numbers beside yours: ``` exp = mechlens.experiment( question="Does ablating a single refusal direction remove refusals?", claim="Ablating one difference-in-means direction drops refusal from 0.95 to 0.01.", reproduces=mechlens.Paper( arxiv="2406.11717", version="v3", reported={"refusal_rate": {"reference": 0.95, "ablate-refusal-direction": 0.01}}, where="Table 10", ), deviations=["Llama-3.1 8B Instruct instead of Llama-3 8B Instruct"], model=..., ) ``` Cloud creates the reproduction for you, fills in its protocol from the runs, and adds a paper-vs-observed table to the draft. Deciding whether the result reproduced is still your call. ### Recording runs one at a time `mechlens.start_run()` records a single run with every argument given explicitly. It still works, and Cloud compares two such runs the same way, but it can't draft a finding because nothing declares the claim or the conditions. ## Try it without a model The repository includes `examples/steering_demo.py`. It uploads a synthetic baseline and steered run so you can check your key and connection before wiring in real code: ``` python examples/steering_demo.py ``` Its outputs are hand-written and marked `[synthetic]`. ## If an upload fails `finish()` writes the run to `.mechlens/runs/.json` **before** uploading, so nothing is lost. A failed upload raises MechLensUploadError, whose message includes Cloud's explanation and what to do next. - If `error.retryable` is true (Cloud was unreachable or had a server error), retry in the same process with `run.retry_upload()`. - If it is false, retrying the same run won't help. A `401` means the API key is wrong or revoked; a `422` means Cloud rejected the run's contents. Fix the cause and record the run again. ``` try: receipt = run.finish() except mechlens.MechLensUploadError as error: if not error.retryable: raise print(f"Saved locally at {error.local_path}; retrying") receipt = run.retry_upload() ``` Retries never create duplicate runs: each upload is sent with the run ID as its idempotency key. # Recording comparable runs # Recording comparable conditions A comparison is only as good as the two records behind it. If a condition used different prompts, model weights or decoding settings from the reference, a change in a metric might not come from your intervention. Cloud can only catch this when every run records those details. ## The rule > Record the same fields in every condition. Keep everything identical except the intervention. Cloud compares each condition with the reference, field by field, and with any other condition it declares in `compare_to`. When a field is missing from either side, Cloud reports it as **unknown**; it is never assumed to match. Declaring the experiment once with `mechlens.experiment()` shares these fields across conditions for you. See [What the checks mean](https://blaketylerfullerton.github.io/mechlens-sdk/integrity-checks/index.md). ## What to record | Argument | Record | Why it matters | | -------------------- | --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `project` | The same string for every condition | Only runs in the same project can be compared. | | conditions | `exp.reference()`, `exp.intervention()` and `exp.control()` | Cloud groups the runs into one experiment and compares each with the reference. With plain `start_run`, a name starting with `baseline` or `control` marks the baseline. | | `model` | `id` **and** `revision` | Without `revision`, Cloud can't confirm both runs used the same weights. | | `dataset` | `id`, `revision`, `split` | These three keys are always checked, and reported as unknown if missing. | | `generation` | Decoding settings: `temperature`, `max_new_tokens`, `do_sample` | A sampling change can shift outputs as much as an intervention. | | `config` | Any other settings, including `seed` | Every key is compared. | | `intervention` | On every condition except the reference | Should be the one difference from the reference. | | `sae` | The SAE release, ID, and hook point, if you use one | A one-sided SAE is reported as a difference. | | `metric_definitions` | How each metric is calculated, and its `example_field` | Cloud checks that both runs define each metric the same way, and computes intervals from the linked field. | | `code_commit` | Usually automatic | Detected with `git rev-parse HEAD` when you run inside a repository. | ## Example IDs Examples are matched across runs by `id`, not by position or prompt text. - **Use IDs that stay the same** across runs, such as dataset row IDs or hashes. Avoid loop counters if the order can change. - **Log the same set of IDs** in every condition. Examples logged in only one run are left out of the side-by-side view, and the coverage check reports the mismatch. - **Keep `input` identical** for each ID. If the same ID has different inputs, Cloud flags it as an input mismatch. ## Per-example fields Any extra keyword passed to `log_example` is stored with the example: ``` run.log_example( id="lockout-001", input=prompt, output=answer, refused=False, helpfulness=4, ) ``` Cloud marks an example as **behavior changed** when these extra fields differ between two runs, and **output changed** when the text differs. Log the measurement your conclusion depends on here, such as a refusal label, a score, or a probability, so a reviewer can see which examples moved. ## Seeds Cloud looks for `seed` or `random_seed` in `generation` first, then in `config`, and compares the first one it finds in each run. Put the seed in the same place in every condition. ## Metric definitions ``` metric_definitions={ "refusal_rate": { "description": "Share of prompts whose response declines to help.", "direction": "lower_is_better", }, } ``` Each description appears on the comparison page beneath the metric. If two runs report the same metric name with different definitions, the check reports a difference, so you know the numbers are not measuring the same thing. When a metric is the mean of a per-example field, name that field: ``` metric_definitions={ "refusal_rate": {"description": "Share of prompts refused.", "example_field": "refused"}, } ``` Cloud then computes each difference in `refusal_rate` from `refused` over the matched examples, with a 95% paired-bootstrap interval. A control that "has no effect" needs that interval to show how close to zero it is. Without `example_field`, Cloud shows the difference alone, and the draft finding lists the missing interval as a candidate limitation. ## A checklist before you compare - Same `project` in every condition - `model` includes `revision` - `dataset` includes `id`, `revision` and `split` - Same `generation` settings and seed - Same example IDs with identical inputs - `intervention` recorded on every condition except the reference - `metric_definitions` for every metric you log, with `example_field` where it applies # What the checks mean When you open a comparison, MechLens Cloud checks the two run records against each other. The checks answer one question: **do these records show that the two runs are comparable?** ## What the checks do and don't tell you The checks verify the **record**: whether the model, data, settings, and examples you recorded are the same in both runs, and which fields can't be compared because they weren't recorded. They do **not** tell you whether your conclusion is true. Cloud does not rerun your code, check that your recorded values are accurate, or judge whether a metric measures what you say it does. The checks show where a reader should trust the comparison and where they need to look more closely. There is deliberately no overall validity score. ## Statuses Each check has one of four statuses: | Status | Meaning | | ------------------ | ----------------------------------------------------------------------------------------------------------------- | | **matches** | Both runs record this field with identical values. | | **differs** | Both runs record it, with different values. For `sae` and `intervention`, also used when only one run records it. | | **unknown** | One or both runs don't record it, so the runs can't be compared on it. | | **not applicable** | The check doesn't apply. For example, input comparison when no example IDs match. | **Unknown is not a pass.** A comparison where every field is unknown has not been checked at all. To turn unknowns into results, record the field in both runs (see [Recording a comparable pair](https://blaketylerfullerton.github.io/mechlens-sdk/comparable-runs/index.md)). ## What is checked | Category | Fields | Notes | | ------------ | ---------------------------------------------------------------------- | ---------------------------------------------------- | | Model | `model.id`, `model.revision`, plus any other keys | `id` and `revision` are always reported. | | Dataset | `dataset.id`, `dataset.revision`, `dataset.split`, plus any other keys | The three named keys are always reported. | | SAE | Every key in `sae` | Recording it in only one run counts as a difference. | | Intervention | Every key in `intervention` | Expected to differ: this is what you're testing. | | Generation | Every key in `generation` except the seed | | | Config | Every key in `config` except the seed | | | Seed | `seed` or `random_seed` | Taken from `generation` first, then `config`. | | Code | `code_commit` | | | Metrics | `metric_definitions.` for each logged metric | Checks the definitions, not the values. | | Examples | Example IDs and inputs | See below. | Nested objects are compared key by key. For example, a `generation` dict produces separate checks for `generation.temperature` and `generation.max_new_tokens`. ## Example coverage Two checks cover the examples: - **IDs.** *Matches* when every example ID appears in both runs. *Differs* when some IDs appear in only one run. *Unknown* when either run logs no examples. - **Inputs.** For the IDs that do match, whether each input text is identical. Any mismatch is reported as a difference and highlighted in the example table. ## Differences in what was evaluated Most differences describe **how** the runs were produced, such as a different steering strength or temperature. Some of these are intended: the intervention, at least. A difference in **model**, **dataset**, or **examples** is more serious: the runs did not evaluate the same thing. Cloud shows these at the top of the comparison page: > The runs differ in what was evaluated. If you see this, the metric changes may come from the different model, data, or prompts rather than from your intervention. Fix the recording before you write a finding, or explain the difference in its limitations. ## Versions Each comparison report records the version of the checker that produced it. A published finding keeps the report it was published with, so later changes to the checks won't change what an earlier finding says. # Reference # API reference ## Environment variables | Variable | Default | Purpose | | ------------------ | ------------------------------------------------- | ------------------------------------------------------- | | `MECHLENS_API_KEY` | none | Workspace API key. Required unless passed as `api_key`. | | `MECHLENS_API_URL` | `https://mechlens-cloud-fgstc.ondigitalocean.app` | The MechLens Cloud deployment to upload to. | | `MECHLENS_DIR` | `.mechlens/runs` | Where finished runs are saved as JSON. | Arguments passed to start_run override these. ## Declaring an experiment ## mechlens.experiment ### Paper ``` Paper( arxiv: str, version: str | None = None, reported: Mapping[str, Mapping[str, float]] = dict(), where: str | None = None, ) ``` The paper an experiment reproduces, and the values it reports. Attributes: | Name | Type | Description | | ---------- | ----------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `arxiv` | `str` | The arXiv ID, such as "2406.11717". | | `version` | \`str | None\` | | `reported` | `Mapping[str, Mapping[str, float]]` | Metric names mapped to the paper's value for each condition, e.g. {"refusal_rate": {"reference": 0.95, "ablate-refusal-direction": 0.01}}. Use the same metric names you pass to log_metrics and the same condition names you record, so Cloud can put the paper's numbers beside yours. | | `where` | \`str | None\` | ### Experiment ``` Experiment( *, id: str, declaration: dict[str, Any], shared: dict[str, Any], ) ``` One reference run and the conditions compared with it, declared once. Create it with experiment. Record the reference with reference, then any number of intervention and control conditions. Conditions may be recorded in separate processes: every process that declares the same experiment joins it. Attributes: | Name | Type | Description | | --------- | ---------------- | ------------------------------------------------------------ | | `id` | | Groups this experiment's runs in Cloud, within its project. | | `project` | `str` | The project the runs are recorded to. | | `runs` | `dict[str, Run]` | Condition names mapped to the runs recorded by this process. | #### url ``` url: str | None ``` The experiment page, once this process has uploaded a condition; otherwise `None`. Also `None` with a custom client that has no `base_url`. #### reference ``` reference(name: str = 'reference', **overrides: Any) -> Run ``` Start the reference run, which every other condition is compared with. Parameters: | Name | Type | Description | Default | | ------------- | ----- | ----------------------------------------------------------------------------------------------------------------------------------- | ------------- | | `name` | `str` | The condition's name. Use the name your paper values are keyed by. | `'reference'` | | `**overrides` | `Any` | Any start_run argument, replacing the shared value for this run only. Cloud's comparison checks report any shared field you change. | `{}` | Returns: | Type | Description | | ----- | ----------------------------------------- | | `Run` | An open run, usable as a context manager. | Raises: | Type | Description | | ------------ | --------------------------------------------------- | | `ValueError` | A reference was already started by this experiment. | #### baseline ``` baseline(**overrides: Any) -> Run ``` Start the reference run, named `baseline` unless `name=` says otherwise. Kept from SDK 0.2. #### intervention ``` intervention( name: str | None = None, *, expect: Mapping[str, str] | None = None, compare_to: Sequence[str] = (), series: str | None = None, level: float | None = None, **overrides: Any, ) -> Run ``` Start a condition that changes the model in the way the claim is about. Pass `intervention={...}` to describe what it changes. Parameters: | Name | Type | Description | Default | | ------------- | ------------------- | --------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | \`str | None\` | The condition's name, unique within the experiment. Defaults to "@" in a sweep, otherwise "intervention". | | `expect` | \`Mapping[str, str] | None\` | Metric names mapped to the change you expect relative to the reference: "increase", "decrease" or "no_effect". Declare it before you see the results; Cloud shows it beside what was observed. | | `compare_to` | `Sequence[str]` | Names of other conditions this one should also be compared with, such as a control. The reference is always compared. | `()` | | `series` | \`str | None\` | Groups conditions that differ only in level, such as a steering-coefficient sweep. Cloud plots a series as a curve. | | `level` | \`float | None\` | This condition's position in the series. | | `**overrides` | `Any` | Any start_run argument, replacing the shared value for this run only. | `{}` | Returns: | Type | Description | | ----- | ----------------------------------------- | | `Run` | An open run, usable as a context manager. | #### control ``` control( name: str | None = None, *, expect: Mapping[str, str] | None = None, compare_to: Sequence[str] = (), series: str | None = None, level: float | None = None, **overrides: Any, ) -> Run ``` Start a condition the claim predicts should not have the intervention's effect. A control usually expects `"no_effect"`, such as ablating a random direction instead of the one under test. Takes the same arguments as intervention, except that `name` is required outside a sweep. Returns: | Type | Description | | ----- | ----------------------------------------- | | `Run` | An open run, usable as a context manager. | ### experiment ``` experiment( *, question: str, model: Mapping[str, Any], id: str | None = None, claim: str | None = None, reproduces: Paper | None = None, deviations: Sequence[str] = (), project: str | None = None, config: Mapping[str, Any] | None = None, dataset: Mapping[str, Any] | None = None, sae: Mapping[str, Any] | None = None, generation: Mapping[str, Any] | None = None, metric_definitions: Mapping[str, Any] | None = None, code_commit: str | None = None, api_key: str | None = None, base_url: str | None = None, local_dir: str | Path | None = None, client: CloudClient | None = None, ) -> Experiment ``` Declare an experiment once, then record its reference and other conditions. Everything Cloud needs to draft the finding lives here: the question, the claim, the deviations from any paper you reproduce, and the paper's reported values. The shared arguments apply to every condition, so the conditions are comparable by construction. No network call is made until a run finishes. Parameters: | Name | Type | Description | Default | | -------------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `question` | `str` | The research question the experiment helps answer. | *required* | | `model` | `Mapping[str, Any]` | Shared by every condition. Must include id; include revision. | *required* | | `id` | \`str | None\` | Identifies the experiment within its project, so conditions recorded in separate processes join it: lowercase letters, digits, ., _ and -. Defaults to an ID derived from the project, question and claim, so running the same declaration again joins the same experiment and editing the claim starts a new one. | | `claim` | \`str | None\` | The specific, checkable claim under test. | | `reproduces` | \`Paper | None\` | The paper this experiment reproduces, with its reported values. | | `deviations` | `Sequence[str]` | Every way this experiment knowingly differs from the paper or the intended protocol. Each becomes a candidate limitation in Cloud. | `()` | | `project` | \`str | None\` | Groups the experiment's runs. Defaults to reproduce- when reproduces is set, otherwise a slug of question. | | `config` | \`Mapping[str, Any] | None\` | Shared settings, such as seed. Defaults to {}. | | `dataset` | \`Mapping[str, Any] | None\` | Shared dataset, ideally with id, revision and split. | | `sae` | \`Mapping[str, Any] | None\` | Shared SAE details, if any. | | `generation` | \`Mapping[str, Any] | None\` | Shared decoding settings. | | `metric_definitions` | \`Mapping[str, Any] | None\` | Shared metric definitions. Give a metric an example_field naming the per-example field it averages, such as {"refusal_rate": {"description": "...", "example_field": "refused"}}, and Cloud shows its differences with confidence intervals. | | `code_commit` | \`str | None\` | Overrides the automatically detected commit. | | `api_key` | \`str | None\` | Overrides MECHLENS_API_KEY. | | `base_url` | \`str | None\` | Overrides MECHLENS_API_URL. | | `local_dir` | \`str | Path | None\` | | `client` | \`CloudClient | None\` | A preconfigured CloudClient, mainly for tests. | Returns: | Type | Description | | ------------ | ---------------------------------------------------------------------- | | `Experiment` | An experiment. Record the reference with with exp.reference() as run:, | | `Experiment` | each other condition with exp.intervention(...) or exp.control(...), | | `Experiment` | then open exp.url. | Raises: | Type | Description | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------ | | `ValueError` | The question or ID is invalid, a deviation is blank, a metric's example_field is not a field name, or the paper's reported values are malformed. | ## mechlens.Experiment ``` Experiment( *, id: str, declaration: dict[str, Any], shared: dict[str, Any], ) ``` One reference run and the conditions compared with it, declared once. Create it with experiment. Record the reference with reference, then any number of intervention and control conditions. Conditions may be recorded in separate processes: every process that declares the same experiment joins it. Attributes: | Name | Type | Description | | --------- | ---------------- | ------------------------------------------------------------ | | `id` | | Groups this experiment's runs in Cloud, within its project. | | `project` | `str` | The project the runs are recorded to. | | `runs` | `dict[str, Run]` | Condition names mapped to the runs recorded by this process. | ### url ``` url: str | None ``` The experiment page, once this process has uploaded a condition; otherwise `None`. Also `None` with a custom client that has no `base_url`. ### reference ``` reference(name: str = 'reference', **overrides: Any) -> Run ``` Start the reference run, which every other condition is compared with. Parameters: | Name | Type | Description | Default | | ------------- | ----- | ----------------------------------------------------------------------------------------------------------------------------------- | ------------- | | `name` | `str` | The condition's name. Use the name your paper values are keyed by. | `'reference'` | | `**overrides` | `Any` | Any start_run argument, replacing the shared value for this run only. Cloud's comparison checks report any shared field you change. | `{}` | Returns: | Type | Description | | ----- | ----------------------------------------- | | `Run` | An open run, usable as a context manager. | Raises: | Type | Description | | ------------ | --------------------------------------------------- | | `ValueError` | A reference was already started by this experiment. | ### intervention ``` intervention( name: str | None = None, *, expect: Mapping[str, str] | None = None, compare_to: Sequence[str] = (), series: str | None = None, level: float | None = None, **overrides: Any, ) -> Run ``` Start a condition that changes the model in the way the claim is about. Pass `intervention={...}` to describe what it changes. Parameters: | Name | Type | Description | Default | | ------------- | ------------------- | --------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | \`str | None\` | The condition's name, unique within the experiment. Defaults to "@" in a sweep, otherwise "intervention". | | `expect` | \`Mapping[str, str] | None\` | Metric names mapped to the change you expect relative to the reference: "increase", "decrease" or "no_effect". Declare it before you see the results; Cloud shows it beside what was observed. | | `compare_to` | `Sequence[str]` | Names of other conditions this one should also be compared with, such as a control. The reference is always compared. | `()` | | `series` | \`str | None\` | Groups conditions that differ only in level, such as a steering-coefficient sweep. Cloud plots a series as a curve. | | `level` | \`float | None\` | This condition's position in the series. | | `**overrides` | `Any` | Any start_run argument, replacing the shared value for this run only. | `{}` | Returns: | Type | Description | | ----- | ----------------------------------------- | | `Run` | An open run, usable as a context manager. | ### control ``` control( name: str | None = None, *, expect: Mapping[str, str] | None = None, compare_to: Sequence[str] = (), series: str | None = None, level: float | None = None, **overrides: Any, ) -> Run ``` Start a condition the claim predicts should not have the intervention's effect. A control usually expects `"no_effect"`, such as ablating a random direction instead of the one under test. Takes the same arguments as intervention, except that `name` is required outside a sweep. Returns: | Type | Description | | ----- | ----------------------------------------- | | `Run` | An open run, usable as a context manager. | ### baseline ``` baseline(**overrides: Any) -> Run ``` Start the reference run, named `baseline` unless `name=` says otherwise. Kept from SDK 0.2. ## mechlens.Paper ``` Paper( arxiv: str, version: str | None = None, reported: Mapping[str, Mapping[str, float]] = dict(), where: str | None = None, ) ``` The paper an experiment reproduces, and the values it reports. Attributes: | Name | Type | Description | | ---------- | ----------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `arxiv` | `str` | The arXiv ID, such as "2406.11717". | | `version` | \`str | None\` | | `reported` | `Mapping[str, Mapping[str, float]]` | Metric names mapped to the paper's value for each condition, e.g. {"refusal_rate": {"reference": 0.95, "ablate-refusal-direction": 0.01}}. Use the same metric names you pass to log_metrics and the same condition names you record, so Cloud can put the paper's numbers beside yours. | | `where` | \`str | None\` | ## Recording runs ## mechlens.start_run ``` start_run( *, project: str, name: str, model: Mapping[str, Any], config: Mapping[str, Any], question: str, dataset: Mapping[str, Any] | None = None, sae: Mapping[str, Any] | None = None, intervention: Mapping[str, Any] | None = None, generation: Mapping[str, Any] | None = None, metric_definitions: Mapping[str, Any] | None = None, code_commit: str | None = None, api_key: str | None = None, base_url: str | None = None, local_dir: str | Path | None = None, client: CloudClient | None = None, ) -> Run ``` Create a local-first experiment run ready to receive examples and metrics. Everything except `project`, `name`, `model`, `config` and `question` is optional, but Cloud can only check what you record: a field missing from either run is reported as unknown rather than assumed to match. Parameters: | Name | Type | Description | Default | | -------------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | | `project` | `str` | Groups runs that can be compared, such as a baseline and its intervention. | *required* | | `name` | `str` | This run's label within the project. A name containing baseline or control is chosen as the default baseline in Cloud. | *required* | | `model` | `Mapping[str, Any]` | Must include id. Include revision so the comparison can confirm both runs used the same weights. | *required* | | `config` | `Mapping[str, Any]` | Any other settings for this run, such as seed or steering_strength. | *required* | | `question` | `str` | The research question the run helps answer. | *required* | | `dataset` | \`Mapping[str, Any] | None\` | Where the examples came from, ideally with id, revision and split. | | `sae` | \`Mapping[str, Any] | None\` | The sparse autoencoder used, if any. | | `intervention` | \`Mapping[str, Any] | None\` | What this run changes relative to the baseline. Leave it out of the baseline run. | | `generation` | \`Mapping[str, Any] | None\` | Decoding settings such as temperature and max_new_tokens. | | `metric_definitions` | \`Mapping[str, Any] | None\` | Metric names mapped to how each is calculated, e.g. {"refusal_rate": {"description": "..."}}. | | `code_commit` | \`str | None\` | Commit of the code that produced the run. Detected with git rev-parse HEAD when omitted. | | `api_key` | \`str | None\` | Overrides the MECHLENS_API_KEY environment variable. | | `base_url` | \`str | None\` | Overrides MECHLENS_API_URL. Defaults to MechLens Cloud, https://mechlens-cloud-fgstc.ondigitalocean.app. | | `local_dir` | \`str | Path | None\` | | `client` | \`CloudClient | None\` | A preconfigured CloudClient, mainly for tests. When given, api_key and base_url are ignored. | Returns: | Type | Description | | ----- | ------------------------------------------------ | | `Run` | An open run. Log examples and metrics, then call | | `Run` | finish. | Raises: | Type | Description | | ---------------------------- | ------------------------------------------------------ | | `ValueError` | A required argument is empty or not JSON-serializable. | | `TypeError` | config is not a mapping. | | `MechLensConfigurationError` | No API key was passed or set in the environment. | ## mechlens.Run ``` Run( data: dict[str, Any], *, client: CloudClient, local_dir: Path, ) ``` One experiment run, recorded locally and uploaded when finished. Create runs with start_run rather than directly. A run can also be used as a context manager: leaving the `with` block without an exception calls finish. Attributes: | Name | Type | Description | | ------------ | ---- | ----------------------------------------------- | | `data` | | The run record that will be saved and uploaded. | | `local_path` | | Where the finished run is written as JSON. | ### log_example ``` log_example( *, id: str, input: str, output: str, **fields: Any ) -> None ``` Record one input and the model's output for it. Use the same `id` for the same prompt in every run you want to compare; MechLens matches examples across runs by ID. Parameters: | Name | Type | Description | Default | | ---------- | ----- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | | `id` | `str` | Stable identifier for this example, unique within the run. | *required* | | `input` | `str` | The prompt or input given to the model. | *required* | | `output` | `str` | What the model produced. | *required* | | `**fields` | `Any` | Per-example measurements such as refused=True or helpfulness=4. Values must be JSON-serializable. Cloud reports an example as "behavior changed" when these fields differ between runs. | `{}` | Raises: | Type | Description | | -------------- | ---------------------------------------------------------------------- | | `ValueError` | The ID is empty or already logged, or fields is not JSON-serializable. | | `TypeError` | input or output is not a string. | | `RuntimeError` | The run is already finished. | ### log_metrics ``` log_metrics(metrics: Mapping[str, Any]) -> None ``` Record run-level metrics, merging with any logged earlier. Parameters: | Name | Type | Description | Default | | --------- | ------------------- | --------------------------------------------------------------------------------------------------------------------------------- | ---------- | | `metrics` | `Mapping[str, Any]` | Metric names mapped to JSON-serializable values. Numbers must be finite. Names containing rate are shown as percentages in Cloud. | *required* | Raises: | Type | Description | | -------------- | --------------------------------------------------------------------------------- | | `ValueError` | A key is empty, a value is not JSON-serializable, or a number is NaN or infinite. | | `RuntimeError` | The run is already finished. | ### finish ``` finish() -> UploadReceipt ``` Seal the run, save it to disk, then upload it. The run is written to local_path before any network call, so a failed upload never loses data. Calling `finish` again returns the same receipt without uploading twice. Returns: | Type | Description | | --------------- | ---------------------------------------- | | `UploadReceipt` | The uploaded run's ID and its Cloud URL. | Raises: | Type | Description | | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | | `MechLensUploadError` | The upload failed. The local copy is kept. If the error's retryable is true, call retry_upload; otherwise its message says what to fix. | ### retry_upload ``` retry_upload() -> UploadReceipt ``` Upload a finished run whose earlier upload failed. Uploads use the run ID as an idempotency key, so retrying never creates a duplicate run in Cloud. Returns: | Type | Description | | --------------- | ---------------------------------------- | | `UploadReceipt` | The uploaded run's ID and its Cloud URL. | Raises: | Type | Description | | --------------------- | ------------------------------ | | `RuntimeError` | The run has not been finished. | | `MechLensUploadError` | The upload failed again. | ## mechlens.UploadReceipt ``` UploadReceipt(run_id: str, url: str) ``` Confirmation that Cloud stored a run. Attributes: | Name | Type | Description | | -------- | ----- | --------------------------------------------- | | `run_id` | `str` | The run's ID, the same as the local record's. | | `url` | `str` | Where to open the run in MechLens Cloud. | ## Errors ## mechlens.MechLensError Bases: `Exception` Base exception raised by the SDK. ## mechlens.MechLensConfigurationError Bases: `MechLensError` The SDK is missing configuration required to contact MechLens. ## mechlens.MechLensUploadError ``` MechLensUploadError( message: str, *, local_path: Path, status_code: int | None = None, retryable: bool = True, ) ``` Bases: `MechLensError` An upload failed after the run was safely stored on disk. Attributes: | Name | Type | Description | | ------------- | ---- | ----------------------------------------------------------------------------------------------------------------- | | `local_path` | | The saved copy of the run. | | `status_code` | | Cloud's HTTP status, or None if Cloud was never reached. | | `retryable` | | Whether retrying the same run could succeed. False for an invalid key or a rejected run, which need fixing first. | ## mechlens.MechLensWarning Bases: `UserWarning` Something about a run will weaken its evidence in Cloud, but it can still be recorded. ## Advanced ## mechlens.CloudClient ``` CloudClient( *, api_key: str, base_url: str, timeout: float = 30.0, transport: BaseTransport | None = None, ) ``` Small synchronous client for the MechLens ingestion API. ### upload_run ``` upload_run( data: Mapping[str, Any], *, local_path ) -> UploadReceipt ``` Send one finished run to Cloud. Raises: | Type | Description | | --------------------- | -------------------------------------------------------------------------------------------------------------- | | `MechLensUploadError` | Cloud could not be reached or rejected the run. Its status_code and retryable attributes say whether to retry. |