API reference

Environment variables

Variable	Default	Purpose
MECHLENS_API_KEY	none	Workspace API key. Required unless passed as api_key.
MECHLENS_API_URL	https://mechlens-cloud-fgstc.ondigitalocean.app	The MechLens Cloud deployment to upload to.
MECHLENS_DIR	.mechlens/runs	Where finished runs are saved as JSON.

Arguments passed to start_run override these.

Declaring an experiment

mechlens.experiment

Paper dataclass

Paper(
    arxiv: str,
    version: str | None = None,
    reported: Mapping[str, Mapping[str, float]] = dict(),
    where: str | None = None,
)

The paper an experiment reproduces, and the values it reports.

Attributes:

Name	Type	Description
arxiv	str	The arXiv ID, such as "2406.11717".
version	str | None	The arXiv version the reported values come from, such as "v3".
reported	Mapping[str, Mapping[str, float]]	Metric names mapped to the paper's value for each condition, e.g. {"refusal_rate": {"reference": 0.95, "ablate-refusal-direction": 0.01}}. Use the same metric names you pass to log_metrics and the same condition names you record, so Cloud can put the paper's numbers beside yours.
where	str | None	Where in the paper the values appear, such as "Table 10".

Experiment

Experiment(
    *,
    id: str,
    declaration: dict[str, Any],
    shared: dict[str, Any],
)

One reference run and the conditions compared with it, declared once.

Create it with experiment. Record the reference with reference, then any number of intervention and control conditions. Conditions may be recorded in separate processes: every process that declares the same experiment joins it.

Attributes:

Name	Type	Description
id		Groups this experiment's runs in Cloud, within its project.
project	str	The project the runs are recorded to.
runs	dict[str, Run]	Condition names mapped to the runs recorded by this process.

url property

url: str | None

The experiment page, once this process has uploaded a condition; otherwise None.

Also None with a custom client that has no base_url.

reference

reference(name: str = 'reference', **overrides: Any) -> Run

Start the reference run, which every other condition is compared with.

Parameters:

Name	Type	Description	Default
name	str	The condition's name. Use the name your paper values are keyed by.	'reference'
**overrides	Any	Any start_run argument, replacing the shared value for this run only. Cloud's comparison checks report any shared field you change.	{}

Returns:

Type	Description
Run	An open run, usable as a context manager.

Raises:

Type	Description
ValueError	A reference was already started by this experiment.

baseline

baseline(**overrides: Any) -> Run

Start the reference run, named baseline unless name= says otherwise. Kept from SDK 0.2.

intervention

intervention(
    name: str | None = None,
    *,
    expect: Mapping[str, str] | None = None,
    compare_to: Sequence[str] = (),
    series: str | None = None,
    level: float | None = None,
    **overrides: Any,
) -> Run

Start a condition that changes the model in the way the claim is about.

Pass intervention={...} to describe what it changes.

Parameters:

Name	Type	Description	Default
name	str | None	The condition's name, unique within the experiment. Defaults to "<series>@<level>" in a sweep, otherwise "intervention".	None
expect	Mapping[str, str] | None	Metric names mapped to the change you expect relative to the reference: "increase", "decrease" or "no_effect". Declare it before you see the results; Cloud shows it beside what was observed.	None
compare_to	Sequence[str]	Names of other conditions this one should also be compared with, such as a control. The reference is always compared.	()
series	str | None	Groups conditions that differ only in level, such as a steering-coefficient sweep. Cloud plots a series as a curve.	None
level	float | None	This condition's position in the series.	None
**overrides	Any	Any start_run argument, replacing the shared value for this run only.	{}

Returns:

Type	Description
Run	An open run, usable as a context manager.

control

control(
    name: str | None = None,
    *,
    expect: Mapping[str, str] | None = None,
    compare_to: Sequence[str] = (),
    series: str | None = None,
    level: float | None = None,
    **overrides: Any,
) -> Run

Start a condition the claim predicts should not have the intervention's effect.

A control usually expects "no_effect", such as ablating a random direction instead of the one under test. Takes the same arguments as intervention, except that name is required outside a sweep.

Returns:

Type	Description
Run	An open run, usable as a context manager.

experiment

experiment(
    *,
    question: str,
    model: Mapping[str, Any],
    id: str | None = None,
    claim: str | None = None,
    reproduces: Paper | None = None,
    deviations: Sequence[str] = (),
    project: str | None = None,
    config: Mapping[str, Any] | None = None,
    dataset: Mapping[str, Any] | None = None,
    sae: Mapping[str, Any] | None = None,
    generation: Mapping[str, Any] | None = None,
    metric_definitions: Mapping[str, Any] | None = None,
    code_commit: str | None = None,
    api_key: str | None = None,
    base_url: str | None = None,
    local_dir: str | Path | None = None,
    client: CloudClient | None = None,
) -> Experiment

Declare an experiment once, then record its reference and other conditions.

Everything Cloud needs to draft the finding lives here: the question, the claim, the deviations from any paper you reproduce, and the paper's reported values. The shared arguments apply to every condition, so the conditions are comparable by construction. No network call is made until a run finishes.

Parameters:

Name	Type	Description	Default
question	str	The research question the experiment helps answer.	required
model	Mapping[str, Any]	Shared by every condition. Must include id; include revision.	required
id	str | None	Identifies the experiment within its project, so conditions recorded in separate processes join it: lowercase letters, digits, ., _ and -. Defaults to an ID derived from the project, question and claim, so running the same declaration again joins the same experiment and editing the claim starts a new one.	None
claim	str | None	The specific, checkable claim under test.	None
reproduces	Paper | None	The paper this experiment reproduces, with its reported values.	None
deviations	Sequence[str]	Every way this experiment knowingly differs from the paper or the intended protocol. Each becomes a candidate limitation in Cloud.	()
project	str | None	Groups the experiment's runs. Defaults to reproduce-<arxiv-id> when reproduces is set, otherwise a slug of question.	None
config	Mapping[str, Any] | None	Shared settings, such as seed. Defaults to {}.	None
dataset	Mapping[str, Any] | None	Shared dataset, ideally with id, revision and split.	None
sae	Mapping[str, Any] | None	Shared SAE details, if any.	None
generation	Mapping[str, Any] | None	Shared decoding settings.	None
metric_definitions	Mapping[str, Any] | None	Shared metric definitions. Give a metric an example_field naming the per-example field it averages, such as {"refusal_rate": {"description": "...", "example_field": "refused"}}, and Cloud shows its differences with confidence intervals.	None
code_commit	str | None	Overrides the automatically detected commit.	None
api_key	str | None	Overrides MECHLENS_API_KEY.	None
base_url	str | None	Overrides MECHLENS_API_URL.	None
local_dir	str | Path | None	Overrides MECHLENS_DIR.	None
client	CloudClient | None	A preconfigured CloudClient, mainly for tests.	None

Returns:

Type	Description
Experiment	An experiment. Record the reference with with exp.reference() as run:,
Experiment	each other condition with exp.intervention(...) or exp.control(...),
Experiment	then open exp.url.

Raises:

Type	Description
ValueError	The question or ID is invalid, a deviation is blank, a metric's example_field is not a field name, or the paper's reported values are malformed.

mechlens.Experiment

Experiment(
    *,
    id: str,
    declaration: dict[str, Any],
    shared: dict[str, Any],
)

One reference run and the conditions compared with it, declared once.

Create it with experiment. Record the reference with reference, then any number of intervention and control conditions. Conditions may be recorded in separate processes: every process that declares the same experiment joins it.

Attributes:

Name	Type	Description
id		Groups this experiment's runs in Cloud, within its project.
project	str	The project the runs are recorded to.
runs	dict[str, Run]	Condition names mapped to the runs recorded by this process.

url property

url: str | None

The experiment page, once this process has uploaded a condition; otherwise None.

Also None with a custom client that has no base_url.

reference

reference(name: str = 'reference', **overrides: Any) -> Run

Start the reference run, which every other condition is compared with.

Parameters:

Name	Type	Description	Default
name	str	The condition's name. Use the name your paper values are keyed by.	'reference'
**overrides	Any	Any start_run argument, replacing the shared value for this run only. Cloud's comparison checks report any shared field you change.	{}

Returns:

Type	Description
Run	An open run, usable as a context manager.

Raises:

Type	Description
ValueError	A reference was already started by this experiment.

intervention

intervention(
    name: str | None = None,
    *,
    expect: Mapping[str, str] | None = None,
    compare_to: Sequence[str] = (),
    series: str | None = None,
    level: float | None = None,
    **overrides: Any,
) -> Run

Start a condition that changes the model in the way the claim is about.

Pass intervention={...} to describe what it changes.

Parameters:

Name	Type	Description	Default
name	str | None	The condition's name, unique within the experiment. Defaults to "<series>@<level>" in a sweep, otherwise "intervention".	None
expect	Mapping[str, str] | None	Metric names mapped to the change you expect relative to the reference: "increase", "decrease" or "no_effect". Declare it before you see the results; Cloud shows it beside what was observed.	None
compare_to	Sequence[str]	Names of other conditions this one should also be compared with, such as a control. The reference is always compared.	()
series	str | None	Groups conditions that differ only in level, such as a steering-coefficient sweep. Cloud plots a series as a curve.	None
level	float | None	This condition's position in the series.	None
**overrides	Any	Any start_run argument, replacing the shared value for this run only.	{}

Returns:

Type	Description
Run	An open run, usable as a context manager.

control

control(
    name: str | None = None,
    *,
    expect: Mapping[str, str] | None = None,
    compare_to: Sequence[str] = (),
    series: str | None = None,
    level: float | None = None,
    **overrides: Any,
) -> Run

Start a condition the claim predicts should not have the intervention's effect.

A control usually expects "no_effect", such as ablating a random direction instead of the one under test. Takes the same arguments as intervention, except that name is required outside a sweep.

Returns:

Type	Description
Run	An open run, usable as a context manager.

baseline

baseline(**overrides: Any) -> Run

Start the reference run, named baseline unless name= says otherwise. Kept from SDK 0.2.

mechlens.Paper dataclass

Paper(
    arxiv: str,
    version: str | None = None,
    reported: Mapping[str, Mapping[str, float]] = dict(),
    where: str | None = None,
)

The paper an experiment reproduces, and the values it reports.

Attributes:

Name	Type	Description
arxiv	str	The arXiv ID, such as "2406.11717".
version	str | None	The arXiv version the reported values come from, such as "v3".
reported	Mapping[str, Mapping[str, float]]	Metric names mapped to the paper's value for each condition, e.g. {"refusal_rate": {"reference": 0.95, "ablate-refusal-direction": 0.01}}. Use the same metric names you pass to log_metrics and the same condition names you record, so Cloud can put the paper's numbers beside yours.
where	str | None	Where in the paper the values appear, such as "Table 10".

Recording runs

mechlens.start_run

start_run(
    *,
    project: str,
    name: str,
    model: Mapping[str, Any],
    config: Mapping[str, Any],
    question: str,
    dataset: Mapping[str, Any] | None = None,
    sae: Mapping[str, Any] | None = None,
    intervention: Mapping[str, Any] | None = None,
    generation: Mapping[str, Any] | None = None,
    metric_definitions: Mapping[str, Any] | None = None,
    code_commit: str | None = None,
    api_key: str | None = None,
    base_url: str | None = None,
    local_dir: str | Path | None = None,
    client: CloudClient | None = None,
) -> Run

Create a local-first experiment run ready to receive examples and metrics.

Everything except project, name, model, config and question is optional, but Cloud can only check what you record: a field missing from either run is reported as unknown rather than assumed to match.

Parameters:

Name	Type	Description	Default
project	str	Groups runs that can be compared, such as a baseline and its intervention.	required
name	str	This run's label within the project. A name containing baseline or control is chosen as the default baseline in Cloud.	required
model	Mapping[str, Any]	Must include id. Include revision so the comparison can confirm both runs used the same weights.	required
config	Mapping[str, Any]	Any other settings for this run, such as seed or steering_strength.	required
question	str	The research question the run helps answer.	required
dataset	Mapping[str, Any] | None	Where the examples came from, ideally with id, revision and split.	None
sae	Mapping[str, Any] | None	The sparse autoencoder used, if any.	None
intervention	Mapping[str, Any] | None	What this run changes relative to the baseline. Leave it out of the baseline run.	None
generation	Mapping[str, Any] | None	Decoding settings such as temperature and max_new_tokens.	None
metric_definitions	Mapping[str, Any] | None	Metric names mapped to how each is calculated, e.g. {"refusal_rate": {"description": "..."}}.	None
code_commit	str | None	Commit of the code that produced the run. Detected with git rev-parse HEAD when omitted.	None
api_key	str | None	Overrides the MECHLENS_API_KEY environment variable.	None
base_url	str | None	Overrides MECHLENS_API_URL. Defaults to MechLens Cloud, https://mechlens-cloud-fgstc.ondigitalocean.app.	None
local_dir	str | Path | None	Overrides MECHLENS_DIR, where finished runs are saved. Defaults to .mechlens/runs.	None
client	CloudClient | None	A preconfigured CloudClient, mainly for tests. When given, api_key and base_url are ignored.	None

Returns:

Type	Description
Run	An open run. Log examples and metrics, then call
Run	finish.

Raises:

Type	Description
ValueError	A required argument is empty or not JSON-serializable.
TypeError	config is not a mapping.
MechLensConfigurationError	No API key was passed or set in the environment.

mechlens.Run

Run(
    data: dict[str, Any],
    *,
    client: CloudClient,
    local_dir: Path,
)

One experiment run, recorded locally and uploaded when finished.

Create runs with start_run rather than directly. A run can also be used as a context manager: leaving the with block without an exception calls finish.

Attributes:

Name	Type	Description
data		The run record that will be saved and uploaded.
local_path		Where the finished run is written as JSON.

log_example

log_example(
    *, id: str, input: str, output: str, **fields: Any
) -> None

Record one input and the model's output for it.

Use the same id for the same prompt in every run you want to compare; MechLens matches examples across runs by ID.

Parameters:

Name	Type	Description	Default
id	str	Stable identifier for this example, unique within the run.	required
input	str	The prompt or input given to the model.	required
output	str	What the model produced.	required
**fields	Any	Per-example measurements such as refused=True or helpfulness=4. Values must be JSON-serializable. Cloud reports an example as "behavior changed" when these fields differ between runs.	{}

Raises:

Type	Description
ValueError	The ID is empty or already logged, or fields is not JSON-serializable.
TypeError	input or output is not a string.
RuntimeError	The run is already finished.

log_metrics

log_metrics(metrics: Mapping[str, Any]) -> None

Record run-level metrics, merging with any logged earlier.

Parameters:

Name	Type	Description	Default
metrics	Mapping[str, Any]	Metric names mapped to JSON-serializable values. Numbers must be finite. Names containing rate are shown as percentages in Cloud.	required

Raises:

Type	Description
ValueError	A key is empty, a value is not JSON-serializable, or a number is NaN or infinite.
RuntimeError	The run is already finished.

finish

finish() -> UploadReceipt

Seal the run, save it to disk, then upload it.

The run is written to local_path before any network call, so a failed upload never loses data. Calling finish again returns the same receipt without uploading twice.

Returns:

Type	Description
UploadReceipt	The uploaded run's ID and its Cloud URL.

Raises:

Type	Description
MechLensUploadError	The upload failed. The local copy is kept. If the error's retryable is true, call retry_upload; otherwise its message says what to fix.

retry_upload

retry_upload() -> UploadReceipt

Upload a finished run whose earlier upload failed.

Uploads use the run ID as an idempotency key, so retrying never creates a duplicate run in Cloud.

Returns:

Type	Description
UploadReceipt	The uploaded run's ID and its Cloud URL.

Raises:

Type	Description
RuntimeError	The run has not been finished.
MechLensUploadError	The upload failed again.

mechlens.UploadReceipt dataclass

UploadReceipt(run_id: str, url: str)

Confirmation that Cloud stored a run.

Attributes:

Name	Type	Description
run_id	str	The run's ID, the same as the local record's.
url	str	Where to open the run in MechLens Cloud.

Errors

mechlens.MechLensError

Bases: Exception

Base exception raised by the SDK.

mechlens.MechLensConfigurationError

Bases: MechLensError

The SDK is missing configuration required to contact MechLens.

mechlens.MechLensUploadError

MechLensUploadError(
    message: str,
    *,
    local_path: Path,
    status_code: int | None = None,
    retryable: bool = True,
)

Bases: MechLensError

An upload failed after the run was safely stored on disk.

Attributes:

Name	Type	Description
local_path		The saved copy of the run.
status_code		Cloud's HTTP status, or None if Cloud was never reached.
retryable		Whether retrying the same run could succeed. False for an invalid key or a rejected run, which need fixing first.

mechlens.MechLensWarning

Bases: UserWarning

Something about a run will weaken its evidence in Cloud, but it can still be recorded.

Advanced

mechlens.CloudClient

CloudClient(
    *,
    api_key: str,
    base_url: str,
    timeout: float = 30.0,
    transport: BaseTransport | None = None,
)

Small synchronous client for the MechLens ingestion API.

upload_run

upload_run(
    data: Mapping[str, Any], *, local_path
) -> UploadReceipt

Send one finished run to Cloud.

Raises:

Type	Description
MechLensUploadError	Cloud could not be reached or rejected the run. Its status_code and retryable attributes say whether to retry.