> ## Documentation Index
> Fetch the complete documentation index at: https://docs.amlok.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Reproducibility and research artifacts

> Reproduce public-source acquisition, the financing-record index, financial-document retrieval results and source-integrity checks.

The current collection includes complete public source releases, field dictionaries, review records and two supervised baseline models trained on real loan observations. The earlier 0.2 release also contains the completed financial-document retrieval study. Each study keeps its own input, prediction and result manifests.

[Download the current research methods and results bundle](https://docs.amlok.io/downloads/research/amlok-public-records-0.3.zip). It includes the documentation, aggregate source metadata, dictionaries, measured predictions and method implementations.

## Current collection and training artifacts

| Artifact | Contents |
| - | - |
| [Dataset inventory](https://docs.amlok.io/research/datasets.md) | Acquired records, field families, coverage and labeling scope. |
| [Collection summary](https://docs.amlok.io/downloads/research/corpus/summary.json) | Counts with explicit observation units. |
| [Acquisition manifest](https://docs.amlok.io/downloads/research/corpus/acquisition-manifest.json) | Original source URLs, acquisition times, byte counts and SHA-256 hashes. |
| [Source tables](https://docs.amlok.io/downloads/research/corpus/source-tables.json) | Complete release sizes, source schemas and populated-field counts. |
| [Baseline result manifest](https://docs.amlok.io/downloads/research/corpus/baseline-results.json) | Model settings, features, partitions, metrics, uncertainty and scientific limits. |
| [Baseline predictions](https://docs.amlok.io/downloads/research/corpus/baseline-predictions.csv) | 11,996 hashed record IDs, source-derived outcomes and both model predictions. |
| [Artifact integrity](https://docs.amlok.io/downloads/research/corpus/integrity.json) | Byte sizes and hashes for the public collection artifacts. |

The controlled research workspace retains source originals, complete normalized tables, review decisions, model checkpoints and cohort-building code. Public artifacts omit borrower names and addresses. Source-specific access and use conditions remain attached to the original files.

The training implementation is `scripts/research/train_public_baselines.py`. Set `AMLOK_RESEARCH_WORKSPACE` to the frozen corpus workspace containing `processed/training-cohort.json`. It fits both models with seed 42, preserves the temporal and normalized-entity partitions, saves predictions and recomputes the grouped uncertainty. Its results can be checked against the recorded corpus digest and runtime. The broader transaction and foundation-model studies use separate datasets and protocols.

## Financial-document study artifacts

The [0.2 public-record bundle](https://docs.amlok.io/downloads/research/amlok-public-records-0.2.zip) preserves the source-inspection and FinQA retrieval workflow. It contains derived artifacts and method code rather than the original borrower-row corpora. The counts in its historical record index describe that inspection, not the expanded 0.3 collection.

### Preserved 0.2 artifacts

| Artifact | Purpose |
| - | - |
| [Source register](https://docs.amlok.io/downloads/research/public-records/source-register.json) | 26 reviewed sources, available fields, acquisition status, record unit and limitations. |
| [Acquisition manifest](https://docs.amlok.io/downloads/research/public-records/acquisition-manifest.json) | Exact URLs, UTC acquisition times, byte counts, retrieval scope and SHA-256 hashes. |
| [Record index](https://docs.amlok.io/downloads/research/public-records/record-index.jsonl) | 1,047 source observations with deterministic IDs, source hashes, page/row locations and field availability. |
| [Source analysis](https://docs.amlok.io/downloads/research/public-records/source-analysis.json) | Counts, aggregate reconciliation, actual financing-table values and extraction probes. |
| [Retrieval results](https://docs.amlok.io/downloads/research/public-records/retrieval-results.json) | Methods, dataset and script digests, measured metrics and paired uncertainty. |
| [Item-level rankings](https://docs.amlok.io/downloads/research/public-records/retrieval-predictions.json) | Every question ID, source filename, gold evidence IDs, top-five predictions and scores. |

The observation index comprises 1,014 parser-identified Québec interventions, 30 EDC disclosure rows, two issuer announcements and one aggregate facility-position table. These are source observations, not 1,047 adjudicated closed loans. The index retains company-text hashes for linkage reproducibility without republishing the company lists. A hash match is not independent legal-entity verification.

## Financial-document workflow

1. Retrieve explicitly listed public URLs through bounded requests. Record unsuccessful retrievals and distinguish complete files from byte prefixes.
2. Parse source tables and dictionaries. Preserve their units, repeated-language presentations and record types.
3. Render selected original PDF pages to inspect values the text layer may omit.
4. Build stable record IDs from the acquired source hash and explicit location. Record field availability and unresolved linkage.
5. Rank real FinQA evidence using two fixed lexical methods; score only after predictions exist.
6. Recompute metrics from saved item-level outputs and verify source, script and prediction hashes.

The source review includes selected manual visual checks, not independent double annotation of the full corpus. No paid registry searches, authenticated borrower-data downloads, hosted-model calls or production training jobs were executed.

## Reproduce the financial-document study

Use Python 3.12 in an isolated environment from the documentation repository or extracted bundle:

```bash theme={null}
python3 -m venv /tmp/amlok-public-research
/tmp/amlok-public-research/bin/python -m pip install -r scripts/research/requirements-public.txt
/tmp/amlok-public-research/bin/python scripts/research/acquire_public_audit.py
/tmp/amlok-public-research/bin/python scripts/research/analyze_public_sources.py
/tmp/amlok-public-research/bin/python scripts/research/build_source_register.py
/tmp/amlok-public-research/bin/python scripts/research/benchmark_public_evidence.py
/tmp/amlok-public-research/bin/python scripts/research/test_public_research.py
```

Acquisition writes original files to `artifacts/research-review/public-records`. Derived public outputs go to `downloads/research/public-records`. Downloading a source requires respecting its current terms; the acquisition script is not a grant of commercial training or redistribution rights. FinanceBench is inspected for source feasibility and is excluded from scoring/training.

The acquisition step skips successful snapshots already present. Reproduce this release with files matching the published hashes. If upstream content changes, preserve the new version separately and treat the run as a new dataset snapshot. If a source is unavailable, retain that status; do not replace it with invented rows. The SBA request deliberately captures only a bounded byte prefix ending on a complete row.

## Benchmark integrity

The FinQA ranking function uses only source text/table fields and the question. It excludes answers, programs, existing retrieved evidence and model-input annotations. Gold evidence IDs enter scoring after ranking. No supervised fitting or test-label tuning is performed.

The evaluation uses all 1,147 test questions. Table rows include their column headings. Recall is averaged per question; complete-evidence rate requires every annotated supporting item. Confidence intervals resample source filenames in paired comparisons and retain all questions in each selected cluster. They do not represent variation across independent institutions.

The checks verify source digests, record-index uniqueness and locators, count denominators, saved metrics, deterministic prediction reproduction, and exclusion of reference-answer fields from ranking. These checks establish artifact consistency, not a trained model's suitability for lending.

## Extending the corpus and experiments

Keep source snapshots immutable. Add retrieval history, licence/permission decisions, schema changes, extraction versions, correction records and adjudication logs. Publish the population flow from fetched documents to parsed observations, resolved facilities, eligible outcomes and final evaluation examples.

For a trained-model run, add checkpoint hashes, approved training purposes, split manifests, seeds, optimization settings, compute, failures and frozen predictions. For a frontier run, add exact API model identifiers, prompt/tool versions, raw responses and cost/latency logs under the [comparison protocol](https://docs.amlok.io/research/frontier-benchmark.md). Never fill an unrun result cell with an expected score.

Prior research artifacts remain versioned separately. They are not pooled with this public-record dataset, its denominators or its performance claims.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.