Review quality

Precision and recall against the planted non-conformities

The sample package contains 39 deliberately planted non-conformities (values, materials, tags, missing sections, outdated references, drawing title blocks and dimensions, Dutch-language procedure). The latest completed run on the sample package is scored here.

Revision A

run 9d8ce9bbb27f · 20 Oct 2026 11:29 UTC · LongCat 2.5 (free fallback model)

100%

recall · 36 of 36 planted found

92%

precision · 37 of 40 findings hit a planted defect

34 / 2

found by rules / by agents

findings not in the planted set (listed below)

By priority

	planted	found	recall	findings	precision
High	19	19	100%	20	100%
Low	6	6	100%	6	83%
Medium	11	11	100%	14	85%

By file format

	planted	found	recall	findings	precision
DOCX	10	10	100%	12	83%
DWG	3	3	100%	3	100%
DXF	6	6	100%	6	100%
PDF	7	7	100%	9	88%
XLSX	10	10	100%	10	100%

By document language

	planted	found	recall	findings	precision
EN	32	32	100%	34	97%
NL	4	4	100%	6	66%

Revision B: revision tracking

run c9e39d901d9e

6/6

fixed defects marked resolved

3/3

defects introduced in revision B found as new

100%

recall on the revised package (33/33)

97%

precision on the revised package

Planted non-conformities (revision A)

ID	Document	Defect	Result
D01	NCT-INS-DS-0101	Calibrated range 0-100 barg is below 1.1 x design pressure (165 barg).	✓ rule
D02	NCT-INS-DS-0101	Ingress protection IP65 instead of IP66.	✓ rule
D03	NCT-INS-DS-0101	Tag 'PT-1012' does not follow the tag format.	✓ rule
D04	NCT-INS-DS-0101	SIS transmitter 110-PT-1013 is only SIL 1 capable.	✓ rule
D06	NCT-MEC-SP-0201	Body material ASTM A105 for valves exposed to -46 degC.	✓ rule
D07	NCT-MEC-SP-0201	O-ring seals in standard NBR, not RGD-qualified.	✓ agent
D08	NCT-MEC-SP-0201	No fire-safe section.	✓ rule
D09	NCT-MEC-SP-0201	Fugitive emission class C instead of class B.	✓ rule
D10	NCT-MEC-SP-0201	Refers to superseded Design Basis NCT-PRC-DB-0001 Rev C.	✓ rule
D11	NCT-MEC-SP-0201	Maximum allowable working pressure given as 2,100 psi (imperial unit).	✓ rule
D12	NCT-PRC-PID-0301	Title block field APPROVED is empty.	✓ rule
D13	NCT-PRC-PID-0301	Instrument 110-PT-1014 on the P&ID is not in the instrument index.	✓ rule
D14	NCT-PRC-PID-0301	Line 16"-CO2-1003-B9C is not in the line list.	✓ rule
D15	NCT-PRC-PID-0301	Note states design pressure 135 barg; design basis is 150 barg.	✓ rule
D17	NCT-PRC-PID-0302	Landfall ESD valve 120-XV-2001 shown as Class 600.	✓ rule
D18	NCT-PRC-PID-0302	Title block revision 'C1' is not a valid revision code.	✓ rule
D19	NCT-PRC-PID-0302	Instrument 120-TT-2012 on the P&ID is not in the instrument index.	✓ rule
D20	NCT-PIP-LL-0401	Line 32"-CO2-2001-B9C design pressure 140 barg instead of 150 barg.	✓ rule
D21	NCT-PIP-LL-0401	Line 16"-CO2-1002-B9C corrosion allowance 1.5 mm instead of 3.0 mm.	✓ rule
D22	NCT-PIP-LL-0401	Vent line 2"-VNT-1005-L9C minimum design temperature -20 degC instead of -46 degC.	✓ rule
D23	NCT-INS-IL-0501	Tag 110-PT-1011 appears twice.	✓ rule
D24	NCT-INS-IL-0501	Instrument references P&ID NCT-PRC-PID-0399, which does not exist.	✓ rule
D25	NCT-INS-IL-0501	Index range for 110-PT-1013 is 0-100 barg.	✓ rule
D26	NCT-PIP-CA-0601	Calculation uses design pressure 135 barg.	✓ rule
D27	NCT-PIP-CA-0601	Corrosion allowance 1.0 mm.	✓ rule
D28	NCT-PIP-CA-0601	Line pipe grade L485ME is not an allowed grade.	✓ rule
D29	NCT-PIP-CA-0601	Design factor 0.77 exceeds 0.72.	✓ rule
D30	NCT-MEC-PR-0701	Houdtijd 8 uur (24 h required).	✓ rule
D31	NCT-MEC-PR-0701	No drying section / dew point requirement.	✓ rule
D32	NCT-MEC-PR-0701	Testdruk 210 barg (225 barg required).	✓ rule
D33	NCT-MEC-PR-0701	Recorder calibration up to 12 months old allowed (6 months required).	✓ rule
D35	NCT-CIV-DW-0801	Vent stack 45 m from the site boundary (60 m required).	✓ rule
D36	NCT-CIV-DW-0801	Control room 22 m from the compressor building (30 m required).	✓ rule
D37	NCT-QAC-ITP-0901	Girth weld NDT extent 10% instead of 100%.	✓ rule
D38	NCT-QAC-ITP-0901	References procedure NCT-QAC-PR-0999, which is not in the document register.	✓ rule
D39	NCT-QAC-ITP-0901	Pressure test is a hold point for the contractor only; the owner only reviews records (owner hold point required).	✓ agent

Findings outside the planted set

Counted as false positives for precision. Some may be genuine issues the corpus author did not plant; they are shown so you can judge.

agent

NCT-MEC-PR-0701 · NCT-R-AFP-005

The instrumentation section lists pressure and temperature recording equipment but does not mention a calibrated hygrometer or dew point measurement at the far end of the section.

agent

NCT-MEC-PR-0701 · NCT-R-AFP-016

The requirement allows dry air or nitrogen as drive medium; the procedure restricts the medium to dry air only, which is more restrictive than the specification and omits the nitrogen option.

agent

NCT-PIP-CA-0601 · NCT-R-PIP-006

The SMYS input is traced only to 'ISO 3183 (number only)', which is not a referenced project document with a revision, so the input is not traceable as required.

How to read this: the requirements register, the documents and the planted defects were all written by PitchAI for this demo, and the deterministic rules were configured from the register. The scores show that the pipeline works end to end on known defects; they are not a claim about quality on real project documents. In a pilot, the same measurement runs on documents engineers have already reviewed, with their comments as the reference.