NIDataCollector

Imported from sgh21/NIDataCollector (AGENTS.md). Install upstream with npx skills add sgh21/NIDataCollector. Copyright stays with the author.

• Put the conclusion first after experiments or debugging.
• Do not generate final HTML reports with Python scripts; summarize them manually.
• Reuse existing code where possible and prefer the smallest practical change.
• Clean temporary code, intermediate files, and __pycache__.

Current Project Notes

• The architecture was simplified on 2026-07-07 and now uses three layers:
  • src/nidata_collector/hardware/: hardware communication and control boundaries.
  • src/nidata_collector/core/: acquisition control, worker scheduling, and data storage.
  • src/nidata_collector/ui/: Qt UI, plotting, and user interaction.
• The acquisition and control devices are managed separately:
  • NI cDAQ: hardware/ni.py.
  • DAMX-8013 NTC temperature card: hardware/damx8013.py.
  • Spindle controller: hardware/spindle.py.
  • X505A tension/compression force sensor: hardware/x505a.py.
• Old root-level compatibility entry modules were intentionally removed. Current entries are:
  • UI: scripts/run_monitor.py -> nidata_collector.ui.qt_app.
  • NI probe: scripts/ni_probe.py -> nidata_collector.hardware.ni.
• NI, DAMX-8013, spindle, and X505A must remain independently usable. One offline device must not block the others.
• DAMX-8013 is fixed as a two-channel NTC temperature card. COM, R value, and B value come from config/temperature_card.json.
• The UI Temperature settings tab shares sample rate, segment length, and temperature range between RTD and NTC. RTD-only fields must be explicitly labeled RTD.
• Spindle serial settings, protocol addresses, polling, and safety limits come from config/spindle_control.json.
• Startup UI defaults are centralized in config/app_startup.json. Keep hardware protocol details in the device-specific config files.
• Per-channel Meta dialog defaults are stored in config/app_startup.json under channel_metadata, keyed by physical channel name.
• X505A integration decisions (2026-07-29):
  • The X505A SDK and acquisition-software integration are implemented: force reading, explicit display zero, manual calibration-parameter backup, independent worker scheduling, live display/plotting, raw and tagged segment storage, summaries, and conditional five-panel trends.
  • The first public Python SDK provides force reading, explicit display zero, and explicit read-only calibration-parameter backup.
  • X505A communication requires PTC mode 0 (Modbus-RTU). The SDK must validate this prerequisite and must not change the protocol mode automatically.
  • The connected force sensor has a declared measurement range of 0-50 kg. The Force live plot defaults to 0-10 kg; this display window does not limit or clip raw acquisition/storage.
  • The current X505A serial port default is COM13; keep the port editable in the Force settings page.
  • Default acquisition is 10 Hz, stored in 10 second segments of 100 samples.
  • Each force reading must expose the signed engineering value, unit, decimal setting, and raw int32 value.
  • The acquisition application integrates X505A reading, display, live plotting, and storage. Its display-zero control is available only while acquisition is stopped and requires confirmation.
  • Display zero writes 32-bit value 10 to address 3000 and reads back once. It must never write calibration weight address 3002 or calibration-save value 20.
  • Calibration-parameter backup is only run after an explicit SDK call. Connection, reading, display zero, and acquisition startup must not trigger backup automatically.
  • The backup includes unit, decimal setting, HDZ, HDF, CAL, ZEP, RAG, NL1, and all U01/F01 through U40/F40 nonlinearity parameters.
  • Store versioned JSON backups under data/device_backups/x505a/<device_alias>/ without overwriting historical files.
  • Weight calibration, digital calibration, and parameter restore are not implemented. Do not infer missing digital-calibration registers, formulas, or save sequences.
  • Offline protocol, storage, worker-isolation, trend-layout, compile, and offscreen UI checks pass. Real-device force reading, calibration-parameter backup, and display-zero execution still require explicit stopped-state validation.
• Raw acquisition segments are stored as structured .npz.xz files using npz payloads compressed with lzma/xz.
  • Each raw segment must include time_s, data, channels, sample_start_index, sample_rate_hz, signal_type, and unit.
  • time_s[i] corresponds to data[:, i]; do not rely on reconstructing time only from sample index.
  • Do not use pickle for experiment data storage.
• Do not generate per-segment JSON sidecars for new runs. Use one run-level manifest.json, segment_records.csv, and segment_summary.csv.
• Default segment duration is about 10 seconds: vibration 256000 samples at 25600 Hz, temperature 100 samples at 10 Hz, and force 100 samples at 10 Hz. The UI may still override these values.
• segment_summary.csv uses fixed 1 second summary windows, even when raw .npz.xz files are saved as 10 second segments.
• Postprocessing after recording should generate segment_summary.csv and trends/summary_overview.png.
  • Vibration summary features: mean_abs, max, min.
  • Temperature summary features: mean, max, min.
  • Force summary features: mean, max, min.
  • Spindle speed/current summary features: mean, max, min.
  • Without force data, trend overview remains four stacked subplots: vibration channels together, temperature channels together, spindle speed alone, spindle current alone.
  • With force data, insert force as a fifth subplot between temperature and spindle speed.
  • Do not plot spindle speed and current on the same subplot or twin Y axis.
• Use scripts/inspect_npz_xz.py as the independent temporary tool for reading and previewing .npz.xz raw segments.
• Channel metadata defaults are loaded on Refresh and saved back to config/app_startup.json on GUI close; blank metadata entries should not be persisted.
• Live plot downsampling must apply a stateful anti-aliasing low-pass filter before decimation. Do not downsample live signals with direct slicing such as values[::step].
  • Anti-alias plot processing is visualization-only; raw acquisition and .npz.xz storage must retain the original sample rate and values.
  • The vibration plot is a single anti-aliased waveform. Do not use a continuously filled min/max peak envelope: it is not a time waveform and caused unacceptable rendering load.
  • The plot point budget is 4096 points per curve. At the default 0.25 s vibration window and 25.6 kHz source rate, the anti-aliased display rate is 12.8 kHz with a 5.12 kHz passband and 3200 displayed points.
  • The live plot redraw interval is about 33 ms (30 FPS). Data ingestion remains independent of redraw timing.
  • Increasing the vibration window automatically reduces display sample rate and anti-alias bandwidth. The plot title must show the effective display rate, passband, and window.
  • High-frequency spindle health decisions must still use raw data, band features, order analysis, or envelope analysis rather than the live waveform alone.
• When the user is actively acquiring data, do not launch the application, instantiate a Qt UI, refresh devices, connect hardware, release device reservations, or manipulate the running Python process. Restrict validation to non-UI unit tests, compile checks, and offline data processing until acquisition is finished.
• 2026-07-07 spindle experiment conclusions after compressed-storage integration:
  • Main analyzed run: data/runs/run_20260707_185820.
  • This run supports the standard data format decision and the 6000 rpm standard acquisition flow.
  • The run stepped through 500, 1000, 2000, 3000, 4000, 5000, 6000, 7000, and 8000 rpm, then returned to 6000 rpm before shutdown.
  • For formal 6000 rpm standard acquisition, do not overshoot to 8000 rpm; ramp and stabilize directly at 6000 rpm.
  • Final switch to 6000 rpm occurred at about 121.015 s; actual speed reached 6000 rpm near 121.5 s.
  • 6000 rpm hold lasted about 582 s, or 9.7 min, before deceleration.
  • NTC is the primary temperature and safety/thermal-stability channel. In this run, NTC rose from about 23.98 degC at 6000 rpm stabilization to about 26.55 degC before deceleration.
  • 6000 rpm thermal stabilization estimate from NTC: at least 8 min after speed stabilizes; use 8.5-9 min for a conservative wait.
  • RTD changed little in this run and should be recorded but not used as the primary thermal-stability gate until its placement/sensitivity is improved.
  • Spindle current data has negative values and spikes; record it for now, but do not use it for load, thermal stability, or data-validity decisions until register meaning/scaling/sign are verified.
• 2026-07-22 7000 rpm monitoring conclusions from data/runs/run_20260722_191713/tags/tag_0003_20260722_193619:
  • The tagged target was 7000 rpm and actual speed was 6999.8 rpm; 7000 rpm corresponds to a 116.7 Hz shaft fundamental.
  • The 60.05 s capture used NTC temperature 27.6 degC with a 0.0066 degC/min slope and passed the configured thermal-balance gate.
  • AI0 and AI1 are the front-bearing vibration directions. AI2 is mounted on the bench and is a foundation/reference channel, not a direct bearing-health channel.
  • The original acceleration data was dominated by higher-frequency energy while 1x and 2x components were small. Do not infer spindle health from a visually downsampled waveform or from one speed alone; compare repeatable speed/load/temperature baselines and use raw-data spectral and envelope features.
• 2026-07-22 6800-7400 rpm speed-scan conclusions from data/runs/run_20260722_speed_scan1_6800_7400rpm:
  • The scan contains seven approximately 60 s tagged captures at 6800, 6900, 7000, 7100, 7200, 7300, and 7400 rpm, with raw vibration retained at 25.6 kHz.
  • Use 7000 rpm as the primary standard speed for spindle-health estimation and 7100 rpm as the secondary confirmation speed. This recommendation currently has moderate confidence because it is based on one ascending scan; validate it with at least three independent thermally balanced runs and a descending-speed check before freezing the production baseline.
  • At 7000 rpm, measured speed was 7000.009 rpm mean with 0.627 rpm standard deviation and about 2.99 rpm peak-to-peak variation. NTC was 27.6 degC with a 0.0066 degC/min slope. The AI0/AI1 front-bearing vector RMS was about 0.1661 g with 2.70% ten-second-segment CV, while AI2 reference RMS was 28.5% of the front-bearing vector RMS.
  • 7100 rpm had the best per-axis short-term repeatability among the central candidates, but a lower front-bearing signal level and a higher NTC slope in this scan; use it to confirm an anomaly observed at 7000 rpm rather than as the sole primary baseline.
  • Do not use 6800 rpm as the primary baseline: AI2 reference RMS was 73.5% of the front-bearing vector RMS and the approximately 52.29x component was strong on both AI0 and AI2, indicating substantial bench/foundation response.
  • Avoid 7400 rpm as the primary baseline because AI1 ten-second-segment RMS CV reached about 15.14%. The approximately 46x response also rose from 7200 through 7400 rpm, so 7300-7400 rpm is more sensitive to structural-response variation.
  • For a repeatable 7000 rpm standard capture, target actual speed within +/-5 rpm, require an NTC slope no greater than 0.02 degC/min and temperature within about +/-0.5 degC of the healthy baseline, then record at least 60 s of raw 25.6 kHz vibration.
  • Health features should include AI0/AI1 overall RMS and crest factor, 10-500 Hz, 500-2000 Hz, 2-5 kHz, and 5-10 kHz band RMS, approximately 46x and 52.29x tracked amplitudes, and a 2-10 kHz envelope spectrum. Use AI2 to reject fixture/foundation changes; do not classify spindle health from AI2 or from 1x/2x alone.
• Notion tracking for the 2026-07-07 experiment:
  • 01.04 acquisition-chain validation: supplemented with the standard data acquisition format.
  • 03.02 stepped speed-up vibration/temperature acquisition: supplemented with run run_20260707_185820 and marked completed.
  • 03.04 thermal-stability analysis: supplemented with the 6000 rpm thermal-stability analysis and marked in progress.
  • 03.05 pre-acquisition wait-time recommendation: supplemented with the 6000 rpm standard acquisition flow V1 and marked completed.
• The full pre-refactor baseline is preserved at tag baseline-before-architecture-refactor-20260707.
• Current function and API docs are:
  • docs/architecture.md
  • docs/api.md
• Historical HTML reports, reference projects, one-off test data, and old test scripts were cleaned. Do not reintroduce unused docs or temporary test artifacts.

Common Validation

• Compile check: E:\software\conda\envs\NI\python.exe -B -m py_compile scripts\run_monitor.py scripts\ni_probe.py scripts\inspect_npz_xz.py src\nidata_collector\hardware\x505a.py src\nidata_collector\core\engine.py src\nidata_collector\core\storage.py src\nidata_collector\ui\plot_resampling.py src\nidata_collector\ui\startup_config.py src\nidata_collector\ui\qt_app.py
• Use QT_QPA_PLATFORM=offscreen for offscreen UI checks.
• Clean generated __pycache__ directories after validation.