Skip to content

ETL Tests #47

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
vladd-bit merged 102 commits into from
Jan 27, 2026
Merged

ETL Tests #47

vladd-bit merged 102 commits into from
Jan 27, 2026

Conversation

@vladd-bit
Copy link
Member

@vladd-bit vladd-bit commented Nov 27, 2025 •
edited
Loading

This PR attempts to accomplish the following:

  • health checks for all core data-infra services: NiFi, NiFi-Registry, NGINX(Optional), Elastic, PgSQL, Kibana
  • tests harness for the provided default NiFi (v2.0+) ingestion workflows.
  • refactoring of older scripts and code-base for cleaner scripting
  • tests for some provided on-the-fly scripts in "nifi/user-scripts"
  • better environment variable validation & handling
  • additions to resource limits & env vars for each of the core containers (NiFi, NiFi Registry, Nginx, Metric/file(beats), Kibana, Gitea, Elastic, PgSQL

@vladd-bit
Deploy: added env var checkup script.
3980318
@vladd-bit
NiFi: updated docker image + packge list to install.
3318a4e
@vladd-bit
Merge branch 'main' of https://github.com/CogStack/CogStack-NiFi into…
a20cf8b 
… etl_tests
@vladd-bit
NiFi: dockerfile fix.
aa306b9
@vladd-bit
Deploy refactioring + pyproject.
b4ef495
@vladd-bit
.gitignore update.
496c612
@vladd-bit
Deploy: added resource limits to core services (NiFi/NiFiRegistry/Ngi…
3e0f848 
…nx/Kibana/ES/Metricbeat/FileBeat}
@vladd-bit
Deploy: env + services def updates (resource constraints per service).
c0f81b8
@vladd-bit
NiFi: dockerfile update.
5e0894c
@vladd-bit
Transioning NiFi scripts to python package.
9eec629
@vladd-bit
Re-added user scripts/templates/python_extensions folders.
66bfc7d
@vladd-bit
Re-added user_scripts folder. Added configs for pgsql/nifi/elastic se…
9f7e697 
…rvices + health tests.
@vladd-bit
NiFi: updated user_scripts.
05fc3b3
@vladd-bit
Security: fixed script typo.
ef26a79
@vladd-bit
Merge branch 'main' of https://github.com/CogStack/CogStack-NiFi into…
3e5cccb 
… etl_tests
@vladd-bit
Dev requirements + docker template gha update.
5c62a3c
@vladd-bit
Merge branch 'main' of https://github.com/CogStack/CogStack-NiFi into…
fe0b072 
… etl_tests
@vladd-bit
Update submodules to latest release tags (or main)
2af6cd8
@vladd-bit
Deploy: updated docker services restart flag on core services.
c730cf3
@vladd-bit
.gitignore update.
f3b805f
@vladd-bit
NiFi: updated processor user_scripts path ref.
b8df82e
@vladd-bit
Updated path refs.
f43467f
@vladd-bit
Updated path refs to NiFi scripts folders.
48a06ee
@vladd-bit
Updated path refs to NiFi scripts folders II.
25d5be0
@vladd-bit
Updated .gitattributes.
e45d5c4
@vladd-bit vladd-bit self-assigned this Dec 10, 2025
@vladd-bit vladd-bit added enhancement New feature or request help wanted Extra attention is needed github_actions Pull requests that update GitHub Actions code and removed help wanted Extra attention is needed labels Dec 10, 2025
vladd-bit and others added 28 commits January 14, 2026 14:57
@vladd-bit
Deploy: fixed nifi startup issue (conf folder change).
daa7e8a
@vladd-bit
Deploy: fixed dev services NiFi conf folder mount.
48248ac
@vladd-bit
Deploy: corrected nifi-registry mount (conf).
7a73a19
@vladd-bit
Services: updated opensearch(-dashboards) version to 3.4.0.
6fe0f40
@vladd-bit
NiFi: added sample workflow Azure -> Blobs[Parquet] -> Elasticsearch.
1628411
@vladd-bit
Services: updated OpenSearch system_indices setting.
cc54ebf
@vladd-bit
Services: updated OS-dashboards to pre-select global tentant.
feb7011
@vladd-bit
Security: updated OpenSearch-Dashbaords roles permissions on default …
97f3e01 
…indices.
@vladd-bit
NiFi: added NDJSON support for OCR processor.
336c2a3
@vladd-bit
NiFi: updated requirements.txt
05e0c0e
@vladd-bit
NiFi: corrected/updated type declarations & returns for native proces…
a78a411 
…sors.
@vladd-bit
Services: updated NGINX conf, removed old way relying on ENVSUBST com…
7f1dbe3 
…mand in compose file for env var replacements.
@vladd-bit
Services: removed duplicate files from NGINX confs, now using templat…
c599a73 
…e as the ground source for configuration.
@vladd-bit
Deploy: updated dev compose file + Makefile.
52e67a2
@vladd-bit
Services: nginx conf - added new location regex matcher for nifi cont…
2c08988 
…ent viewer, delete old refs.
@vladd-bit
Deploy: updated services.yml (removed redundant env vars)
ae4b7eb
@vladd-bit
Deploy: updated Pgsql to 17.7. Removed Nginx env var.
9caf816
@vladd-bit
NiFi: added convertjson to attr processor.
22fe7ab
@vladd-bit
GHA: smoke tests for NiFi.
1833bb4
@vladd-bit
Updated all .sh scripts in repo to be executable.
a0580b2
@vladd-bit
GHA: updated smoke test for NiFi + added shell script exec enforcer c…
aee255c 
…heck.
@vladd-bit
Services: updated opensearch timeout settings.
298920e
@vladd-bit
Services: Opensearch(/+Dashboards) enabled SQL-like query settings.
9b7123a
@vladd-bit
Fixed executable permissions.
aed1847
@vladd-bit
GHA: fix nifi smoke text II.
a138b3e
@vladd-bit
Merge branch 'main' into etl_tests
12de564
@vladd-bit
NiFi scripts: deleted old file refs (parquet utils etc.)
d6997b7
@vladd-bit
NiFi scripts: cerner blob decompression additions.
6eb2b3d
@vladd-bit vladd-bit merged commit cbe2bf7 into main Jan 27, 2026
35 checks passed
@vladd-bit vladd-bit deleted the etl_tests branch January 27, 2026 22:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@vladd-bit vladd-bit

Labels

enhancement New feature or request github_actions Pull requests that update GitHub Actions code python Pull requests that update python code

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@vladd-bit