[HWORKS-284] Documentation for exporting logs#142
Open
kouzant wants to merge 1 commit intologicalclocks:mainfrom
Open
[HWORKS-284] Documentation for exporting logs#142kouzant wants to merge 1 commit intologicalclocks:mainfrom
kouzant wants to merge 1 commit intologicalclocks:mainfrom
Conversation
SirOibaf
suggested changes
Jul 7, 2023
|
|
||
| ## Introduction | ||
| Hopsworks collects services and applications logs to [Logstash](https://www.elastic.co/logstash/) which then forwards them to OpenSearch for indexing. | ||
| Often organizations already have logging systems in place so streaming Hopsworks logs is necessary. |
Contributor
There was a problem hiding this comment.
so exporting HOpsworks logs is necessary?
| Often organizations already have logging systems in place so streaming Hopsworks logs is necessary. | ||
|
|
||
| ## Prerequisites | ||
| To configure Logstash streaming logs outside of Hopsworks you will need SSH access to the cluster (Logstash node). Also, depending on the target system you might |
Contributor
There was a problem hiding this comment.
To configure Logstash to stream logs.
| need authentication tokens or opening firewall rules. | ||
|
|
||
| ## Export logs | ||
| Logstash is a well established log collection service with many output [plugins](https://www.elastic.co/guide/en/logstash/7.17/output-plugins.html) available. |
Contributor
There was a problem hiding this comment.
Logstash is not a log collection service, it's just a processing pipeline.
| Logstash process logs in *pipelines* where each pipeline is responsible for a logical group of logs. In Hopsworks we have multiple pipelines and their configuration files are under `/srv/hops/logstash/config` | ||
|
|
||
| ### Export services logs | ||
| To stream various services' logs outside of Hopsworks you will need to **create another pipeline** similar to `services`. |
Contributor
There was a problem hiding this comment.
similar to the services pipeline.
| !!! note | ||
| Take a note of the pipeline address as we will use it in Step 2 | ||
|
|
||
| At the end of the file is the `output` section which currently forwards them to OpenSearch. Replace the output section with a sample block such as |
Contributor
There was a problem hiding this comment.
forwards the logs to OpenSearch
| pipeline.batch.size: 50 | ||
| ``` | ||
|
|
||
| **Instruct** the services pipeline to push logs also in the newly created pipeline by appending to `services-intake` for example: |
Contributor
There was a problem hiding this comment.
also to the newly created pipeline
|
|
||
|
|
||
| ### Export Spark logs | ||
| To stream applications' logs to another system the Steps are fairly similar to exporting services logs but need some additional configuration. |
|
|
||
|
|
||
| ### Export Spark logs | ||
| To stream applications' logs to another system the Steps are fairly similar to exporting services logs but need some additional configuration. |
Contributor
There was a problem hiding this comment.
I would also specify Spark applications logs so users don't confuse with Hopsworks application.
| Finally you should restart Logstash `sudo systemctl restart logstash` | ||
|
|
||
| ## Conclusion | ||
| It is not easy to write a guide for a task that can be achieved in many different ways but in this guide we gave solid |
Contributor
There was a problem hiding this comment.
Users don't care that it's not easy. Just refer them to the different plugins configuration to understand how they need to configure their new pipeline to send data wherever they need to.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://hopsworks.atlassian.net/browse/HWORKS-284