Handling text logs in Formant

Q: My program writes text logs. How can I ingest these in Formant?

My program writes diagnostic information to a log file.

Formant agent logs

You can read more about Formant agent logs here: How do I generate Formant agent logs?

Custom text logs from your program

If your program writes logs to a text file which you want to ingest into Formant, and multiple processes write to that log file at the same time, you may miss some log entries due to rate limiting on your stream. You can work around rate limiting in the following ways.

File tail with different regexes

You can configure multiple file tail streams on your log text file with a separate regex search for each process writing to the log file. For example, if you have two processes, BATTERY and MOTOR writing to the log file, you can set up two file tail streams on the same text file, one with a regex for BATTERY* and one with a regex for MOTOR*. Each file tail stream will only ingest new lines in the log file which contain its respective regex match.

Queue data either on robot or in adapter

You can buffer text data from your log either on your robot-side application, or using an adapter, which the Formant agent can ingest from at the rate limit you specify.

Upload entire log files using directory watch

You can upload an entire log file when the logging process is complete using a directory watch stream. This will ingest the entire file when it is created in a directory you specify.