Fixes#1876
Problem
Files are processed in ~13KB chunks by default (10KB of actual data plus a 3KB peek into the next chunk). To infer the line the engine would take the line from chunk's metadata SourceMetadata.Line and would consider it absolute, that is it's relative to the entire file. The problem is it was never set anywhere. Since every chunk had line equals to 1 in metadata, all secret scan results appeared at line 1 + FragmentLineOffset() which resulted in line being relative to the chunk not the file itself.
Solution
Track cumulative line numbers in handleNonArchiveContent() by counting newlines as chunks are processed. Each DataOrErr now carries the correct starting line, which flows through to SourceMetadata.Filesystem.Line, giving FragmentFirstLineAndLink() the correct fragStart for the final calculation.
Affected Sources
Directly fixed:
Filesystem - now reports accurate line numbers. I have tested it using the test file provided in the issue wget https://gist.githubusercontent.com/det/1526b4c16d0e07ac023d75c912a68658/raw/c3061c14a811205a65cbdcf0065bd3c11d88bfcb/test.txt
Not affected (no Line field in proto):
S3, GCS, Jenkins, stdin - use handlers but their metadata protos lack a Line field. I think it makes sense for S3 and GCS to report line numbers so it could be a good future change.
Partially affected (own line tracking):
Git, GitHub, GitLab - regular text diffs use git-based scanning with built-in line tracking (unaffected). However, binary archives (tar.gz, zip) go through HandleBinary() → handlers.HandleFile() and benefit from this fix.
others like BitBucket use their own implementations so not affected
Gleb Haranin
f3eff52825