Skip to content

Split long Bugzilla buglist references into per-bug references - #2512

Open
NachiketPatil26 wants to merge 1 commit into
aboutcode-org:mainfrom
NachiketPatil26:2500-split-long-bugzilla-references
Open

NachiketPatil26 wants to merge 1 commit into
aboutcode-org:mainfrom
NachiketPatil26:2500-split-long-bugzilla-references

Conversation

@NachiketPatil26

Copy link
Copy Markdown

Fixes #2500

Some advisories (for example the Mozilla ones coming through NVD) reference a buglist.cgi?bug_id=... URL that lists dozens of bugs and is longer than the 1024 characters AdvisoryReference.url allows, so inserting the references failed with value too long for type character varying(1024).

get_or_create_advisory_references now splits a Bugzilla buglist URL that is too long into one show_bug.cgi?id=N reference per bug, keeping the reference type and setting reference_id to the bug id. Everything else, including short buglist URLs, is stored as before. Doing it there covers every importer.

I went with splitting rather than a TextField because url is part of unique_together with reference_type, so a long enough URL would still hit Postgres' index row size limit.

Tests:

  • test_get_or_create_advisory_references_splits_long_bugzilla_buglist uses a 120-bug buglist URL; it fails on main with the same DataError as in the issue.
  • test_get_or_create_advisory_references_keeps_short_bugzilla_buglist checks short buglist URLs are left alone.
  • pytest -m "not webtest": 919 passed, 1 skipped.

Some advisories reference a buglist.cgi URL that lists dozens of bug ids
and is longer than the 1024 characters allowed for a reference URL, so
inserting the advisory references failed with a DataError. When such a
URL is too long, store one show_bug.cgi reference per bug instead.

Reference: aboutcode-org#2500
Signed-off-by: Nachiket <nachiketpatil2604@gmail.com>

@sanchit955 sanchit955 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked this out locally against PostgreSQL 16 (head 0f248c9). Both new tests pass, and the full vulnerabilities/tests/pipes/test_advisory.py file is 14/14 green. I also confirmed the new splits test fails on upstream main with the exact DataError: value too long for type character varying(1024) from #2500, so the test pins the fix. black --check is clean on both changed files.

One thing worth a thought for a follow-up: a long URL that is not a Bugzilla buglist still goes through unchanged, so it would still raise the same DataError and abort the advisory import. The Bugzilla case from the issue is covered — just flagging that the general case is still open.

@kmlyz kmlyz left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

split_long_bugzilla_buglist cleanly splits oversized /buglist.cgi URLs into show_bug.cgi?id=<bug_id> references for both Reference and ReferenceV2, while bulk_create(..., ignore_conflicts=True) safely handles duplicate IDs.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Error while inserting long bugzilla reference URLs

3 participants